Text only | Text with Images

Cyotek Forums

Products => WebCopy => Topic started by: hajzlik on December 09, 2021, 11:02:09 AM

Title: Limit distance from root URL - always one step less for HTML files
Post by: hajzlik on December 09, 2021, 11:02:09 AM
There should be an option to set different limits for HTML and for non-HTML files.

Let's say you have a sitemap. You want to download all the linked pages, but not any further links.

You can limit the distance from root URL to 1, but then you end up with HTML pages without images and other content.

You can limit the distance from root URL to 2, and the other content will download, but you will end up with a bunch of unwanted HTML files (but without any images they contain, which makes them even more useless).
Title: Re: Limit distance from root URL - always one step less for HTML files
Post by: david72 on March 19, 2023, 08:04:24 AM
I am trying to only scan the folders without saving html files

Anyone have idea how to do the same ?
Due to huge files and deep folders , its difficult store index files

Thanks
Dav
Title: Re: Limit distance from root URL - always one step less for HTML files
Post by: Richard Moss on March 19, 2023, 09:35:02 AM
Quote from: hajzlik on December 09, 2021, 11:02:09 AMThere should be an option to set different limits for HTML and for non-HTML files.

Let's say you have a sitemap. You want to download all the linked pages, but not any further links.

You can limit the distance from root URL to 1, but then you end up with HTML pages without images and other content.

You can limit the distance from root URL to 2, and the other content will download, but you will end up with a bunch of unwanted HTML files (but without any images they contain, which makes them even more useless).

Hello,

Belated thanks for the feedback. This makes a lot of sense! I don't think it makes sense to add another option (or maybe add it but not expose it for now, or hide it away somewhere), but I do think it makes sense to completely ignore distance for non-HTML. I've logged that as issue #464.

Thanks again!

Regards;
Richard Moss
Title: Re: Limit distance from root URL - always one step less for HTML files
Post by: Richard Moss on March 19, 2023, 09:37:48 AM
Quote from: david72 on March 19, 2023, 08:04:24 AMI am trying to only scan the folders without saving html files

Anyone have idea how to do the same ?
Due to huge files and deep folders , its difficult store index files

Thanks
Dav


I haven't quite worked out if this is an AI generated spam post, but decided to err on the side of caution this time instead of deleting account and post (https://forums.cyotek.com/announcements/deletion-of-generic-posts/) as I usually do. I did edit the post the remove the link though.

Open the Project menu, click Scan Website
Text only | Text with Images