Quote from: david72 on March 19, 2023, 08:04:24 AMI am trying to only scan the folders without saving html files
Anyone have idea how to do the same ?
Due to huge files and deep folders , its difficult store index files
Quote from: hajzlik on December 09, 2021, 11:02:09 AMThere should be an option to set different limits for HTML and for non-HTML files.
Let's say you have a sitemap. You want to download all the linked pages, but not any further links.
You can limit the distance from root URL to 1, but then you end up with HTML pages without images and other content.
You can limit the distance from root URL to 2, and the other content will download, but you will end up with a bunch of unwanted HTML files (but without any images they contain, which makes them even more useless).
Quote... it doesn't yet support TLS 1.3