Text only | Text with Images

Cyotek Forums

Products => WebCopy => Topic started by: End User on September 16, 2016, 12:27:53 AM

Title: Limit download to only first page?
Post by: End User on September 16, 2016, 12:27:53 AM
Hello!

I would like to scan several news sites regularly.
But I don't want to download entire sites.

How should I limit my project to download only first pages of several sites?
(I have a list of URLs)

I tried to read documentation, but this question was not explained there.

Thank you!
Title: Re: Limit download to only first page?
Post by: Richard Moss on September 19, 2016, 04:47:02 PM
Hello,

Thanks for the question and welcome to the forums. And sorry for the delay in responding, I'm not allocating much time for support requests at the moment! Unfortunately, I have to concede a bit of defeat here - it's not a use case I originally considered, and the software makes it remarkably difficult to achieve - I spent a bit of time trying, but the software is a little bit stubborn in this regard - if you add a rule to exclude everything, then it doesn't matter if you add a rule to say include something, it ignores it. Unhelpful :(

You could add rules to exclude all top level pages, but that is time consuming and isn't future proof if the site's structure changes. There's a hacked in level flag which is used by Quick Scan, but as it was just a hack I never exposed it via the front end for normal use.

If you *only* want to download a single page and no other resources at all, you could use the command line version easily enough, using something similar to the below:

wcopy http://www.cyotek.com /o download.html

You could then stick multiple calls to this in a batch file to download the URL list you mentioned. However, this doesn't perform any remapping - it's a direct download only.

I will make sure to include this scenario when I finally get WebCopy's rule system overhauled, so thank you for the suggestion. I hope the above information helps, if not please let me know.

Regards;
Richard Moss
Title: Re: Limit download to only first page?
Post by: End User on September 20, 2016, 01:10:01 AM
G'day!

It's good to hear that the product is growing in functionality!

I'll stick around to be informed ::)

Meanwhile, gonna try to get wget for Windows :D

Thank you!
Title: Re: Limit download to only first page?
Post by: Richard Moss on September 21, 2016, 06:31:48 AM
Hello

Thanks for the follow up. I'm sorry that WebCopy wasn't suitable for your needs, but thanks for the feedback which will hopefully improve the product for future users.

Regards;
Richard Moss
Text only | Text with Images