Cyotek Forums

Products => WebCopy => Topic started by: cybersalt on February 14, 2023, 12:14:33 AM

Title: Lazy Loaded Content
Post by: cybersalt on February 14, 2023, 12:14:33 AM
Thanks for a great tool for downloading a copy of a site.

I realize that WebCopy does not actually download the site but rather saves pages that load as they are spidered.

Is there a way to get WebCopy to scroll to the bottom of each page before saving it? A site we are trying to download for archival purposes ends up missing a bunch of content (text, links, images) because many of the pages are more than one screen in length, with the rest being lazy loaded.
Title: Re: Lazy Loaded Content
Post by: Richard Moss on February 15, 2023, 07:12:55 AM
Hello,

Thanks for the post. I'm sorry, but this is something I currently have no intention of ever supporting. Infinite scroll requires JavaScript, but WebCopy doesn't execute this. While it does haven an experimental mode which uses a headless browser to construct pages, the documentation states (https://docs.cyotek.com/cyowcopy/1.9/javascript.html#unsupported-features) that interactive functions such as scrolling or clicking are not supported - and likely never will be, even when I switch that experimental mode to use modern webview2. Never say never as they say, but right now I have no plans to support it.

Regards;
Richard Moss