Cyotek Forums

Products => WebCopy => Topic started by: Awful_Genius on June 08, 2024, 08:14:29 AM

Title: Need to control timeout for web requests
Post by: Awful_Genius on June 08, 2024, 08:14:29 AM
Hi all!

I'm using WebCopy for a long time and often get error: "The operation has timed out" for some files while crawling:

(https://gcdnb.pbrd.co/images/fjT9Xxvxs7cv.png?o=1)

After some researching I got that the most likely that WebCopy use something like WebRequest .NET method and use small timeout for it:

System.Net.WebException: The operation has timed out

Sometimes Exception raised for big files, which web server can't send quickly.

So, can you add to WebCopy functionality to control requests timeouts?

Other option - redownload option for Errors tab

Title: Re: Need to control timeout for web requests
Post by: Awful_Genius on August 09, 2024, 12:42:16 PM
Hi!
Any reaction to post? Any updates? Is this forum for serious discussion or just for throwing dust in the eyes?

Update from my side: tested with latest night build (cyowcopy-1.10.0.898-x64-nightly) - problem is still exists. For one test site with low speed ALL files targeted for crawling (jpg pictures) is get "The operation has timed out" and files wasn't downloaded.

Developers, have you plans to fix it?

Thanks
Title: Re: Need to control timeout for web requests
Post by: Richard Moss on August 18, 2024, 08:00:09 AM
Hello,

The default timeout is 10 seconds but WebCopy currently doesn't expose this as a configuration setting. I've added issue #516 to expose the setting and it will be in the next nightly update.

Retrying to failed downloads is currently slated for 1.11 (#206). That was intended to be an automatic feature, but I've added a note to the issue regarding the possibility of manually triggering a retry.

PS: I don't have a huge amount of time for support requests unless they relate to genuine bugs or feasible feature enhancements. The software is free, so for the most part you're on your own.