Need to control timeout for web requests

Started by Awful_Genius, June 08, 2024, 08:14:29 AM

Previous topic - Next topic

Awful_Genius

Hi all!

I'm using WebCopy for a long time and often get error: "The operation has timed out" for some files while crawling:



After some researching I got that the most likely that WebCopy use something like WebRequest .NET method and use small timeout for it:

System.Net.WebException: The operation has timed out

Sometimes Exception raised for big files, which web server can't send quickly.

So, can you add to WebCopy functionality to control requests timeouts?

Other option - redownload option for Errors tab


Awful_Genius

Hi!
Any reaction to post? Any updates? Is this forum for serious discussion or just for throwing dust in the eyes?

Update from my side: tested with latest night build (cyowcopy-1.10.0.898-x64-nightly) - problem is still exists. For one test site with low speed ALL files targeted for crawling (jpg pictures) is get "The operation has timed out" and files wasn't downloaded.

Developers, have you plans to fix it?

Thanks

Richard Moss

Hello,

The default timeout is 10 seconds but WebCopy currently doesn't expose this as a configuration setting. I've added issue #516 to expose the setting and it will be in the next nightly update.

Retrying to failed downloads is currently slated for 1.11 (#206). That was intended to be an automatic feature, but I've added a note to the issue regarding the possibility of manually triggering a retry.

PS: I don't have a huge amount of time for support requests unless they relate to genuine bugs or feasible feature enhancements. The software is free, so for the most part you're on your own.
Read "Before You Post" before posting (https://forums.cyotek.com/cyotek-webcopy/before-you-post/). Do not send me private messages. Do not expect instant replies.

All responses are hand crafted. No AI involved. Possibly no I either.