Cyotek Forums

Products => WebCopy => Topic started by: whiggs on November 20, 2022, 03:46:33 AM

Title: Blank error message when attempting to scan a site
Post by: whiggs on November 20, 2022, 03:46:33 AM
Not sure what the heck this is supposed to mean, but I see this whenever I try to scan a particular website:

(https://i.imgur.com/TfLyNNk.png)
Title: Re: Blank error message when attempting to scan a site
Post by: angtucker on January 20, 2023, 10:39:17 PM
Hi there
I am having same issues did anyone get back to you on this?
Thanks!  :)
Title: Re: Blank error message when attempting to scan a site
Post by: Richard Moss on February 20, 2023, 08:35:50 PM
Hello,

Apologies for the delay in responding.

I've just been looking into this and confirm it is caused by not one, or two, but three bugs. (sigh).

Bug number 1 is the decompression failing (possibly only for streams larger than 65535 bytes).

Bug number 2 is after the decompression fails, it throws an error with a blank message (which is what you are seeing).

Bug number 3 is that when WebCopy runs validation checks to test for common problems that can cause crawling to fail, it ignores the content encoding settings of the project, which by default are Gzip and Deflate. The fact it is ignoring the settings mean you can't work around it by disabling Brotli.

Bugs number 2 and 3 have been fixed in the latest nightly build. The actual issue decompressing data is something I'm still looking into and hopefully will have a fix for in a subsequent nightly. In the interim, if you get a Brotli decompression error, disable it (this is the default for most versions of WebCopy, but there are a few builds where it was on by default).

Regards;
Richard Moss



Title: Re: Blank error message when attempting to scan a site
Post by: Richard Moss on February 22, 2023, 07:02:48 PM
I've pushed out a fix resolving a decompression problem with brotli streams above 65535 bytes, with the added bonus of no longer having to distribute additional binaries for brotli support. If you have version 1.9.1.836 or higher all three issues above should be resolved.

Regards;
Richard Moss
Title: Re: Blank error message when attempting to scan a site
Post by: scottywatty on May 23, 2023, 09:33:35 PM
i am getting this same issue on Windows but do not see a version available past 1.9.0.822. It was working perfectly this morning and not stopped.
Title: Re: Blank error message when attempting to scan a site
Post by: Richard Moss on May 24, 2023, 07:22:19 AM
Hello,

Version 1.9.1 is currently listed in the "Nightly Builds" section of the Downloads (https://www.cyotek.com/cyotek-webcopy/downloads) page.

Regards;
Richard Moss