- Welcome to Cyotek Forums.
Recent posts
#21
WebCopy / Using 'input-file'
Last post by Jake Morrison - June 11, 2024, 09:19:27 PMHi,
Both the CLI & GUI command lines allow '/input-file <file>' to be used.
I'm wondering if there is a description available of the file format or any other help with using 'input-file'.
Also, I do not find a way to use an input file within the program. Have I missed it somehow?
thanks,
Jake
Both the CLI & GUI command lines allow '/input-file <file>' to be used.
I'm wondering if there is a description available of the file format or any other help with using 'input-file'.
Also, I do not find a way to use an input file within the program. Have I missed it somehow?
thanks,
Jake
#22
WebCopy / Need to control timeout for we...
Last post by Awful_Genius - June 08, 2024, 08:14:29 AMHi all!
I'm using WebCopy for a long time and often get error: "The operation has timed out" for some files while crawling:
After some researching I got that the most likely that WebCopy use something like WebRequest .NET method and use small timeout for it:
System.Net.WebException: The operation has timed out
Sometimes Exception raised for big files, which web server can't send quickly.
So, can you add to WebCopy functionality to control requests timeouts?
Other option - redownload option for Errors tab
I'm using WebCopy for a long time and often get error: "The operation has timed out" for some files while crawling:
After some researching I got that the most likely that WebCopy use something like WebRequest .NET method and use small timeout for it:
System.Net.WebException: The operation has timed out
Sometimes Exception raised for big files, which web server can't send quickly.
So, can you add to WebCopy functionality to control requests timeouts?
Other option - redownload option for Errors tab
#23
WebCopy / How to crawl remote sitemap.xm...
Last post by chillybang - June 03, 2024, 05:42:33 PMHey!
I must say, I'm hardly impressed after switching from HTTrack to Cyotek - it is like a millenium switch
As a newbie I miss a feature or rather the way to find it - namely to crawl URLs from a remote, live sitemap.
I tried to add a sitemap as https://example.com/sitemap.xml to "Load additional URLs from file" - but this doesn't work (rises an alert on saving).
Do I miss something? What is the way to crawl all URLs from a remote live sitemap? Saving site's sitemap locally to use it as an URL list isn't very useful because sites update their sitemaps...
I must say, I'm hardly impressed after switching from HTTrack to Cyotek - it is like a millenium switch
As a newbie I miss a feature or rather the way to find it - namely to crawl URLs from a remote, live sitemap.
I tried to add a sitemap as https://example.com/sitemap.xml to "Load additional URLs from file" - but this doesn't work (rises an alert on saving).
Do I miss something? What is the way to crawl all URLs from a remote live sitemap? Saving site's sitemap locally to use it as an URL list isn't very useful because sites update their sitemaps...
#24
WebCopy / Local File Links issue + some ...
Last post by DavesNotHereDude - June 02, 2024, 04:07:54 PMI've having a hard time getting URL's to link properly on a site where the pages are hosted in multiple sub directories.
Example.
Although the root directory is https://guides.goingviralnewsletter.hookpoint.com/
The page I want to download started at https://guides.goingviralnewsletter.hookpoint.com/0c6dd8e9/school-of-hard-knockz/
I'm able download the site, and when I open the below index.htm, I'm able to load the page (some images don't load, more on that later)
C:\Downloaded Web Sites\guides.goingviralnewsletter.hookpoint.com\0c6dd8e9\school-of-hard-knockz\index.htm
When I click a link on the loaded page, it redirects to file:///C:/0c6dd8e9/school-of-hard-knockz/LINK
Instead of the full URL.
Note, this is only when I disable "Remap references within downloaded file". When this is enabled, the index.htm on https://guides.goingviralnewsletter.hookpoint.com/0c6dd8e9/school-of-hard-knockz/ also loads "Your file couldn't be accessed" because it references file:///C:/0c6dd8e9/school-of-hard-knockz/
Second issue is some of the content doesn't load.
Couple examples:
If I download https://guides.goingviralnewsletter.hookpoint.com/0c6dd8e9/school-of-hard-knockz/ the banner image doesn't load.
If I download https://guides.goingviralnewsletter.hookpoint.com/0c6dd8e9/school-of-hard-knockz/video-references/ there are links to video's that don't render (not sure if that's fixable?)
The secondary issue is moot if I can't get the primary issue resolved.
I've searched through the forum and I've seen a couple suggestions (I thought Remap references within downloaded file was the fix for this issue, but oddly, when it's turned on, it seems to break the main index file within the root directly I added to start the crawl from)
Example.
Although the root directory is https://guides.goingviralnewsletter.hookpoint.com/
The page I want to download started at https://guides.goingviralnewsletter.hookpoint.com/0c6dd8e9/school-of-hard-knockz/
I'm able download the site, and when I open the below index.htm, I'm able to load the page (some images don't load, more on that later)
C:\Downloaded Web Sites\guides.goingviralnewsletter.hookpoint.com\0c6dd8e9\school-of-hard-knockz\index.htm
When I click a link on the loaded page, it redirects to file:///C:/0c6dd8e9/school-of-hard-knockz/LINK
Instead of the full URL.
Note, this is only when I disable "Remap references within downloaded file". When this is enabled, the index.htm on https://guides.goingviralnewsletter.hookpoint.com/0c6dd8e9/school-of-hard-knockz/ also loads "Your file couldn't be accessed" because it references file:///C:/0c6dd8e9/school-of-hard-knockz/
Second issue is some of the content doesn't load.
Couple examples:
If I download https://guides.goingviralnewsletter.hookpoint.com/0c6dd8e9/school-of-hard-knockz/ the banner image doesn't load.
If I download https://guides.goingviralnewsletter.hookpoint.com/0c6dd8e9/school-of-hard-knockz/video-references/ there are links to video's that don't render (not sure if that's fixable?)
The secondary issue is moot if I can't get the primary issue resolved.
I've searched through the forum and I've seen a couple suggestions (I thought Remap references within downloaded file was the fix for this issue, but oddly, when it's turned on, it seems to break the main index file within the root directly I added to start the crawl from)
#25
ImageBox / Scroll Behavior Adjustment Aft...
Last post by Zeeshanef - May 30, 2024, 10:23:32 AMI am creating a rectangle overlay on an image by a Button click using the code below. My query is, after painting the rectangle, what's the correct method to automatically scroll to the image so that the painted rectangle is visible in the center of the available container?
Code Select
private void ImageBox1_Paint(object sender, PaintEventArgs e)
{
if (drawRectangle)
{
using (Brush brush = new SolidBrush(Color.FromArgb(64, 1, 165, 220)))
{
Rectangle OffSetRect = imageBox1.GetOffsetRectangle(RectDraw);
e.Graphics.FillRectangle(brush, OffSetRect);
}
}
} #26
WebCopy / Special Character in Web Addre...
Last post by blaidd31204 - May 28, 2024, 04:30:10 PMI am trying to copy the following website and I believe the special U character (the one with a tent symbol like over the #6) is causing a problem. I have the most current version of Webcopy. The Yield message that appears does not have any words to indicate the exact problem (I have changed my screen resolution to see if that may help see any message wording but, still no idea if that is the case).
https://forgottenrealms.fandom.com/wiki/Faer%C3%BBn
How do I get this website? Thanks!
https://forgottenrealms.fandom.com/wiki/Faer%C3%BBn
How do I get this website? Thanks!
#27
ImageBox / Re: Selection on image
Last post by needfulhead - May 27, 2024, 08:07:03 AMIf the click occurred outside the selection, we clear the selection by setting selection to Rectangle.Empty. You can also choose to refresh the ImageBox control to update the display and remove the selection visually.
#28
WebCopy / Schedule / Periodically / Auto...
Last post by Pirreke - May 17, 2024, 09:38:19 AMHi,
How do I setup a periodcally autmoated download (scheduled once a month) in WebCopy?
Is there an option to do this? Can't find it in the menu. Online results are confusing.
Regards,
Pirreke
How do I setup a periodcally autmoated download (scheduled once a month) in WebCopy?
Is there an option to do this? Can't find it in the menu. Online results are confusing.
Regards,
Pirreke
#29
WebCopy / URL not writing to download fo...
Last post by FrankWard - April 11, 2024, 03:59:41 PMI'm trying to download a blog. I've encountered several errors I was able to mitigate using regex. However, I'm not sure how to address this one.
When downloading the site it pulls various images from domains like.
http://3.bp.blogspot.com/_-sFohRgxOBI/R3natx-8SXI/AAAAAAAABiA/mp-2BeZnnYk/s1600-h/happy-new-year+Woody+Woodpecker.jpg
In the download folder it's not putting the domain as a folder. It's just writing a TON of the urls starting with the "_"character after the domain like so..
_-sFohRgxOBI/R3natx-8SXI/AAAAAAAABiA/mp-2BeZnnYk/s1600-h/happy-new-year+Woody+Woodpecker.jpg
This happens with the following domains, but not others, so it appears it may be a bug.
http://1.bp.blogspot.com/
http://2.bp.blogspot.com/
http://3.bp.blogspot.com/
http://4.bp.blogspot.com/
Is there a way to make the program remap these domains to specific folders in the downloaded site hierarchy?
Any advice? Thanks!
When downloading the site it pulls various images from domains like.
http://3.bp.blogspot.com/_-sFohRgxOBI/R3natx-8SXI/AAAAAAAABiA/mp-2BeZnnYk/s1600-h/happy-new-year+Woody+Woodpecker.jpg
In the download folder it's not putting the domain as a folder. It's just writing a TON of the urls starting with the "_"character after the domain like so..
_-sFohRgxOBI/R3natx-8SXI/AAAAAAAABiA/mp-2BeZnnYk/s1600-h/happy-new-year+Woody+Woodpecker.jpg
This happens with the following domains, but not others, so it appears it may be a bug.
http://1.bp.blogspot.com/
http://2.bp.blogspot.com/
http://3.bp.blogspot.com/
http://4.bp.blogspot.com/
Is there a way to make the program remap these domains to specific folders in the downloaded site hierarchy?
Any advice? Thanks!
#30
WebCopy / How to auto skip warning of Ja...
Last post by IVAN_CYBERPUNK - April 05, 2024, 01:48:34 PMHi! I'm trying to make copy of some local web page using Cyotek WebCopy 1.10.0.898. After a little while app shows me message "This website appears to be a JavaScript application". After pressing "Ok" on warning screen - app starts to scan web page and successfully finished this process. I have no need to run this script, i'm only want to save web page as html file. So i wonder if is a solution to not press this button all the time? I will add warning screen and fragment of html file with this JavaScript. Thank you.