chugathon

Author Topic: Did my login succeed?  (Read 191 times)

Offline Thebasser123

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
Did my login succeed?
« on: January 03, 2019, 11:52:59 AM »
Hi,

First of all, thank you for this great tool!
My question is as follows.

Background
I'm trying to download a site with a lot of "courses" on it. Normally I just login and navigate to any page I want.
Only the initial login is required.

Problem
I've set up webcopy to download a portion of this site, using a form for the login part. A lot of stuff is downloaded, but nothing that normally requires a password. The thing is, I have used the "form tester" and it says "you are being redirected". This seems like the form is working, because I get the standard "invalid password" page if I input a wrong password.
note: The actual logging in happens on a different domain. My form uses an URL on that domain. I don't know if that's enough, so I also tried adding this domain to "additional hosts" and explicitly including the login URL by using a rule.

Later on in the results tab, I see a lot of login attempts again, each starting from a particular "private course". Of course, these fail because the URL is different each time and not corresponding to my form's URL.

Question
Does webcopy normally stay logged in like a normal user navigating the site?
And more importantly: do you have any idea what might be going wrong?

I've included a picture of the first results, I don't know if it's of any use.
Thanks in advance

Offline Richard Moss

  • Cyotek Team
  • Administrator
  • Sr. Member
  • *****
  • Posts: 291
  • Karma: +17/-0
    • cyotek.com
Re: Did my login succeed?
« Reply #1 on: January 07, 2019, 04:50:03 PM »
Hello,

Welcome to the forums and sorry you're having troubles with WebCopy.

As you mentioned that you've already tried the tester and it looks like it logged you in, this one is a little perplexing! The most obvious thing to check is that you aren't immediately hitting a "log out" URL - make sure there is a rule to exclude any URL that performs the logout.

Are you using WebCopy 1.6 or below, or 1.7? 1.7 includes a new "use web browser to authenticate" option which may be of help although in this case I'm less inclined to think it will give your apparent success using the existing form support.

Regards;
Richard Moss

Offline Thebasser123

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
Re: Did my login succeed?
« Reply #2 on: January 08, 2019, 10:35:47 AM »
It worked! (kinda)

Long story short, I downloaded the entire site because I just couldn't get a partial download to work.
Disabling the log out url was necessary though, which I hadn't been doing either.

Thanks for the reply!

Offline Richard Moss

  • Cyotek Team
  • Administrator
  • Sr. Member
  • *****
  • Posts: 291
  • Karma: +17/-0
    • cyotek.com
Re: Did my login succeed?
« Reply #3 on: January 08, 2019, 04:40:08 PM »
Hello,

Thanks for the follow up, I'm glad that worked. I really should update the documentation, it's pretty poor.

Would you mind sharing a bit more detail on the issues you were having with a "partial download"?

Regards;
Richard Moss

Offline Thebasser123

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
Re: Did my login succeed?
« Reply #4 on: January 09, 2019, 12:26:45 PM »
Sure, I'll elaborate.

First of all, I forgot to mention that I'm using WebCopy version 1.6.

As for the rules, forms and such that I used, I tried a lot of different combinations. I'll outline the one that had the highest chance at success in my view. I'll also include a screenshot of the rules I used (not with the actual urls), since that's easier than explaining them in detail.


So far so good (I thought). But only 9 seconds after clicking "copy", I get the "website copied" prompt, but the folder courses.site.com is empty.

Now I noticed that the first url in the results is not https://courses.site.com/
Instead it's the one from the form (https://sso.othersite.com/blahblah)
Also, after the form is posted, I do see https://courses.site.com/ coming up (instead of https://courses.site.com/course1), and it's being excluded by a rule!
Could this be the problem? If so, how would one include the base url?

Hope this wasn't too long-winded  ;)
Best regards
« Last Edit: January 09, 2019, 01:15:14 PM by Thebasser123 »

Offline Richard Moss

  • Cyotek Team
  • Administrator
  • Sr. Member
  • *****
  • Posts: 291
  • Karma: +17/-0
    • cyotek.com
Re: Did my login succeed?
« Reply #5 on: January 10, 2019, 07:36:56 PM »
Hello,

Thanks for the detailed feedback. It's normal to see the posted form appear first in the results as WebCopy will first GET that URL (in order to get default cookies and also for extracting form values) before it POST 's it using an amalgamation of the extract values and the form defined values.

The root URL will (probably) be getting excluded because of the last rule which states to exclude everything. By default, rules only consider the path and query, e.g. course1 or blahblahblah in your examples. To get it to include the domain as well, you need to set the Use full URI flag on the rule - you should be able to use the Rule Checker to find out which rule it is that halts a given URL.

Regards;
Richard Moss