Can't post login before copy website

lswang · January 11, 2016, 02:58:08 AM

Hi Richard,

It's very nice software better than Teleport. but during my backup of one wordpress website, it seems that it can not login in and keep logined in during the process of copying.
I checked the relating topics in the Forum but problem persists. could you pls help to check?

Target website: http://joesenforex.com
Login page: http://joesenforex.com/wp-login.php
My form is set like this:

Did I set it correct? but it's just not working. you can try with the credentials above. Thank you very much!

and the RAW data of the login page is:
<!DOCTYPE html>
   
   <!--[if !(IE

]>
   <head>
   <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
   <title>Joesen Forex – 外汇交易策略|外汇喊单 &rsaquo; 登录</title>
   <link rel='stylesheet' id='buttons-css' href='http://joesenforex.com/wp-includes/css/buttons.min.css?ver=4.1.1' type='text/css' media='all' />
<link rel='stylesheet' id='open-sans-css' href='//fonts.googleapis.com/css?family=Open+Sans%3A300italic%2C400italic%2C600italic%2C300%2C400%2C600&subset=latin%2Clatin-ext&ver=4.1.1' type='text/css' media='all' />
<link rel='stylesheet' id='dashicons-css' href='http://joesenforex.com/wp-includes/css/dashicons.min.css?ver=4.1.1' type='text/css' media='all' />
<link rel='stylesheet' id='login-css' href='http://joesenforex.com/wp-admin/css/login.min.css?ver=4.1.1' type='text/css' media='all' />
<script type='text/javascript' src='http://joesenforex.com/wp-includes/js/jquery/jquery.js?ver=1.11.1'></script>
<script type='text/javascript' src='http://joesenforex.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1'></script>

<style type="text/css">
html, body { border:0 !important; background:none !important; }
html { background-color:#FFFFFF !important; }
html { background-image:url() !important; }
html { background-repeat:repeat !important; }
@media (max-width: 767px) { html, body { background-size: contain !important; } }
body, body * { font-size:12px !important; }
body, body * { font-family:'Verdana', 'Arial', sans-serif !important; }
div#login { width: 100% !important; max-width:550px !important; }
div#login h1 a { background:url(http://joesenforex.com/wp-content/uploads/2015/03/logo-black.png) no-repeat top center !important; background-size:contain !important; }
div#login h1 a { display:block !important; width:100% !important; height:100px !important; }
div#login form { box-shadow:1px 1px 2px #EEEEEE, -1px -1px 2px #EEEEEE !important; border-radius:5px !important; padding-bottom:16px !important; }
div#login p#nav, div#login p#nav a, div#login p#nav a:hover, div#login p#nav a:active, div#login p#nav a:focus { color:#000000 !important; text-shadow:1px 1px 3px #EEEEEE !important; }
div#login p#backtoblog, div#login p#backtoblog a, div#login p#backtoblog a:hover, div#login p#backtoblog a:active, div#login p#backtoblog a:focus { color:#000000 !important; text-shadow:1px 1px 3px #EEEEEE !important; }
div#login form p { margin:2px 0 16px 0 !important; }
div#login form input[type="text"], div#login form input[type="email"], div#login form input[type="password"], div#login form textarea, div#login form select { margin:0 !important; padding:3px !important; border-radius:3px !important; box-sizing:border-box !important; width:100% !important; background:#FBFBFB repeat scroll 0 0 !important; border:1px solid #E5E5E5 !important; font-size:18px !important; font-weight:normal !important; color:#333333 !important; }
@supports (-moz-appearance: none){ div#login form select { font-size:15px !important; } }
div#login form label { cursor:pointer !important; } div#login form label.ws-plugin--s2member-custom-reg-field-op-l { opacity:0.7 !important; font-size:90% !important; vertical-align:middle !important; }
div#login form input[type="checkbox"], div#login form input[type="radio"] { margin:0 3px 0 0 !important; vertical-align:middle !important; }
div#login form input#ws-plugin--s2member-custom-reg-field-user-pass2[type="password"] { margin-top:5px !important; }
div#login form div.ws-plugin--s2member-custom-reg-field-divider-section { margin:2px 0 16px 0 !important; border:0 !important; height:1px !important; line-height:1px !important; background:#CCCCCC !important; }
div#login form div.ws-plugin--s2member-custom-reg-field-divider-section-title { margin:2px 0 16px 0 !important; border:0 solid #CCCCCC !important; border-width:0 0 1px 0 !important; padding:0 0 10px 0 !important; font-size:110% !important; }
div#login form input[type="submit"], div#login form input[type="submit"]:hover, div#login form input[type="submit"]:active, div#login form input[type="submit"]:focus { color:#666666 !important; text-shadow:2px 2px 5px #EEEEEE !important; border:1px solid #999999 !important; border-radius:3px !important; background:#FBFBFB !important; box-shadow:0 -1px 2px 0 rgba(0,0,0,0.2) inset !important; }
div#login form input[type="submit"]:hover, div#login form input[type="submit"]:active, div#login form input[type="submit"]:focus { color:#000000 !important; text-shadow:2px 2px 5px #CCCCCC !important; border-color:#000000 !important; }
div#login form#registerform p.submit { float:none !important; margin-top:-10px !important; } div#login form#registerform input[type="submit"] { float:none !important; width:100% !important; box-sizing:border-box !important; }
div#login form#lostpasswordform p.submit { float:none !important; } div#login form#lostpasswordform input[type="submit"] { float:none !important; width:100% !important; box-sizing:border-box !important; }
div.ws-plugin--s2member-password-strength { margin-top:3px !important; font-color:#000000 !important; background-color:#EEEEEE !important; padding:3px !important; border-radius:3px !important; } div.ws-plugin--s2member-password-strength-short { background-color:#FFA0A0 !important; } div.ws-plugin--s2member-password-strength-bad { background-color:#FFB78C !important; } div.ws-plugin--s2member-password-strength-good { background-color:#FFEC8B !important; } div.ws-plugin--s2member-password-strength-strong { background-color:#C3FF88 !important; } div.ws-plugin--s2member-password-strength-mismatch { background-color:#D6C1AB !important; }
div#login form#registerform p#reg_passmail { font-style:italic !important; }
div#login form#registerform p#reg_passmail { display:none !important; }
</style>

<meta name='robots' content='noindex,follow' />
   </head>
   <body class="login login-action-login wp-core-ui locale-zh-cn">
   <div id="login">
      <h1><a href="http://joesenforex.com" title="Joesen Forex" tabindex="-1">Joesen Forex – 外汇交易策略|外汇喊单</a></h1>

<form name="loginform" id="loginform" action="http://joesenforex.com/wp-login.php" method="post">
   <p>
      <label for="user_login">用户名<br />
      <input type="text" name="log" id="user_login" class="input" value="" size="20" /></label>
   </p>
   <p>
      <label for="user_pass">密码：<br />
      <input type="password" name="pwd" id="user_pass" class="input" value="" size="20" /></label>
   </p>
      <p class="forgetmenot"><label for="rememberme"><input name="rememberme" type="checkbox" id="rememberme" value="forever" checked='checked' /> 记住我的登录信息</label></p>
   <p class="submit">
      <input type="submit" name="wp-submit" id="wp-submit" class="button button-primary button-large" value="登录" />
      <input type="hidden" name="redirect_to" value="http://joesenforex.com/wp-admin/" />
      <input type="hidden" name="testcookie" value="1" />
   </p>
</form>

<p id="nav">
   <a href="http://joesenforex.com/wp-login.php?action=lostpassword" title="找回密码">忘记密码？</a>
</p>

<script type="text/javascript">
function wp_attempt_focus(){
setTimeout( function(){ try{
d = document.getElementById('user_login');
d.focus();
d.select();
} catch(e){}
}, 200);
}

wp_attempt_focus();
if(typeof wpOnload=='function')wpOnload();
</script>

   <p id="backtoblog"><a href="http://joesenforex.com/" title="不知道自己在哪？">← 回到Joesen Forex – 外汇交易策略|外汇喊单</a></p>

   </div>


   <link rel='stylesheet' id='mailchimp-for-wp-checkbox-css' href='http://joesenforex.com/wp-content/plugins/mailchimp-for-wp/assets/css/checkbox.min.css?ver=2.2.6' type='text/css' media='all' />
<script type='text/javascript' src='http://joesenforex.com/wp-content/plugins/s2member/s2member-o.php?ws_plugin__s2member_js_w_globals=1&qcABC=1&ver=150225-357223310'></script>
   <div class="clear"></div>
   </body>
   </html>

Richard Moss · January 12, 2016, 06:34:32 PM

Hello,

Sorry you're having troubles with WebCopy. I haven't looked into your issue yet, and I can't see the image you've posted, but given that this is a public forum if you did include your login credentials it might be a good idea to change your password ASAP!

I'll post another reply once I've had a chance to look a bit further.

Regards;
Richard Moss

Richard Moss · January 12, 2016, 06:48:11 PM

Hello again,

For the user name and password fields, can you try using

pwd=<somevalue>
log=<somevalue>

Looks like there's a bug in WebCopy's Capture tool - currently it's returning the id of HTML elements, which on your form are user_pass and user_login. However, it needs to be using the name attribute instead.

I'll look into getting that fixed in the next updated, but if you could confirm if using pwd/log works for you, that would be great.

Regards;
Richard Moss

lswang · January 13, 2016, 05:04:07 AM

Hi Moss,

Greetings.
I tried with your recommend using pwd and log, but it's not working. it was posted, but not return right.
this is the snapshot:

Quote from: Richard Moss on January 12, 2016, 06:48:11 PM
Hello again,

For the user name and password fields, can you try using

pwd=<somevalue>
log=<somevalue>

Looks like there's a bug in WebCopy's Capture tool - currently it's returning the id of HTML elements, which on your form are user_pass and user_login. However, it needs to be using the name attribute instead.

I'll look into getting that fixed in the next updated, but if you could confirm if using pwd/log works for you, that would be great.

Regards;
Richard Moss

Richard Moss · January 13, 2016, 06:35:48 AM

Hello,

Found (http status 302) is generally actually the correct response for a login. Usually it means that the login has been processed, and the user is to be redirected to a logged in version of the page - if you check the Response Headers tab there would have been a Location header pointing to the URL to navigate too.

Edit: I just thought I'd clarify, the Test URL tool won't follow the redirect, it just displays all the information. But it looks like it's working with pwd and log (and I've now fixed that bug ready for the next update) so if you do a copy of your website it should log in correctly.

Regards;
Richard Moss

lswang · January 13, 2016, 07:26:54 AM

Thanks for the reply Moss.

Just now I let the Copy work to be continued, and here is some info I found, hope it will help me to find out the way to complete the download of website:
1. When Start with the Form that I prepared, it shows "POSTED" now.

2. When Webcopy finished the work, some pages that required login showed "Remapped", not "Downloaded"

3. If you open this file from local by IE, it will show that you need to login to see this page.

What might be this reason? my settings in the cookies or?
If you don't mind I would like to send you my credentials for this issue, thank you very much.

kind regards,
Louis

Quote from: Richard Moss on January 13, 2016, 06:35:48 AM
Hello,

Found (http status 302) is generally actually the correct response for a login. Usually it means that the login has been processed, and the user is to be redirected to a logged in version of the page - if you check the Response Headers tab there would have been a Location header pointing to the URL to navigate too.

Edit: I just thought I'd clarify, the Test URL tool won't follow the redirect, it just displays all the information. But it looks like it's working with pwd and log (and I've now fixed that bug ready for the next update) so if you do a copy of your website it should log in correctly.

Regards;
Richard Moss

Richard Moss · January 13, 2016, 04:36:28 PM

Hello,

I've said this elsewhere I think, but I'm reluctant to be sent credentials to other peoples systems. Partly because you shouldn't be sharing them in the first place, and partly to protect myself - if I log into your system and it becomes compromised... not going there! Plus you're storing these credentials in a forum and you have no real idea if they are secure or not. I didn't write the software so even I don't know if it has some fatal flaw. There isn't even SSL (yet) to protect them in transmission.

Anyhoo, to address your points. The Remapped status is fine and expected. I guess our documentation is lacking on the subject, so I'll clarify here. Downloaded is the final status for resources such as images, zip files and other content that WebCopy downloads, but doesn't post process. Remapped on the other hand is a status that follows Downloaded and applies to things that WebCopy does process, such as HTML files, CSS etc, as WebCopy needs to re-write these files to modify URL's suitable for offline. So that is entirely normally. (It also only happens after every single file has been downloaded, so if you cancel a crawl, any HTML/CSS will only have that Downloaded status and wouldn't have been post-processed)

In regards to the pages that think you aren't logged in. Silly question perhaps, but are you excluding logout pages via rules? Remember will crawl any GET link it can find, meaning if you have logout links that aren't triggered as the result of posting a form, WebCopy will follow them, and the log out code will then be executed.

Regards;
Richard Moss

lswang · January 14, 2016, 03:45:19 AM

Hi Richard,

Thanks for the reply and explanation regarding the Remapped Status, and I am sorry regarding the request of doing the trouble shooting for me. Based on the "Rules" that you mentioned, I checked and I didn't apply any rule at all.
on the other hand, I found the same page from the NOT Login Remapped local page link, it seems to have the same look during the Form Post test-------they are all blank.

compared with form post test result page:

So I think there still an error in my post step.Could you help to have a look on this step?

Best regards,
Louis

Richard Moss · January 14, 2016, 06:37:24 AM

Scanned is fine as well, it means the content has been downloaded and read to pull out a list of URL's, and is waiting to be Remapped. I should probably document the statuses, or remove them from the UI to simplify things!

Regarding the final screenshot of the post, that's not entirely unexpected. Usually web servers include a tiny bit of HTML which essentially provides a link to the resource in the event the Location header isn't followed. (However, WebCopy does not download the body for certain status codes as there isn't any point - 302 is probably one of those, I can't recall off the top of my head)

Check the Headers tab - does it include a Location header which points to another page indicating the log in was successful ?

Check the Cookies tab - are any values listed that indicate the log in was successful?

Sorry if the answers are slightly vague as I can't see what you're seeing unless I actually have valid log in credentials.

Regards;
Richard Moss

lswang · January 14, 2016, 07:06:48 AM

Hi Richard,

Sorry keep you busy on this issue, but I really hope to have this website to learn the FX knowledge all the time.
after checked the headers response and cookies, the info is as below:

I would like to send the credential for test purpose, and any issue that might caused due to the share of this credential will be burden on my side. and will update the password after the process. Can I send via private message or email？ thank you.

best regards,
Louis

Richard Moss · January 14, 2016, 07:44:41 AM

(writing this on a phone so forgive the brevity or any errors)

Logon looks to be working perfectly so must be something else. Will try and have a look as soon as I can. If possible, send credentials to a new user account that has permission to do nothing except view at least one protected page and which you can delete without causing issue.

lswang · January 14, 2016, 10:20:24 AM

Thanks Richard, I have send the view only credential via Forum private message already. Really appreciate your support:)

Richard Moss · January 16, 2016, 11:12:51 AM

Hello,

WebCopy doesn't make it easy to diagnose this type of issue I have to say

(not to mention I can't read Chinese

)

I start scanning the site, checking the cookies after each request. WebCopy is definitely logging you in, and sending the cookie information with each request. What I found however, that if you visit the login page you are automatically logged out. Interesting behaviour, but there you go.

I added a rule to exclude wp-login.php, thus preventing WebCopy from requesting this URL and therefore not trigging an automatic logout (this does not prevent WebCopy from posting the original login via this page as rules have no effect on forms you define for posting). This allowed the scan to progress and I observed that the login cookie was continued to be sent. I didn't allow the scan to complete.

Try excluding wp-login.php and see how you get on.

Hope that solves the issue!

Regards;
Richard Moss

lswang · January 16, 2016, 11:52:45 AM

Hi Richard,

Thank you very much for spending the time with me dealing this issue, especially on weekend. after follow the recommendation that Exclude wp-login.php as key patter word, the process goes smoothly and all the login required pages are now accessible during download.

I really appreciate your kind support and guidance, Wish you a good luck and wonderful year!

Best regards,
Louis

Richard Moss · January 16, 2016, 12:10:11 PM

You're more than welcome. Makes a change actually being able to solve a problem on this forum rather than just saying "Yes, it's a bug in WebCopy I need to fix" or "No, WebCopy can't do that".

Best wishes to yourself also!

Regards;
Richard Moss

PS: Don't forgot to change that password, just in case