[SOLVED] Web - secure downloads

[devwithoutpower]

Originally Posted by Kastoi

Hi,

perhaps a stupid question but... Is that "patch" linked to the browser or not? I mean, is that possible to apply it to advanced browser and not to the standard browser ?

Thanks !

It is linked, but it is also possible to patch advanced browser (haven't tried myself, but it *should* be possible due to nature of patch) ... eventually I could add support for Advanced Browser into install.sh script (and uninstall) ... but for v0.0.5 current priority is to add support for login/password prompt so it will be possible to download files using 'http://<username>:<password>@server/rest/of/the/url' ... which should cover pages that don't use persistent cookies, but http authentication instead

[touchpadmd1203]

Originally Posted by devwithoutpower

It is linked, but it is also possible to patch advanced browser (haven't tried myself, but it *should* be possible due to nature of patch) ... eventually I could add support for Advanced Browser into install.sh script (and uninstall) ... but for v0.0.5 current priority is to add support for login/password prompt so it will be possible to download files using 'http://<username>:<password>@server/rest/of/the/url' ... which should cover pages that don't use persistent cookies, but http authentication instead

I double click on file in InternalPro to "install" it. Am I doing it wrong? Here is my script:

[ -f /etc/palm-build-info ] || exit 1
echo remount / as rw
mount -o rw,remount /
echo Part I
CONF=/etc/jail_cookies.conf
cp /etc/jail_hybrid.conf $CONF
echo "mkdir /var/palm" >> $CONF
echo "mkdir /var/palm/data" >> $CONF
echo "mount ro /var/palm/data" >> $CONF
echo Part II
[ -f /usr/palm/applications/com.palm.app.browser/source/BrowserApp.js.orig ] || cp /usr/palm/applications/com.palm.app.browser/source/BrowserApp.js /usr/palm/applications/com.palm.app.browser/source/BrowserApp.js.orig
cp BrowserApp.js /usr/palm/applications/com.palm.app.browser/source/BrowserApp.js
echo remount / as ro
mount -o ro,remount /
echo Part III
ipkg remove -o /media/cryptofs/apps xx.downloader
ipkg install -o /media/cryptofs/apps xx.downloader_0.0.4_all.ipk
echo Done

Still no luck..but I think the https username/password issue is what was preventing me from logging on to my school website....thanks for working on this.


I tried installing IPK directly and when it opened it read "Target URL missing".

[devwithoutpower]

Originally Posted by touchpadmd1203

I double click on file in InternalPro to "install" it. Am I doing it wrong?
...
I tried installing IPK directly and when it opened it read "Target URL missing".

I am not sure if double clicking on script file in InternalPro executes it. I recommend using xterm or similar terminal application.

Here is example log:

root@TP:/media/internal# unzip downloader-0.0.4.zip 
Archive:  downloader-0.0.4.zip
   creating: downloader/
  inflating: downloader/install.sh
  inflating: downloader/BrowserApp.js
  inflating: downloader/xx.downloader_0.0.4_all.ipk
  inflating: downloader/uninstall.sh
root@TP:/media/internal# cd downloader/
root@TP:/media/internal/downloader# ./install.sh 
remount / as rw
Part I
Part II
remount / as ro
Part III
Removing package xx.downloader from root...
Begin installation of xx.downloader
Installing xx.downloader (0.0.4) to root...
Configuring xx.downloader
Done
root@TP:/media/internal/downloader#

"Target URL missing" is just a feedback message - you are not supposed to launch Downloader directly at the moment ... in future versions there will be some extra functionality available when launched directly; but not yet.

[devwithoutpower]

Originally Posted by knownboyofno

devwithoutpower
Could you make the program prompt the user to enter username and password? If the cookie isn't found. This could solve the problem with some sites.

v0.0.5 allows you to enter U+P .. you have 5 seconds to press that button ... then download starts automatically

[touchpadmd1203]

Originally Posted by devwithoutpower

I am not sure if double clicking on script file in InternalPro executes it. I recommend using xterm or similar terminal application.

Here is example log:

root@TP:/media/internal# unzip downloader-0.0.4.zip 
Archive:  downloader-0.0.4.zip
   creating: downloader/
  inflating: downloader/install.sh
  inflating: downloader/BrowserApp.js
  inflating: downloader/xx.downloader_0.0.4_all.ipk
  inflating: downloader/uninstall.sh
root@TP:/media/internal# cd downloader/
root@TP:/media/internal/downloader# ./install.sh 
remount / as rw
Part I
Part II
remount / as ro
Part III
Removing package xx.downloader from root...
Begin installation of xx.downloader
Installing xx.downloader (0.0.4) to root...
Configuring xx.downloader
Done
root@TP:/media/internal/downloader#

"Target URL missing" is just a feedback message - you are not supposed to launch Downloader directly at the moment ... in future versions there will be some extra functionality available when launched directly; but not yet.

Got it!

Will do today and update you. Thanks so much.

MB

[dconeg]

devwithoutpower,
I don't have administrative access (I am a student) therefore I cannot test persistent cookies; i also found no alternative after logging into blackboard to change this setting.

I also updated to version .0.0.5 of your patch and still no luck. Even when entering username and pass, i still get {"error" : "script error"}

[devwithoutpower]

PM me with url you are trying to download from (no username and password, just full url to target file)... i will send you some diagnostic commands that you can run from commandline...

[touchpadmd1203]

GETTING CLOSER!!

DOWNLOADER LAUNCHED BUT THE USERNAME AND PWORD THAT COMES UP..is that for the site or the one I have for my touchpad. If I bypass it then pdf just spins and document does not open...


Now I can get to some documents but most stuff open up within the frame and it doesnt prompt me to download. I am using carmen.osu.edu.

EXAMPLE:
https://carmen.osu.edu/d2l/lms/conte...58&tId=4183090

[devwithoutpower]

Originally Posted by touchpadmd1203

...
DOWNLOADER LAUNCHED BUT THE USERNAME AND PWORD THAT COMES UP..is that for the site or the one I have for my touchpad. If I bypass it then pdf just spins and document does not open...

This button (and dialog that it opens) is optional - for use cases where cookies are not used ... is simply modifies target url, eg. if it is originally http: // whatever.com/some/random/path/file.pdf ... it will become http:// username : password@whatever.com/some/random/path/file.pdf - so you enter credentials for that particular page ... btw. in later versions Downloader will remember what pages need that info (and will offer option to keep any of server/login/password so it does not even ask you next time)

Originally Posted by touchpadmd1203

EXAMPLE:
https://carmen.osu.edu/d2l/lms/conte...58&tId=4183090

This looks to me like 3rd (and from our point of view the most complicated) way of protecting content - where 'Referer' is also used ... it means that to download from such site you need to have session cookies stored ... it is possible to add support for this type as well ... but I need somehow get access to such pages so I can do some proper testing

[devwithoutpower]

Originally Posted by eblade

Sure thing. I'd like to give it a try to use it with some Google services that won't work properly due to needing to download files via https, as well as I could definitely see someone writing clients for this chalkboard site or whatever it is that everyone is having problems with...

v0.0.6 is available for testing ... it has simple callback functionality ... ie you launch Downloader with caller id ... and Downloader will send (via 'launch' method) response. See first post for example code...

[devwithoutpower]

Originally Posted by dconeg

devwithoutpower,
I don't have administrative access (I am a student) therefore I cannot test persistent cookies; i also found no alternative after logging into blackboard to change this setting.

I also updated to version .0.0.5 of your patch and still no luck. Even when entering username and pass, i still get {"error" : "script error"}

Try v0.0.6 - I've fixed url handling that could fix 'script error' issue for you.

[dconeg]

v0.0.6 still not working with blackboard, but different error message this time.
Here is what the downloader app reads:

Target:
http://(username) : (password)@blackboard.bu.edu:80/courses/1/11fallengbe401_a1/content/_1081280_1/Project3.pdf?bsession=10741832&bsession_str=session_id=10741832,user_id_pk1=458718,user_id_sos_id_pk 2=1,one_time_token.... (gets cut off)

Warning: Failed to create the file Warning: Project3.pdf?
bsession=10741832&bsession_str=session_id=10741832,user_i Warning:
d_pk1=458718,user_id_sos_id_pk2=1,one_time_token=: Invalid argument curl: (23) Failed writing body (0 != 1402)

[Kastoi]

Well, in finally tried... (waited some new releases )

And it works in my case ! Thanks a lot DWP

I installed 0.06 with Xterm/Xecutah
and the website that wasn't working was : ICHEC Campus and iCampus (which are using the same platform actually: "claroline" Claroline . NET - Home )

It is the equivalent in Belgium of your blackboard i think

[devwithoutpower]

Originally Posted by dconeg

v0.0.6 still not working with blackboard, but different error message this time.
Here is what the downloader app reads:

Target:
http://(username) : (password)@blackboard.bu.edu:80/courses/1/11fallengbe401_a1/content/_1081280_1/Project3.pdf?bsession=10741832&bsession_str=session_id=10741832,user_id_pk1=458718,user_id_sos_id_pk 2=1,one_time_token.... (gets cut off)

Warning: Failed to create the file Warning: Project3.pdf?
bsession=10741832&bsession_str=session_id=10741832,user_i Warning:
d_pk1=458718,user_id_sos_id_pk2=1,one_time_token=: Invalid argument curl: (23) Failed writing body (0 != 1402)

Looking at bu.edu pages (and source code) it looks to me that only solution is 2-steps download process ... 1. weblogin 2.download - with session cookies passed from 1 -> 2 ... this is doable, but requires 'customised' code that will only work with this single server and only until html form (for weblogin) is unchanged ...

I will add support for this kind of pages - BUT I will only provide generic template - maybe with some how-to add your own server (or any other server that requires login first).

If someone has test account or knows about some public server where I can register - please send me so I have more testing places...

[touchpadmd1203]

Thanks for all your work on this devwithoutpower. I will try v0.6. How do I uninstall v0.5 ...via Webos or Xterm. I had a hard time figuring out the command for "unsintall.sh" to do its function. Thanks again for your work on this. I am so surprised it hasnt received more attention.

[devwithoutpower]

Originally Posted by touchpadmd1203

Thanks for all your work on this devwithoutpower. I will try v0.6. How do I uninstall v0.5 ...via Webos or Xterm. I had a hard time figuring out the command for "unsintall.sh" to do its function. Thanks again for your work on this. I am so surprised it hasnt received more attention.

You don't really need uninstall if you want to update 0.5 -> 0.6 ... install.sh takes care of it. But if you really want - then just run uninstall.sh the same way how you would run install.sh ... check few posts above for example (at the time when v0.0.4 was the latest)

[touchpadmd1203]

Originally Posted by devwithoutpower

You don't really need uninstall if you want to update 0.5 -> 0.6 ... install.sh takes care of it. But if you really want - then just run uninstall.sh the same way how you would run install.sh ... check few posts above for example (at the time when v0.0.4 was the latest)

Thanks! Got it! Works! It seems like I cannot get access to documents that open in a "frame"..which is 90% of the content. At one point there was a right click that allowed me to download...I think that would help in this situation. I will try it out...

[devwithoutpower]

So I spent some time investigating downloads from sites that does not store credentials in persistent cookies ... GOOD NEWS - I was able to download file from command line ... BAD NEWS - it is really complicated and most probably each site would require few trial-error steps ...

For better understanding - this is what was required to download ZIP file attached to this thread (when no persistent cookies are stored on device already)

1. try to download file (will fail because login is required first)
curl http://forums.webosnation.com/attach...ader-0.0.6.zip

This will return HTML page where you have to find POST form ... either manually or with help of little perl script called 'formfind'

2. extract needed form attributes

in this case (output from formfind):

--- FORM report. Uses POST to URL "/login.php?do=login"
Input: NAME="do" VALUE="login" (HIDDEN)
Input: NAME="url" VALUE="/attachment.php?attachmentid=62850" (HIDDEN)
Input: NAME="vb_login_md5password" (HIDDEN)
Input: NAME="vb_login_md5password_utf" (HIDDEN)
Input: NAME="s" VALUE="e9f1fdae0c04f8947856b1464a0b46b0" (HIDDEN)
Input: NAME="securitytoken" VALUE="guest" (HIDDEN)
Input: NAME="vb_login_username" (TEXT)
Input: NAME="vb_login_password" (PASSWORD)
Input: NAME="cookieuser" VALUE="1" (CHECKBOX)
Button: "Log in" (SUBMIT)
--- end of FORM

3. to make it more difficult for automated login (and to improve security by not sending plaintext password) - before form is submitted MD5 sum of password entered is calculated - from HTML obtained in step 1:
<form action="/login.php?do=login" method="post" onsubmit="md5hash(vb_login_password, vb_login_md5password, vb_login_md5password_utf, 0)">
<script type="text/javascript" src="http://cdn-forums.precentral.net/clientscript/vbulletin_md5.js?v=387"></script>

So we need to actually run JavaScript from commandline before we can send login data to server! We could use SpiderMonkey (after doing some quick reverse engineering you will end up with something like this):

var vb_login_password = { value: "your password here" };
var vb_login_md5password = {};
var vb_login_md5password_utf = {};
var hash = md5hash(vb_login_password, vb_login_md5password, vb_login_md5password_utf, 0);
print("vb_login_password="+vb_login_password.value);
print("vb_login_md5password="+vb_login_md5password.value);
print("vb_login_md5password_utf="+vb_login_md5password_utf.value);

Luckily for us this is standard MD5 so we dont really need JS runtime in this particular case.

4. POSTing login data to server

curl -d vb_login_username="devwithoutpower" -d vb_login_password="" -d cookieuser=0 -d securitytoken=guest -d -d vb_login_md5password=ac94d33dd04368c6421f0edb6949ffa6 -d vb_login_md5password_utf=ac94d33dd04368c6421f0edb6949ffa6 -d url=/attachment.php?attachmentid=62850 -d do=login http://forums.webosnation.com/login.php?do=login

Almost there ... you will notice from saved file that this is still no the right one ... as there is redirection used ...

5. getting the real file

curl -L -b bbsessionhash=66a6cc704c20420757ff27f64a1a2274 http://forums.webosnation.com/attach...hmentid=62850& -J -O


Here is complete script that will automate everything - $1 and $2 are parameters passed on commandline - url and password (username is hardcoded):

URL=$1
MD5=`echo -n $2 | md5sum | awk '{ print $1 }'` 

curl $URL > first.out
SEED=`cat first.out | ./formfind | grep -e "^Input: NAME=\"s\" VALUE=" | head -1 | awk -F "\"" '{ print $4 }'`
DST=`cat first.out | ./formfind | grep -e "^Input: NAME=\"url\" VALUE=" | head -1 | awk -F "\"" '{ print $4 }'`

curl -D headers.out -L -d vb_login_username="devwithoutpower" -d vb_login_password="" -d cookieuser=0 -d securitytoken=guest -d s=$SEED -d vb_login_md5password=$MD5 -d vb_login_md5password_utf=$MD5 -d url="$DST" -d do=login http://forums.webosnation.com/login.php?do=login > second.out
rm first.out

U2=`cat second.out | grep -e "URL=" | awk -F "URL=" '{ print $2 }' | awk -F "\"" '{ print $1 }'`
rm second.out

HASH=`cat headers.out | grep bbsessionhash | awk -F "=" '{ print $2 }' | awk -F ";" '{ print $1 }'`
rm headers.out

curl -L -b bbsessionhash=$HASH $U2 -J -O

Simple, isn't it?

Sorry for long post - but I wanted to make sure that you understand that getting secured downloads is always doable, but sometimes very tricky.

And there is almost no chance to make it generic for all different servers / sites...

[Weidbrewer]

Okay, we're getting closer!

Old version uninstalled as easily as it installed, and the new one behaved just as well. Also, I was able to download files from one of my sites with an ip address. Excellent.

Some tweaking things that I experienced, however:

1.) Name field defaults to a capital letter, and the only way to get lower-case is to type it twice and delete the first one. This is a pain with servers that are case-sensitive
2.) On the other site I use, the password has an '@' in it, throwing an error that it cannot resolve <password>@xx.xx.xxx.x...which means the downloader won't work there. (this might be the "special characters" issue mentioned in the OP.)
3.) Would be fantastic if it could save login info.

Almost there - keep up the great work!

[e-gadget-guy]

Just got a chance to revisit this issue. I downloaded and unzipped the file [0.0.6] to my PC, then connected to my TP and copied it over, then used WOSQI to run the unstall, then reset and went to BB to test it... and it worked! [would be nice if it was a patch, or allowed passing password, or gave an option of where to download it] but it works! and that is great!!!