ProxHTTPSProxyMII: Development

[whenever]

(Oct. 16, 2014 03:09 AM)GunGunGun Wrote:  I know that Privoxy or Proxomitron can block url but if I remember corrrectly, both software can only see domain of the url and only block at domain level, cannot do anything with path level...

That's only true if you use Proxomitron to proxy https and ask it not to filter https. If you ask it to filter https, or use ProxHTTPSProxy to convert https to http, Proxomitron could see full url with the path part, and could block it.

BTW, about the memory leak, I think it is occupied by the connection pools urllib3 caches for reusing. I had reduced the number of pools to cache and will watch how it affects the memory usage.

[GunGunGun]

(Oct. 16, 2014 07:35 AM)whenever Wrote:  That's only true if you use Proxomitron to proxy https and ask it not to filter https. If you ask it to filter https, or use ProxHTTPSProxy to convert https to http, Proxomitron could see full url with the path part, and could block it.

Thank you! And the http content that send to Proxomitron is only its url or it also contains reponse body ?

And I think raise value of timeout from 10.0 to 115.0 seem better, I sometimes got timeout when I was watching video from Youtube because timeout value is too low. Just in my opinion.

[whenever]

Proxomitron sees everything of the http.

I am not sure if 115.0 is good as the global value for every sites. I will still use 10.0 and you can adjust it as you wish.

Version 1.1b will reload the config when it detects change of the modify time of the file.

[GunGunGun]

(Sep. 04, 2014 09:43 AM)whenever Wrote:  However, you can do it manually if you decide to trust it. Major browsers have a "view certificate" function where you can export the CA to a file then you can append the content to ProxHTTPSProxyMII's cacert.pem file.

Hi whenever, I've tried export certificate from my Firefox to cacert.pem but never make this task work just one time..

Example this page: https://eu.static.mega.co.nz/
This is not a valid page of Mega.co.nz but it sometime load mega.co.nz content like js, css.. so sometime it make this page unloadable because certificate problem.

Code:

Error response

Error code: 417

Message: Exception <class 'urllib3.exceptions.SSLError'>.

Error code explanation: 417 - [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:598).

I added my cert file export from this page to cacert.pem but still unloadable, here is my step by step:
- Load the page and see the error
- View page info -> Security -> View Cert -> Details -> click to ~*.static.mega.co.nz and Export it to X509 Format and then open it with Notepad and copy all content to cacert.pem. Reload page but nothing happen..

Here is my cert:

Code:

# eu.mega.co.nz
-----BEGIN CERTIFICATE-----
MIIC7TCCAdUCBgzbvcQr/jANBgkqhkiG9w0BAQUFADBWMQswCQYDVQQGEwJDTjEX
MBUGA1UEChMOUHJveEhUVFBTUHJveHkxEjAQBgNVBAsTCXB5T3BlblNTTDEaMBgG
A1UEAxMRUHJveEhUVFBTUHJveHkgQ0EwHhcNMTQxMDIwMDgwMTUyWhcNMjQxMDE5
MDgwMTUyWjAeMRwwGgYDVQQDDBMqLnN0YXRpYy5tZWdhLmNvLm56MIIBIjANBgkq
hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7ubVpQYcO5CeGqr/lHPwskwzWLxOUlee
L4yae1cFNZlav1VK3JkrDckq6breH3NoyGLPfFx9CFkZxFUqm27VYLWCmzDg0Rdg
07XbXxq8Y7z1tQiWUxj1fK5Z9XUk6dWTzqfZ93x10UY0ch0zhcPoLND6K8UEhB2T
bS0H6Vw/qLuTy9eIRN3/5zttu3hbA0fUCRqNeFOtdFPtE9WzF/+5nNIGy4l6D6kC
lRagQg9WbKv2Smx39ow5IOcI2iur3xklBMhWnmFp01XlA3Rsu+F0TmQ1S1tZQyTN
oJWeyNLOlnIEa62KAylpAJf1/knhZE0LopKFF1PCVxnlrrpeqeUY0wIDAQABMA0G
CSqGSIb3DQEBBQUAA4IBAQBYXpl5SzPTmVI4EIJV2qmkXJ8M8YbEspd0RtUZ+dxf
7jYRiha1yc2B/lAu92ukS3NZZqusXAhHfsgYQRJbUv3nUiLJWRZRrcZpoUMl1QBo
7dDj8qY5eHFX/ERO/hw0Rb3mP2hKZayXSQ7Sr0av9f0DEjdWfR4+dOudljjFdEHM
P5L8Wd4aTeJxirC14VaygWEnTs2gC5u2tVjsq+BmptmjvsByqb0yYbEKPK3tZUTy
WgURm10ZJ1HTmC+v+HWYf4Bx3M7RhgGPUzOkSz1iC1J8YLboDsStd9dVm5vmAFGV
rcV2J5iKADGZ+kKscJk8S4RhjL++oHT8hli/4k0LU0oM
-----END CERTIFICATE-----

[JJoe]

(Oct. 22, 2014 03:08 AM)GunGunGun Wrote:  Example this page: https://eu.static.mega.co.nz/
This is not a valid page of Mega.co.nz but it sometime load mega.co.nz content like js, css.. so sometime it make this page unloadable because certificate problem.

Only sometimes? They must have multiple servers.

I think it is a cypher mismatch problem. The site's servers and the proxy can't agree which cypher to use. I don't think this is a certificate problem. Google appears to agree, http://www.google.com/search?num=100&saf...KE_FAILURE .

Tests ran by http://www.ssllabs.com show the servers using old protocols and cypher suites.

I'm attaching a screen shot of a results page, which the site's terms seem to allow. The tests take more time and energy, if not cached.

Test's urls were
https://www.ssllabs.com/ssltest/analyze....Results=on
https://www.ssllabs.com/ssltest/analyze....Results=on

Edit: Updating Python did not help.

[whenever]

Maybe you can put that host name under [SSL Pass-Thru] section as a work around.

[GunGunGun]

(Oct. 22, 2014 07:20 AM)whenever Wrote:  Maybe you can put that host name under [SSL Pass-Thru] section as a work around.

Hi whenever, is there a way to generate cacert.pem from source like Firefox or Chrome/Opera ?

Have you tried AdGuard ? I installed this software to my virtual machine, installed Everything to search related file, I try to find how its man in the middle work but I cannot find a cert.pem, seem it use an other method is something like this:



Clearly no file that reach 200KB like cacert.pem, how ? I doubt... Maybe method pvk+cer+AdGuard CA.cer can do the job! Maybe but I'm not really sure.

Here is some key that I found, hope you can analyze something from this file:
https://app.box.com/s/izri97yke6l6ct4xpufv
https://app.box.com/s/axipz0si5p4thd7ild31

And here is certutil folder.


Seem AdGuard don't have mis-certificate problem, this software can load eu.static.mega.co.nz really fine.

Okay, someone have the same problem, maybe bad design of mega causes that: http://forum.adguard.com/showthread.php?...ls-to-load
I found some related informations: http://netfiltersdk.com/help/ProtocolFilters/FT_SSL.htm

[whenever]

(Oct. 22, 2014 03:05 PM)GunGunGun Wrote:  Hi whenever, is there a way to generate cacert.pem from source like Firefox or Chrome/Opera ?

The cacert.pem ProxHTTPSProxy currently uses is from Firefox. I think python's SSL certificate verify routine is not complete yet. That's why some sites work under Firefox but doesn't work with ProxHTTPSProxy, though they use the same CA store.

(Oct. 22, 2014 03:05 PM)GunGunGun Wrote:  Clearly no file that reach 200KB like cacert.pem, how ? I doubt...

It can use the windows default CA store.

Python has some problem with that, see http://bugs.python.org/issue20916

(Oct. 22, 2014 03:05 PM)GunGunGun Wrote:  Seem AdGuard don't have mis-certificate problem, this software can load eu.static.mega.co.nz really fine.

If you change the line from

Code:

sslparams = dict(cert_reqs="REQUIRED", ca_certs="cacert.pem", ssl_version="TLSv1")

to

Code:

sslparams = dict(cert_reqs="REQUIRED", ca_certs="cacert.pem", ssl_version="SSLv2")

And add COMODO SSL CA to cacert.pem, you can load it well.

As JJoe pointed out, the server is using old protocols and cypher suites. I don't want to low down ProxHTTPSProxy's security level to be compatible with it.

[herbalist]

(Oct. 07, 2014 01:14 AM)JJoe Wrote:  

(Oct. 05, 2014 03:30 PM)herbalist Wrote:  Would it be possible to add support for Socks5 to the server facing side so that it could be used with Tor?

config.ini contains

Quote:# Socks proxy support
# https://github.com/shazow/urllib3/pull/284

Is this contained in the 1.0 zip version or something that has to be added to the python version? If it is part of the precompiled executable, then I'm not getting the config.ini entry correct. The page gives:
ProxyManager("socks5://localhost:9050")
The port number is corrected for use with Tor.
Apparently I'm not putting this in the correct location.

[JJoe]

(Oct. 26, 2014 04:13 AM)herbalist Wrote:  Apparently I'm not putting this in the correct location.

I posted the comment to show that socks is not yet supported.

From bottom of https://github.com/shazow/urllib3/pull/284 :

Anorov Jun 7, 2014 Wrote:Sorry for basically going AWOL on this. I've been really busy with school and work in the past few months.

Basically I just need to hunker down, write the full test suite for this while taking into consideration some of the changes to the dummy servers, then re-apply something close to the commits I already have here.

Sorry for the confusion.

[Quaraxkad]

I've got ProxHTTPSProxyMII 1.1 working alongside Privoxy, using instructions I found in this thread. I've still got some issues to work out for a seamless experience.

I'm getting a bunch of errors in the output that pop up seemingly out of nowhere. I have only Opera 12 pointed at the MII port, I'm not browsing any web pages (HTTPS or otherwise), and the timestamps on the errors don't match up to anything in a packet sniffer showing network traffic. I haven't got a clue what the messages even mean (as they are all followed by gibberish) so I don't even know where to begin. I like to fix things myself but the console output is not particularly helpful to an end user... I don't know if it *thinks* that there are HTTPS requests being made or what.
It's a ton of these:
code 400, message Bad HTTP/0.9 request type
code 400, message Bad request syntax
code 400, message Bad request version
I get those lines when it's just sitting idle, and also lots more of them when it actually successfully loads an HTTPS page.

EDIT: I should have mentioned that the random errors only show up when Opera is open, so it is obviously coming from Opera but the nature of the logs makes it impossible to tell what Opera is doing so I can fix it.

Also it's beeping! Out loud, from my speakers... It seems that random console output lines cause it to beep. I skimmed through the code and didn't see any references to sound, but I don't know Python.

[whenever]

Could you please provide the full console output?

[GunGunGun]

(Nov. 09, 2014 05:09 AM)Quaraxkad Wrote:  I've got ProxHTTPSProxyMII 1.1 working alongside Privoxy, using instructions I found in this thread. I've still got some issues to work out for a seamless experience.

I'm getting a bunch of errors in the output that pop up seemingly out of nowhere. I have only Opera 12 pointed at the MII port, I'm not browsing any web pages (HTTPS or otherwise), and the timestamps on the errors don't match up to anything in a packet sniffer showing network traffic. I haven't got a clue what the messages even mean (as they are all followed by gibberish) so I don't even know where to begin. I like to fix things myself but the console output is not particularly helpful to an end user... I don't know if it *thinks* that there are HTTPS requests being made or what.
It's a ton of these:
code 400, message Bad HTTP/0.9 request type
code 400, message Bad request syntax
code 400, message Bad request version
I get those lines when it's just sitting idle, and also lots more of them when it actually successfully loads an HTTPS page.

EDIT: I should have mentioned that the random errors only show up when Opera is open, so it is obviously coming from Opera but the nature of the logs makes it impossible to tell what Opera is doing so I can fix it.

Also it's beeping! Out loud, from my speakers... It seems that random console output lines cause it to beep. I skimmed through the code and didn't see any references to sound, but I don't know Python.

A user made a guide how to make ProxHTTPSProxy work with Privoxy, you can read here, I tried and seem really work: https://sourceforge.net/p/ijbswa/support-requests/1667/

So lucky, I also found that user Privoxy bundle here, very nice: http://www.ghacks.net/2014/11/06/what-we...nt-2888841

[Quaraxkad]

(Nov. 09, 2014 08:26 AM)whenever Wrote:  Could you please provide the full console output?

Here's something, I just launched it, sat and waited for something to happen:

Code:

C:\Programs\Privoxy\ProxHTTPS>C:\Programs\Python\python.exe ProxHTTPSProxy.py
============================================================================
ProxHTTPSProxyMII 1.1 (urllib3/1.9.1)

  FrontServer  : localhost:8079
  RearServer   : localhost:8081
  ParentServer : None
  Proxomitron  : http://localhost:8118
============================================================================
127.0.0.1 - - [2014-11-09 11:41:39] code 400, message Bad request syntax ('\x16\x03\x01\x00\x83\x01\x00\x00\x7f\x03\x01T_\x99C\xc0y\\)\x85\x0e\xb6æ\x0c\xb44Ä\t\x9e9PÅ\xfeL\xa
f\xd4\x1e,\x8d\x96\xfd\x87k \x9bNq\xb6\xc0\xca\x15\x8d\x11\xb3J]6\xd8Wd#\x8b-é¡\x0b\x0e \xc0\x06Ñ\x04n\x1a\x7f2\x008\x00ÿ\x009\x00k\x00j\x00i\x00h\x00=\x008\x007\x006\x005\x00
g\x00@\x00?\x00>\x00<\x003\x002\x001\x000\x00/\x00\x05\x00\x04\x00\x13\x00\r\x00\x16\x00\x10\x00')
127.0.0.1 - - [2014-11-09 11:41:39] "▬♥☺ \x83☺  âŒ‚♥☺T_\x99C\xc0y\)\x85♫\xb6æ♀\xb44Ä     \x9e9PÃ…\xfeL\xaf\xd4â–²,\x8d\x96\xfd\x87k \x9bNq\xb6\xc0\xca§\x8dâ—„\xb3J]6\xd8Wd#\x8b-é¡♂♫
▬ ► " 400 - 8 ÿ 9 k j i h = 8 7 6 5 g @ ? > < 3 2 1 0 / ♣ ♦ ‼
127.0.0.1 - - [2014-11-09 11:42:44] code 400, message Bad HTTP/0.9 request type ('\x16\x03\x01\x00c\x01\x00\x00_\x03\x01T_\x99\x84\x94\x1a\x7f\xa6\x00\x8a\x9c\xd2\x93èfKZ&\x8
9?\xc2\xddÉZ°É\xd9\x19\xb4û\xcd\x11\x00\x008\x00ÿ\x00k\x00j\x00i\x00h\x00=\x009\x008\x007\x006\x005\x00g\x00@\x00?\x00>\x00<\x003\x002\x001\x000\x00/\x00\x05\x00\x04\x00\x13\x
00')
127.0.0.1 - - [2014-11-09 11:42:44] "▬♥☺ c☺  _♥☺T_\x99\x84\x94→⌂\xa6 \x8a\x9c\xd2\x93èfKZ&\x89?\xc2\xddÉZ°É\xd9↓\xb4û\xcdâ—„  8 ÿ k j i h = 9 8 7 6 5 g @ ? > < 3 2 1 0 / ♣ ♦ ‼
â–¬ â–º " 400 -
127.0.0.1 - - [2014-11-09 11:43:35] code 400, message Bad request version ('\x00\x16\x00\x10\x00')
â–¬ â–º " 400 -- [2014-11-09 11:43:35] "▬♥☺ c☺  _♥☺T_\x99·l\xcde\x9c\x90z§j▼☻°♣Z\x8cU½\æ]X↓\xd0\x9c/¡§T·  8 ÿ k j i h = 9 8 7 6 5 g @ ? > < 3 2 1 0 / ♣ ♦ ‼
127.0.0.1 - - [2014-11-09 11:44:39] code 400, message Bad request syntax ('\x16\x03\x01\x00c\x01\x00\x00_\x03\x01T_\x99÷\x05-"Eä\x1d\x06\xdb\x92-\xca\xa0\x88\xdb2uß[*#8]J$\x0
f\xd3\xfd\xc2\x00\x008\x00ÿ\x00k\x00j\x00i\x00h\x00=\x009\x008\x007\x006\x005\x00g\x00@\x00?\x00>\x00<\x003\x002\x001\x000\x00/\x00\x05\x00\x04\x00\x13\x00\r\x00\x16\x00\x10\x
00')
â–¬ â–º " 400 -- [2014-11-09 11:44:39] "▬♥☺ c☺  _♥☺T_\x99÷♣-"Eä↔♠\xdb\x92-\xca \x88\xdb2uß[*#8]J$☼\xd3\xfd\xc2  8 ÿ k j i h = 9 8 7 6 5 g @ ? > < 3 2 1 0 / ♣ ♦ ‼
[11:45:09] [D] "POST https://lastpass.com/poll_server.php HTTP/1.1" 200 134
127.0.0.1 - - [2014-11-09 11:45:29] code 400, message Bad request syntax ("\x16\x03\x01\x00c\x01\x00\x00_\x03\x01T_\x9a)\xc21\x9a\x81zá'Æ\xe3²¥\x17\x0b&\xd2%=/\x1f \xa66\xd8á
\x1f\x07¼\x83\x00\x008\x00ÿ\x00k\x00j\x00i\x00h\x00=\x009\x008\x007\x006\x005\x00g\x00@\x00?\x00>\x00<\x003\x002\x001\x000\x00/\x00\x05\x00\x04\x00\x13\x00\r\x00\x16\x00\x10\x
00")
â–¬ â–º " 400 -- [2014-11-09 11:45:29] "▬♥☺ c☺  _♥☺T_\x9a)\xc21\x9a\x81zá'Æ\xe3²¥↨♂&\xd2%=/â–¼ \xa66\xd8á▼¼\x83  8 ÿ k j i h = 9 8 7 6 5 g @ ? > < 3 2 1 0 / ♣ ♦ ‼

It did beep here, right as the last two lines came up.

[JJoe]

(Nov. 09, 2014 04:50 PM)Quaraxkad Wrote:  Here's something, I just launched it, sat and waited for something to happen:
...
It did beep here, right as the last two lines came up.

These errors appear to be related to the LastPass extension for Opera. I don't know how it works.

Do errors persist after disabling the LastPass extension?
Can you test another browser?

I think the beeps are from printing "\xa0" in the console window.