Post Reply 
ProxHTTPSProxyMII: Development
May. 28, 2014, 03:13 AM (This post was last modified: Jun. 17, 2018 02:17 AM by JJoe.)
Post: #1
ProxHTTPSProxyMII: Development
What about this routine? Smile!

[Image: file.php?id=300]


.zip  ProxHTTPSProxyMII_py 1.5.zip (Size: 171.89 KB / Downloads: 828)
Add Thank You Quote this message in a reply
May. 28, 2014, 05:34 AM
Post: #2
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Welcome back stranger. Smile!

(May. 28, 2014 03:13 AM)whenever Wrote:  What about this routine? Smile!

So local ProxHTTPSProxyPartA and ProxHTTPSProxyPartB... Think
I'm guessing ProxHTTPSProxyPartA and ProxHTTPSProxyPartB are using the remote server's cert, as described at http://mitmproxy.org/doc/howmitmproxy.html under "Explicit HTTPS", to hide the Proxomitron. Pray
Is the file at your forum? Drool

For casual browsing, I have been using the 'SSL Certificate CN Always Matches' patch at http://prxbx.com/forums/showthread.php?tid=2156 . Also, using my browser's command line options to disable its cert checks.

I would like to get proper verification back (and maybe a little more) tho. Eventually, I or we will probably need a new proxy or scheme.

Have you noticed? HandyCache users say thank you very much. Wink
Just found a patch at http://rghost.net/51510677 , referenced http://handycache.ru/component/option,co...pic,471.0/ .
Have fun
Add Thank You Quote this message in a reply
May. 28, 2014, 09:39 AM
Post: #3
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2014 05:34 AM)JJoe Wrote:  I'm guessing ProxHTTPSProxyPartA and ProxHTTPSProxyPartB are using the remote server's cert, as described at http://mitmproxy.org/doc/howmitmproxy.html under "Explicit HTTPS", to hide the Proxomitron. Pray

It's just like that. The front server will use a self made CA to generate server certificates on the fly.

(May. 28, 2014 05:34 AM)JJoe Wrote:  I would like to get proper verification back (and maybe a little more) tho. Eventually, I or we will probably need a new proxy or scheme.

The rear server will do the certificates verification. It needs Python version after 3.4.

(May. 28, 2014 05:34 AM)JJoe Wrote:  Have you noticed? [url=http://www.google.com/search?q=ProxHTTPSProxy]HandyCache users say thank you very much. Wink

I hadn't expected that. Smile! In deed, I haven't been using Proxomitron for a long time. I just got some time to learn Python again so I think I would update ProxHTTPSProxy to use a new scheme.

I am still working on the script. At the same time you can prepare for it:
Add Thank You Quote this message in a reply
May. 28, 2014, 07:23 PM
Post: #4
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2014 09:39 AM)whenever Wrote:  I just got some time to learn Python again so I think I would update ProxHTTPSProxy to use a new scheme.

I am still working on the script.

There is a library that may help, http://www.python-requests.org/en/latest/
.

Have fun
Add Thank You Quote this message in a reply
May. 29, 2014, 02:39 AM
Post: #5
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2014 07:23 PM)JJoe Wrote:  There is a library that may help, http://www.python-requests.org/en/latest/

I looked at it before but think it would be too heavy for our such lightweight script.

Here is the version 0.5 for you to play with. http://forum.proxomitron.cn/download/file.php?id=301
  • * Import "CA.crt " to your browser's trusted certificate authorities
    * Point browser's https proxy to front server's port (default 8079)
    * Set Proxomitron's parent proxy to rear server (default 8081) for tagged connections

Exceptions-U.ptxt:
Code:
$OHDR(Tagged:Proxomitron FrontProxy/*)        $SET(0=i_proxy:3.) $SETPROXY(127.0.0.1:8081)

The script should just throw exception for bad server certification. Need to find a server with bad certification for testing.

BTW, I still couldn't upload attachment.
Add Thank You Quote this message in a reply
May. 30, 2014, 07:58 PM (This post was last modified: May. 31, 2014 12:31 AM by JJoe.)
Post: #6
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Ok. Installed and running. CheersApplause

Fresh install of "Python3.4.1" didn't work for me. All is well after an uninstall and reinstall of "3.4.1", tho.
"Win32 OpenSSL v1.0.1g Light" required "Visual C++ 2008 Redistributables".
Don't forget to bypass the Proxomitron before using "pip install pyopenssl" at the Window's command prompt.
Assuming Python34 is installed in the default folder

Code:
c:\Python34\Scripts>pip install pyopenssl

To enable $SETPROXY(127.0.0.1:8081), add the Rear proxy to the Proxomitron via "Proxy" on the Proxomitron's main dialog

Code:
127.0.0.1:8081 ProxHTTPSProxyRear

(More detailed instructions later or per request)

For now, I'm using these Exceptions-U entries

Code:
$OHDR(Tagged:Proxomitron FrontProxy/*) $SET(keyword=$GET(keyword)i_proxy:3.) $SETPROXY(127.0.0.1:8081)(^)
~(^$TST(keyword=i_proxy:[03].))$OHDR(Tagged:Proxomitron FrontProxy/*) $SET(keyword=$GET(keyword)i_proxy:3.) $SETPROXY(127.0.0.1:8081)(^)

in sidki's set. This way the keyword is always set regardless of the other list entries and the other list entries are checked.
When I $RDIR in Exceptions-U to block a connection I'll be using something like

Code:
unwanted.com/ $USEPROXY(false)$SET(keyword=i_proxy:0.)$RDIR($GET(blackhole)/killed.gif?\u)

otherwise the rear proxy will be asked for "killed.gif".

However, the new routine works better with sidki's set in bypass while using the Proxomitron's "Use Remote Proxy" setting. I'm not sure why.
So, I'm also using the Proxomitron's default set with the web filters bypassed and this,

Code:
[HTTP headers]
In = FALSE
Out = TRUE
Key = "tagged: (Out)"
Match = "$SETPROXY(127.0.0.1:8081)(^)"

, header filter to test the routine.

Current issues

1. When searching at startpage.com

Code:
127.0.0.1 - - [30/May/2014 08:42:40] "GET http://s10-us4.startpage.com/tst2/engl
ish/?anticache=989016 HTTP/1.1" 200 -
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 60636)
Traceback (most recent call last):
  File "C:\Python34\lib\socketserver.py", line 609, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python34\lib\socketserver.py", line 344, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python34\lib\socketserver.py", line 665, in __init__
    self.handle()
  File "C:\Python34\lib\http\server.py", line 398, in handle
    self.handle_one_request()
  File "C:\Python34\lib\http\server.py", line 386, in handle_one_request
    method()
  File "C:\Users\E3\Programs\ProxHTTPSProxy 0.5\proxytool.py", line 97, in do_CO
NNECT
    self.handle_one_request()
  File "C:\Python34\lib\http\server.py", line 386, in handle_one_request
    method()
  File "C:\Users\E3\Programs\ProxHTTPSProxy 0.5\ProxHTTPSProxy.py", line 59, in
do_METHOD
    self.wfile.write(response.read())
  File "C:\Python34\lib\http\client.py", line 512, in read
    s = self._safe_read(self.length)
  File "C:\Python34\lib\http\client.py", line 664, in _safe_read
    raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(0 bytes read, 42 more expected)

2. Need to find a server with bad certification for testing.

Got to go.
Add Thank You Quote this message in a reply
May. 31, 2014, 01:27 AM
Post: #7
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
I created an instance of The Proxomitron in a sandbox to test against self signed certs.
So the chain was FrontServer>Proxomitron>RearServer>Sandboxied Proxomitron.

After a request for https://www.google.com, RearServer says, "ssl.CertificateError: hostname 'www.google.com' doesn't match 'Proxomitron'"

Code:
============================================================================
ProxHTTPSProxy 0.5 serving now, <Ctrl-C> to stop ...

  FrontServer  : localhost:8079
  RearServer   : localhost:8081
  ParentServer : 127.0.0.1:8082
  Proxomitron  : localhost:8080
============================================================================
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 50004)
Traceback (most recent call last):
  File "C:\Python34\lib\socketserver.py", line 609, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python34\lib\socketserver.py", line 344, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python34\lib\socketserver.py", line 665, in __init__
    self.handle()
  File "C:\Python34\lib\http\server.py", line 398, in handle
    self.handle_one_request()
  File "C:\Python34\lib\http\server.py", line 386, in handle_one_request
    method()
  File "C:\Users\E3\Programs\ProxHTTPSProxy 0.5\ProxHTTPSProxy.py", line 84, in
do_METHOD
    conn.request(self.command, path, data, self.headers)
  File "C:\Python34\lib\http\client.py", line 1090, in request
    self._send_request(method, url, body, headers)
  File "C:\Python34\lib\http\client.py", line 1128, in _send_request
    self.endheaders(body)
  File "C:\Python34\lib\http\client.py", line 1086, in endheaders
    self._send_output(message_body)
  File "C:\Python34\lib\http\client.py", line 924, in _send_output
    self.send(msg)
  File "C:\Python34\lib\http\client.py", line 859, in send
    self.connect()
  File "C:\Python34\lib\http\client.py", line 1230, in connect
    server_hostname=sni_hostname)
  File "C:\Python34\lib\ssl.py", line 364, in wrap_socket
    _context=self)
  File "C:\Python34\lib\ssl.py", line 578, in __init__
    self.do_handshake()
  File "C:\Python34\lib\ssl.py", line 813, in do_handshake
    match_hostname(self.getpeercert(), self.server_hostname)
  File "C:\Python34\lib\ssl.py", line 288, in match_hostname
    % (hostname, dnsnames[0]))
ssl.CertificateError: hostname 'www.google.com' doesn't match 'Proxomitron'
----------------------------------------
Add Thank You Quote this message in a reply
May. 31, 2014, 06:29 AM
Post: #8
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Using a header filter to apply the parent proxy seems to be the best way without impacting the Exceptions-U.ptxt entries.

Code:
http.client.IncompleteRead: IncompleteRead(0 bytes read, 42 more expected)

This was caused by the server setting "Content-Length" to 42 bytes but sending less bytes. Seems like a kind of anti caching technology.

Test URL:
Code:
https://s7-us4.startpage.com/cgi-bin/ccspacer?ns=1&anticache=15007&/filename.gif

Result without proxying:

.png  anti cache.png (Size: 24.64 KB / Downloads: 930)

Version 0.6 suppresses this kind of error and adds warning for certificate error.

Test site: https://kyfw.12306.cn/


Attached File(s)
.zip  ProxHTTPSProxy 0.6.zip (Size: 1.91 KB / Downloads: 749)
Add Thank You Quote this message in a reply
May. 31, 2014, 05:14 PM
Post: #9
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 31, 2014 06:29 AM)whenever Wrote:  Using a header filter to apply the parent proxy seems to be the best way without impacting the Exceptions-U.ptxt entries.

However, I think using either will cause forwarding to be disabled when using the Proxomitron in Bypass mode.

I'll probably use an undocumented 'feature'. Entries in the "Bypass any URLs that match this expression" field are checked when the Proxomitron is in Bypass mode.
So Bypass list entries like

Code:
$OHDR(Tagged:Proxomitron FrontProxy/*) $SETPROXY(127.0.0.1:8081)(^)
~$OHDR(Tagged:Proxomitron FrontProxy/*) $SETPROXY(127.0.0.1:8081)(^)

are (or "should be", since I haven't actually done this yet) executed when the Proxomitron is in Bypass mode. $SETPROXY is executed when found but (^) never matches and prevents a bypass.

(May. 31, 2014 06:29 AM)whenever Wrote:  Version 0.6 suppresses this kind of error and adds warning for certificate error.

Version 0.6 is working better. Also works much better with web page filtering enabled.

Current Issue: False alarm (I hope) at yahoo login.

Code:
WARNING:    [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)

at https://login.yahoo.com/config/login_verify2?&.src=ym
Add Thank You Quote this message in a reply
Jun. 01, 2014, 01:35 PM
Post: #10
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 31, 2014 05:14 PM)JJoe Wrote:  Current Issue: False alarm (I hope) at yahoo login.

Code:
WARNING:    [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)

at https://login.yahoo.com/config/login_verify2?&.src=ym

Same error at https://www.yahoo.com
Add Thank You Quote this message in a reply
Jun. 02, 2014, 07:05 AM (This post was last modified: Jun. 02, 2014 07:06 AM by whenever.)
Post: #11
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
I didn't get that on either https://login.yahoo.com/config/login_verify2?&.src=ym or https://www.yahoo.com.

Could you reproduce that with or without ProxHTTPSProxy?

I got below IPs when I ping them:

Code:
login.yahoo.com -> 98.139.21.169
www.yahoo.com -> 202.43.192.109

What if you using a Hosts file to resolve to those IPs and trying again?
Add Thank You Quote this message in a reply
Jun. 02, 2014, 03:40 PM (This post was last modified: Jun. 02, 2014 05:12 PM by JJoe.)
Post: #12
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 02, 2014 07:05 AM)whenever Wrote:  Could you reproduce that with or without ProxHTTPSProxy?

Verification works without ProxHTTPSProxy.

(Jun. 02, 2014 07:05 AM)whenever Wrote:  What if you using a Hosts file to resolve to those IPs and trying again?

Same result at your IPs. I see

Code:
login.yahoo.com -> 98.138.79.21
www.yahoo.com -> 98.139.180.149

Is ProxHTTPSProxy using Python's certificate store?
After reading some of the Python documents and ProxHTTPSProxy's code, I assumed it was and that the problem might be a missing cert. Tried replacing the store, same result. I might have replaced the wrong file tho...

(Jun. 02, 2014 07:05 AM)whenever Wrote:  I didn't get that

Now I'm not so sure that it is the store.
Are you using Python 3.4.1 from https://www.python.org/downloads/ ?
Which operating system?

I'm also seeing same error at https://www.verisigninc.com/ , expected since yahoo uses verisign.

I'll do more reading later.

Edit: added a question and more info.
Add Thank You Quote this message in a reply
Jun. 02, 2014, 06:22 PM
Post: #13
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Which OpenSSL package, http://slproweb.com/products/Win32OpenSSL.html , did you use?
Add Thank You Quote this message in a reply
Jun. 03, 2014, 01:26 AM (This post was last modified: Jun. 03, 2014 01:27 AM by whenever.)
Post: #14
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
This might be the problem: newer Windows has only a very minimal set of root CA certificates shipped by default. See http://bugs.python.org/issue20916

Version 0.6a uses CA bundle from http://curl.haxx.se/docs/caextract.html. Let me know if it works.


Attached File(s)
.zip  ProxHTTPSProxy 0.6a.zip (Size: 169.93 KB / Downloads: 643)
Add Thank You Quote this message in a reply
Jun. 03, 2014, 02:29 AM (This post was last modified: Jun. 04, 2014 01:57 AM by JJoe.)
Post: #15
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 03, 2014 01:26 AM)whenever Wrote:  Let me know if it works.

Ummm... Thumbs UpDancing Yes!

(Jun. 03, 2014 01:26 AM)whenever Wrote:  This might be the problem: newer Windows has only a very minimal set of root CA certificates shipped by default. See http://bugs.python.org/issue20916

Version 0.6a uses CA bundle from http://curl.haxx.se/docs/caextract.html.

I was aware of Win Vista+ downloading certs on demand but I am sure my system's store has verisign certs. There was an issue with verisign certs and OpenSSL. So it could be that my system's certs are insufficient.

However, I thought the Python script had to use "ssl.enum_certificates(store_name)" or "ssl.enum_crls(store_name)" to retrieve certificates from Windows’ store. Since I found a CA bundle from http://curl.haxx.se/docs/caextract.html in Python, I assumed it was the default store but no.

Probably best for the proxy to have its own store, anyway.

I will study some more.

Thanks

Edit:
Note, Version 0.6 uses "ssl.create_default_context()" which

Quote: load the system’s trusted CA certificates, enable certificate validation and hostname checking, and try to choose reasonably secure protocol and cipher settings.

https://docs.python.org/3.4/library/ssl....l-security
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: