ProxHTTPSProxyMII: Development
|
May. 28, 2014, 03:13 AM
(This post was last modified: Jun. 17, 2018 02:17 AM by JJoe.)
Post: #1
|
|||
|
|||
ProxHTTPSProxyMII: Development | |||
May. 28, 2014, 05:34 AM
Post: #2
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Welcome back stranger.
(May. 28, 2014 03:13 AM)whenever Wrote: What about this routine? So local ProxHTTPSProxyPartA and ProxHTTPSProxyPartB... I'm guessing ProxHTTPSProxyPartA and ProxHTTPSProxyPartB are using the remote server's cert, as described at http://mitmproxy.org/doc/howmitmproxy.html under "Explicit HTTPS", to hide the Proxomitron. Is the file at your forum? For casual browsing, I have been using the 'SSL Certificate CN Always Matches' patch at http://prxbx.com/forums/showthread.php?tid=2156 . Also, using my browser's command line options to disable its cert checks. I would like to get proper verification back (and maybe a little more) tho. Eventually, I or we will probably need a new proxy or scheme. Have you noticed? HandyCache users say thank you very much. Just found a patch at http://rghost.net/51510677 , referenced http://handycache.ru/component/option,co...pic,471.0/ . Have fun |
|||
May. 28, 2014, 09:39 AM
Post: #3
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2014 05:34 AM)JJoe Wrote: I'm guessing ProxHTTPSProxyPartA and ProxHTTPSProxyPartB are using the remote server's cert, as described at http://mitmproxy.org/doc/howmitmproxy.html under "Explicit HTTPS", to hide the Proxomitron. It's just like that. The front server will use a self made CA to generate server certificates on the fly. (May. 28, 2014 05:34 AM)JJoe Wrote: I would like to get proper verification back (and maybe a little more) tho. Eventually, I or we will probably need a new proxy or scheme. The rear server will do the certificates verification. It needs Python version after 3.4. (May. 28, 2014 05:34 AM)JJoe Wrote: Have you noticed? [url=http://www.google.com/search?q=ProxHTTPSProxy]HandyCache users say thank you very much. I hadn't expected that. In deed, I haven't been using Proxomitron for a long time. I just got some time to learn Python again so I think I would update ProxHTTPSProxy to use a new scheme. I am still working on the script. At the same time you can prepare for it:
|
|||
May. 28, 2014, 07:23 PM
Post: #4
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2014 09:39 AM)whenever Wrote: I just got some time to learn Python again so I think I would update ProxHTTPSProxy to use a new scheme. There is a library that may help, http://www.python-requests.org/en/latest/ . Have fun |
|||
May. 29, 2014, 02:39 AM
Post: #5
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2014 07:23 PM)JJoe Wrote: There is a library that may help, http://www.python-requests.org/en/latest/ I looked at it before but think it would be too heavy for our such lightweight script. Here is the version 0.5 for you to play with. http://forum.proxomitron.cn/download/file.php?id=301
Exceptions-U.ptxt: Code: $OHDR(Tagged:Proxomitron FrontProxy/*) $SET(0=i_proxy:3.) $SETPROXY(127.0.0.1:8081) The script should just throw exception for bad server certification. Need to find a server with bad certification for testing. BTW, I still couldn't upload attachment. |
|||
May. 30, 2014, 07:58 PM
(This post was last modified: May. 31, 2014 12:31 AM by JJoe.)
Post: #6
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Ok. Installed and running.
Fresh install of "Python3.4.1" didn't work for me. All is well after an uninstall and reinstall of "3.4.1", tho. "Win32 OpenSSL v1.0.1g Light" required "Visual C++ 2008 Redistributables". Don't forget to bypass the Proxomitron before using "pip install pyopenssl" at the Window's command prompt. Assuming Python34 is installed in the default folder Code: c:\Python34\Scripts>pip install pyopenssl To enable $SETPROXY(127.0.0.1:8081), add the Rear proxy to the Proxomitron via "Proxy" on the Proxomitron's main dialog Code: 127.0.0.1:8081 ProxHTTPSProxyRear (More detailed instructions later or per request) For now, I'm using these Exceptions-U entries Code: $OHDR(Tagged:Proxomitron FrontProxy/*) $SET(keyword=$GET(keyword)i_proxy:3.) $SETPROXY(127.0.0.1:8081)(^) in sidki's set. This way the keyword is always set regardless of the other list entries and the other list entries are checked. When I $RDIR in Exceptions-U to block a connection I'll be using something like Code: unwanted.com/ $USEPROXY(false)$SET(keyword=i_proxy:0.)$RDIR($GET(blackhole)/killed.gif?\u) otherwise the rear proxy will be asked for "killed.gif". However, the new routine works better with sidki's set in bypass while using the Proxomitron's "Use Remote Proxy" setting. I'm not sure why. So, I'm also using the Proxomitron's default set with the web filters bypassed and this, Code: [HTTP headers] , header filter to test the routine. Current issues 1. When searching at startpage.com Code: 127.0.0.1 - - [30/May/2014 08:42:40] "GET http://s10-us4.startpage.com/tst2/engl 2. Need to find a server with bad certification for testing. Got to go. |
|||
May. 31, 2014, 01:27 AM
Post: #7
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
I created an instance of The Proxomitron in a sandbox to test against self signed certs.
So the chain was FrontServer>Proxomitron>RearServer>Sandboxied Proxomitron. After a request for https://www.google.com, RearServer says, "ssl.CertificateError: hostname 'www.google.com' doesn't match 'Proxomitron'" Code: ============================================================================ |
|||
May. 31, 2014, 06:29 AM
Post: #8
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Using a header filter to apply the parent proxy seems to be the best way without impacting the Exceptions-U.ptxt entries.
Code: http.client.IncompleteRead: IncompleteRead(0 bytes read, 42 more expected) This was caused by the server setting "Content-Length" to 42 bytes but sending less bytes. Seems like a kind of anti caching technology. Test URL: Code: https://s7-us4.startpage.com/cgi-bin/ccspacer?ns=1&anticache=15007&/filename.gif Result without proxying: anti cache.png (Size: 24.64 KB / Downloads: 930) Version 0.6 suppresses this kind of error and adds warning for certificate error. Test site: https://kyfw.12306.cn/ |
|||
May. 31, 2014, 05:14 PM
Post: #9
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 31, 2014 06:29 AM)whenever Wrote: Using a header filter to apply the parent proxy seems to be the best way without impacting the Exceptions-U.ptxt entries. However, I think using either will cause forwarding to be disabled when using the Proxomitron in Bypass mode. I'll probably use an undocumented 'feature'. Entries in the "Bypass any URLs that match this expression" field are checked when the Proxomitron is in Bypass mode. So Bypass list entries like Code: $OHDR(Tagged:Proxomitron FrontProxy/*) $SETPROXY(127.0.0.1:8081)(^) are (or "should be", since I haven't actually done this yet) executed when the Proxomitron is in Bypass mode. $SETPROXY is executed when found but (^) never matches and prevents a bypass. (May. 31, 2014 06:29 AM)whenever Wrote: Version 0.6 suppresses this kind of error and adds warning for certificate error. Version 0.6 is working better. Also works much better with web page filtering enabled. Current Issue: False alarm (I hope) at yahoo login. Code: WARNING: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598) at https://login.yahoo.com/config/login_verify2?&.src=ym |
|||
Jun. 01, 2014, 01:35 PM
Post: #10
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program | |||
Jun. 02, 2014, 07:05 AM
(This post was last modified: Jun. 02, 2014 07:06 AM by whenever.)
Post: #11
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
I didn't get that on either https://login.yahoo.com/config/login_verify2?&.src=ym or https://www.yahoo.com.
Could you reproduce that with or without ProxHTTPSProxy? I got below IPs when I ping them: Code: login.yahoo.com -> 98.139.21.169 What if you using a Hosts file to resolve to those IPs and trying again? |
|||
Jun. 02, 2014, 03:40 PM
(This post was last modified: Jun. 02, 2014 05:12 PM by JJoe.)
Post: #12
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 02, 2014 07:05 AM)whenever Wrote: Could you reproduce that with or without ProxHTTPSProxy? Verification works without ProxHTTPSProxy. (Jun. 02, 2014 07:05 AM)whenever Wrote: What if you using a Hosts file to resolve to those IPs and trying again? Same result at your IPs. I see Code: login.yahoo.com -> 98.138.79.21 Is ProxHTTPSProxy using Python's certificate store? After reading some of the Python documents and ProxHTTPSProxy's code, I assumed it was and that the problem might be a missing cert. Tried replacing the store, same result. I might have replaced the wrong file tho... (Jun. 02, 2014 07:05 AM)whenever Wrote: I didn't get that Now I'm not so sure that it is the store. Are you using Python 3.4.1 from https://www.python.org/downloads/ ? Which operating system? I'm also seeing same error at https://www.verisigninc.com/ , expected since yahoo uses verisign. I'll do more reading later. Edit: added a question and more info. |
|||
Jun. 02, 2014, 06:22 PM
Post: #13
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Which OpenSSL package, http://slproweb.com/products/Win32OpenSSL.html , did you use?
|
|||
Jun. 03, 2014, 01:26 AM
(This post was last modified: Jun. 03, 2014 01:27 AM by whenever.)
Post: #14
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
This might be the problem: newer Windows has only a very minimal set of root CA certificates shipped by default. See http://bugs.python.org/issue20916
Version 0.6a uses CA bundle from http://curl.haxx.se/docs/caextract.html. Let me know if it works. |
|||
Jun. 03, 2014, 02:29 AM
(This post was last modified: Jun. 04, 2014 01:57 AM by JJoe.)
Post: #15
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 03, 2014 01:26 AM)whenever Wrote: Let me know if it works. Ummm... Yes! (Jun. 03, 2014 01:26 AM)whenever Wrote: This might be the problem: newer Windows has only a very minimal set of root CA certificates shipped by default. See http://bugs.python.org/issue20916 I was aware of Win Vista+ downloading certs on demand but I am sure my system's store has verisign certs. There was an issue with verisign certs and OpenSSL. So it could be that my system's certs are insufficient. However, I thought the Python script had to use "ssl.enum_certificates(store_name)" or "ssl.enum_crls(store_name)" to retrieve certificates from Windows’ store. Since I found a CA bundle from http://curl.haxx.se/docs/caextract.html in Python, I assumed it was the default store but no. Probably best for the proxy to have its own store, anyway. I will study some more. Thanks Edit: Note, Version 0.6 uses "ssl.create_default_context()" which Quote: load the system’s trusted CA certificates, enable certificate validation and hostname checking, and try to choose reasonably secure protocol and cipher settings. https://docs.python.org/3.4/library/ssl....l-security |
|||
« Next Oldest | Next Newest »
|