ProxHTTPSProxyMII: Reloaded

[JJoe]

(Jun. 23, 2018 09:29 AM)kichrot Wrote:  Hello!
...
Sorry for my English, I'm writing through an interpreter.

Welcome!

Thank you for your work and post.

The English was good enough.

[pepak]

(Jun. 24, 2018 07:41 PM)JJoe Wrote:  What are you using to generate and apply the patches?
I have been unable to find a utility that will apply all the patches correctly on Windows 10.

It's exported from git (git format-patch) and you should be able to re-apply it using git apply.

Quote:Could you zip the source and upload?

Sure. I am uploading everything including git info - it stared as a clone of https://github.com/wheever/ProxHTTPSProxyMII, I attached branch jjoe with your changes to it, then attached branch pepak with my changes.

Quote:For our purposes, you could temporarily tag it as 1.5.1wip.

I prefer to leave the versioning to you.

Quote:ProxHTTPSProxyMII: Development may be the more appropriate thread.

It might, but then again, if I post here, I consider it a part of a discussion and hopefully everyone will consider it so, rather than an official release. I think my changes are working fine, but as they are my first attempt at developing in Python, I would rather have someone read the changes before they are committed.

[ryszardzonk]

I got problem with certificate validation on ProxHTTPSProxyMII 1.5 on that site https://www.ssllabs.com/ssltest/analyze.....240.18.19 even after adding proper certtificate to cacert.pem from http://cacerts.digicert.com/DigiCertSHA2...rverCA.crt Any ideas what may be wrong?

Code:

502: HTTPError

The following error occurred while trying to access https://static.xx.fbcdn.net/

HTTPSConnectionPool(host='static.xx.fbcdn.net', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'),))
Generated on 2018-07-10 09:49:06.136996 by ProxHTTPSProxyMII RearProxy/v1.5

[JJoe]

(Jul. 10, 2018 07:48 AM)ryszardzonk Wrote:  Any ideas what may be wrong?

'static.xx.fbcdn.net' works for me.
I think, validation only requires 'DigiCert High Assurance EV Root CA'.

'502: Bad Gateway' message may be caused by site's server failing to respond. Which could be caused by incorrect url, server or network problems, router, dns, missing or incorrect data in the client's request, etc.

Try https://static.xx.fbcdn.net/rsrc.php/v3/yb/r/GsNJNwuI-UM.gif
which loads for me in a 'new private window', (no cookies, no referer, etc).
Using Opera portable.

[ryszardzonk]

Yes you are right. It works. Turned out for some reason it ended up in my hosts file so it was a network problem after all.

[vlad_s]

Hello! I can not get the right column of two sites to be displayed https://vc.ru/ and https://tjournal.ru/
Specifically on trying to add to the section [SSL Pass-Thru] and [BYPASS URL] it vc.ru/chan/ and https://vc.ru/chan/* accordingly does not help. With the rule iptables

Code:

iptables -t nat -I PREROUTING -s 192.168.2.211/32 -p tcp -m tcp --dport 443 -j ACCEPT

everything works correctly, but it does not fit, because past the proxy.

[JJoe]

(Sep. 02, 2018 05:57 PM)vlad_s Wrote:  Hello! I can not get the right column of two sites to be displayed https://vc.ru/ and https://tjournal.ru/

Code:

[SSL Pass-Thru]
vc.ru
tjournal.ru

Works for me but yuck.

When we do something that they do not like we get "bad_user_visit"

MII shows

Code:

584 [D] "GET https://tt.onthe.io/?k[]=12300:bad_user_visit...

The server does not send the stream to 'bad_users'

You may need to clear cookies and data. Also, there may be a time penalty assigned to your browser and/or ip address.

[Sudenr]

Hi! I'm using ProxHTTPSProxMII v1.5 on Python 3.6.6 on Windows and sometimes (especially when program generates many certificates at once) I have errors like

SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:841)" while trying to establish local SSL tunnel for [site.example.com:443]

If I delete certificate for site.example.com and renew page, re-created certificate usually works as it should be.

I'm using EC prime256 EC certificate and key. It works perfectly with v1.4 but not with 1.5

Thank you!

[JJoe]

(Sep. 04, 2018 09:21 PM)Sudenr Wrote:  If I delete certificate for site.example.com and renew page, re-created certificate usually works as it should be.

Have you tried renewing the page with the original certificate? If yes, what happened?
Have you compared the certificates?

[Sudenr]

UPD:
Alas, even without "SSL Accelerator" addon in Firefox, ProxHTTPSProxyMII still continues to spawn errors, albeit somewhat less often.
So I did some research:

1. Verification showed that the problem only occurs with Firefox, Chrome-based is not affected.

2. The problem arises even in the clean, fresh-installed Firefox.

3. The problem arises if a site is opened that loads a lot of other encrypted sites simultaneously. Most often this is a variety of imgNN.example.com

4. The problem occurs regardless of certificate type - EC or RSA
But generated certificates are valid in both cases (if check it with Windows)

5. When I try to download a picture from subdomain with an incorrect certificate, Firefox gives an error:
"SEC_ERROR_REUSED_ISSUER_AND_SERIAL"

It seems, that it's caused by identical serial number in generated certificates (and paranoid Firefox security), so I check how certs generated, and found line

Code:

cert.set_serial_number(int(time.time()*10000))

in CertTool.py and thats explains everything.
I changed it to

Code:

cert.set_serial_number(int(time.time()*random.randint(1, 10000)))

(yep, dirty hack), delete Certs folder and restart ProxHTTPSProxyMII.
No SSLv3 errors for 3 days.

[pepak]

Personally, I opted for:

Code:

cert.set_serial_number(int.from_bytes(os.urandom(16), byteorder='big'))

This version is cryptographically secure, anything based on time or random is not.

[Sudenr]

(Sep. 11, 2018 03:53 PM)pepak Wrote:  This version is cryptographically secure, anything based on time or random is not.

Thanks, that's better. I'm not a programmer myself, but I understand that time*random is dirty hack

[ryszardzonk]

(Sep. 04, 2018 04:01 AM)JJoe Wrote:  

(Sep. 02, 2018 05:57 PM)vlad_s Wrote:  Hello! I can not get the right column of two sites to be displayed https://vc.ru/ and https://tjournal.ru/

Code:

[SSL Pass-Thru]
vc.ru
tjournal.ru

Works for me but yuck.

(Sep. 09, 2018 02:01 PM)Sudenr Wrote:  5. When I try to download a picture from subdomain with an incorrect certificate, Firefox gives an error:
"SEC_ERROR_REUSED_ISSUER_AND_SERIAL"

It seems, that it's caused by identical serial number in generated certificates (and paranoid Firefox security), so I check how certs generated, and found line

Code:

cert.set_serial_number(int(time.time()*10000))

in CertTool.py and thats explains everything.
I changed it to

Code:

cert.set_serial_number(int(time.time()*random.randint(1, 10000)))

(yep, dirty hack), delete Certs folder and restart ProxHTTPSProxyMII.
No SSLv3 errors for 3 days.

I don't think I ever stumbled upon "SEC_ERROR_REUSED_ISSUER_AND_SERIAL" in the logs and I use Firefox almost exclusively, but maybe it was only my luck to visit some sites first before they were subdomains in others.
Anyways would this code change also fix problems with vc.ru and tjournal.ru so they would not need [SSL Pass-Thru]? Clearly they were subdomains in the shown example

EDIT:
OpenSSL now supports TLS1.3. Does that mean ProxHTTPSProxyMII would have to be updated to use this new updated library or OpenSSL-1.1.1 can be safely used?

more here:
https://www.openssl.org/blog/blog/2018/0...elease111/

[pepak]

(Sep. 12, 2018 06:58 AM)ryszardzonk Wrote:  I don't think I ever stumbled upon "SEC_ERROR_REUSED_ISSUER_AND_SERIAL" in the logs and I use Firefox almost exclusively, but maybe it was only my luck to visit some sites first before they were subdomains in others.

I did. Had to restart Firefox or re-generate the certificates when that happened.

It also seems that with this change, the quite frequent ResponseNotReady errors generated by ProxHTTPSProxy for some sites are a thing of the past. So I would quite recommend using this patch.

Quote:EDIT:
OpenSSL now supports TLS1.3. Does that mean ProxHTTPSProxyMII would have to be updated to use this new updated library or OpenSSL-1.1.1 can be safely used?

ProxHTTPSProxy does not need any change, but a modification may be necessary for the underlying OpenSSL bindings. Although a quick check suggest that the necessary change may already be in.

[vlad_s]

(Sep. 12, 2018 06:58 AM)ryszardzonk Wrote:  Anyways would this code change also fix problems with vc.ru and tjournal.ru so they would not need [SSL Pass-Thru]? Clearly they were subdomains in the shown example

Yes, just like subdomains. The proposed option in [SSL Pass-Thru] works, but this method is not desirable.

pepak, question to you or to someone who understands. I work on a router (ubuntu server 16.04). And sometimes there is an error that the certificate is not valid because of different time on the router and clients. Clients are synchronized from this router to NTP, but not always accurately, there is a difference of 0.5 seconds and this error occurs. I can not make exact synchronization of time. I wrote about this here https://prxbx.com/forums/showthread.php?...5#pid19135
The question is, is it possible to specify the time when the certificate should start validating more early, rather than at the time the certificate was generated?