Post Reply 
ProxHTTPSProxyMII: Development
Jul. 22, 2019, 02:12 AM
Post: #241
RE: ProxHTTPSProxyMII: Development
(Jul. 20, 2019 06:48 PM)vlad_s Wrote:  Why are hop-by-hop headers (line 233) removed from the ProxyTool.py file?

The HTTP/1.1 specifications require proxies to remove hop-by-hop headers.

https://tools.ietf.org/html/draft-ietf-h...on-7.1.3.1 Wrote:Hop-by-hop header fields, which are meaningful only for a single
transport-level connection, and are not stored by caches or
forwarded by proxies.
Add Thank You Quote this message in a reply
Jul. 22, 2019, 07:04 PM (This post was last modified: Jul. 22, 2019 07:05 PM by vlad_s.)
Post: #242
RE: ProxHTTPSProxyMII: Development
Well, I have a bad understanding of all this, but I will explain what I asked for. Privoxy still puts these headers for clients, it’s visible in the log, but in fact they don’t. Does Privoxy violate standards?
Some sites, such as tjournal.ru, vc.ru, require a protocol change (the Upgrade header), in this form in the developer console I get an error HTTP 403 (it is generated by the site, not Privoxy). I modified this line in ProxyTool.py and finally saw instead of HTTP 403 in HTTP 101. I don’t know what the site developers were trying to change the protocol, but it didn’t work. So I decided to ask.
Add Thank You Quote this message in a reply
Jul. 23, 2019, 06:08 AM
Post: #243
RE: ProxHTTPSProxyMII: Development
This will take some more study. The short story so far is...

tjournal.ru and vc.ru are trying to use the WebSocket protocol.

I'm not sure our proxies can or should handle WebSocket.

Although the specs do not allow proxies to "forward" hop-by-hop headers, proxies can add their own hop-by-hop headers.
We need to send or allow (at least)

Code:
Upgrade: WebSocket
Connection: Upgrade

Did the sites work correctly after you made the changes to ProxyTool.py?

Can Privoxy filter the WebSocket data?
https://sourceforge.net/p/ijbswa/mailman/message/34532990/
Add Thank You Quote this message in a reply
Jul. 23, 2019, 12:03 PM
Post: #244
RE: ProxHTTPSProxyMII: Development
WebSocket is actually one of the things I tried to filter. I had an experimental version of Proxomitorn Reborn that attempted to tunnel WS connections.

The big difference between WS and HTTP is that it's truly bidirectional instead of "half duplex"; either side can send at any time and one side can send multiple times before the other side does (if ever). A typical proxy's "read client, filter, send to server; read server, filter, send to client" loop will work only in very limited cases and make other cases look like a hang.
Add Thank You Quote this message in a reply
Jul. 23, 2019, 05:34 PM (This post was last modified: Jul. 23, 2019 05:40 PM by vlad_s.)
Post: #245
RE: ProxHTTPSProxyMII: Development
After the changes in ProxyTool.py, the above listed sites work without changes. Does Privoxy support WebSocket? The developer’s response is rather vague. Here is what is recorded in the Privoxy log:
Code:
...
2019-07-23 20:14:15.328 7fe55effd700 Header: scan: GET http://tjournal.ru/chan/system HTTP/1.1
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Accept: */*
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Host: tjournal.ru
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Accept-Encoding: gzip, deflate, br
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Sec-WebSocket-Version: 13
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Origin: https://tjournal.ru
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Sec-WebSocket-Extensions: permessage-deflate
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Sec-WebSocket-Key: xLQdEBcAZbVdlHZTmiHG4g==
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Connection: keep-alive, Upgrade
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Cookie: pushVisitsCount=45;fingerprint=f794e8cfdfc384g67de; adblock-state=1; audio_
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Pragma: no-cache
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Cache-Control: no-cache
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Upgrade: websocket
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Tagged: ProxHTTPSProxyMII FrontProxy/v1.5 Python/3.7.3:397
2019-07-23 20:14:15.329 7fe56d32f700 Header: scan: GET http://static-osnova.gcdn.co/static/build/tjournal.ru/dropdown.7ed182cc2aa8236bff77.min.js HT
2019-07-23 20:14:15.329 7fe56d32f700 Header: scan: Accept: */*
2019-07-23 20:14:15.329 7fe56d32f700 Header: scan: Host: static-osnova.gcdn.co
...
2019-07-23 20:14:15.360 7fe55effd700 Header: Keeping the client header 'Connection: keep-alive, Upgrade' around. The server connection will be kept alive if possible.
...
2019-07-23 20:14:15.400 7fe55effd700 Header: scan: HTTP/1.1 101 Switching Protocols
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Server: nginx
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Date: Tue, 23 Jul 2019 17:14:15 GMT
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Content-Length: 0
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Connection: upgrade
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Sec-WebSocket-Extensions: permessage-deflate
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Access-Control-Allow-Credentials: true
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Access-Control-Allow-Origin: https://tjournal.ru
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Sec-WebSocket-Accept: sGlz46w7kC9gyLQiIHLN6IA81u4=
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Upgrade: websocket
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Strict-Transport-Security: max-age=31536000; preload
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: X-XSS-Protection: 1; mode=block
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: X-Content-Type-Options: nosniff
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Referrer-Policy: origin
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Content-Security-Policy: default-src * data: 'unsafe-eval' 'unsafe-inline'
2019-07-23 20:14:15.406 7fe55effd700 Header: Adding: Proxy-Connection: keep-alive
...

What I changed in ProxyTool.py to avoid misunderstandings was:
Code:
except (ssl.SSLEOFError, ssl.SSLError) as e:
            logger.error(Fore.RED + Style.BRIGHT + "[SSLError]")
            self.send_error(417, message="Exception %s" % str(e.__class__), explain=str(e))

    def purge_headers(self, headers):
        "Remove hop-by-hop headers that shouldn't pass through a Proxy"
        for name in ["Connection", "Keep-Alive", "Upgrade",
                     "Proxy-Connection", "Proxy-Authenticate"]:
            del headers[name]

    def purge_write_headers(self, headers):
        self.purge_headers(headers)
        for key, value in headers.items():
            self.send_header(key, value)
        self.end_headers()

changes:
Code:
except (ssl.SSLEOFError, ssl.SSLError) as e:
            logger.error(Fore.RED + Style.BRIGHT + "[SSLError]")
            self.send_error(417, message="Exception %s" % str(e.__class__), explain=str(e))

    def purge_headers(self, headers):
        "Remove hop-by-hop headers that shouldn't pass through a Proxy"
        for name in [""]:
            del headers[name]

    def purge_write_headers(self, headers):
        self.purge_headers(headers)
        for key, value in headers.items():
            self.send_header(key, value)
        self.end_headers()
235 line.
Add Thank You Quote this message in a reply
Jul. 26, 2019, 10:16 PM
Post: #246
RE: ProxHTTPSProxyMII: Development
This is what I see in the ProxHTTPSProxyMII logs after enabling the Upgrade header:
Code:
...
[01:06] 624 [D] "GET https://tjournal.ru/chan/content-108174" 101 0
127.0.0.1 - - [27/Jul/2019 01:06:38] code 400, message Bad HTTP/0.9 request type ('Á©J+ÕÞà}\x9f\x10')
127.0.0.1 - - [27/Jul/2019 01:06:38] "Á©J+ÕÞà}......a¢....çêú.áo.zÿwfc.lÈbüJ/.Cyì..ÜJ...1nv" 400 -
[01:06] 625 [D] "GET https://leonardo.osnova.io/b72c9bb3-5240-82b3-feaf-3b7fba1d0419/-/resize/48/" 200 598
[01:06] 626 [D] "GET https://tjournal.ru/chan/content-108199" 101 0
[01:07] 627 [D] "GET https://leonardo.osnova.io/05fcbd49-02d1-cf13-f121-ee47f4a0e09a/-/scale_crop/1300x819/center/-/format/webp/
127.0.0.1 - - [27/Jul/2019 01:07:08] code 400, message Bad HTTP/0.9 request type ('\x88\x80O\x8d')
127.0.0.1 - - [27/Jul/2019 01:07:08] "..O.      L" 400 -
127.0.0.1 - - [27/Jul/2019 01:07:22] code 400, message Bad request syntax ('\x88\x80\x05¡¤§')
127.0.0.1 - - [27/Jul/2019 01:07:22] "...¡¤§" 400 -
127.0.0.1 - - [27/Jul/2019 01:08:48] code 400, message Bad request version ('260')
127.0.0.1 - - [27/Jul/2019 01:08:48] "CNT 1 CON 260" 400 -
...
Add Thank You Quote this message in a reply
Sep. 12, 2019, 08:24 PM
Post: #247
RE: ProxHTTPSProxyMII: Development
Is it possible to somehow transfer such sections as [PROXY http://127.0.0.1:80], [PROXY socks5: //127.0.0.1: 9050], [SSL Pass-Thru] additionally into a separate file?
Add Thank You Quote this message in a reply
Sep. 15, 2019, 01:22 AM
Post: #248
RE: ProxHTTPSProxyMII: Development
(Sep. 12, 2019 08:24 PM)vlad_s Wrote:  Is it possible to somehow transfer such sections as [PROXY http://127.0.0.1:80], [PROXY socks5: //127.0.0.1: 9050], [SSL Pass-Thru] additionally into a separate file?

The current version can't do it.
Somebody would have to add some code.

A feature request.
Add Thank You Quote this message in a reply
Sep. 16, 2019, 03:01 AM
Post: #249
RE: ProxHTTPSProxyMII: Development
(Jul. 26, 2019 10:16 PM)vlad_s Wrote:  This is what I see in the ProxHTTPSProxyMII logs after enabling the Upgrade header:
That looks like WebSocket frames, and the "CNT" is, as before, MSNP; neither of which HTTP(S) proxies are prepared to deal with.
Add Thank You Quote this message in a reply
Sep. 21, 2019, 07:14 PM
Post: #250
RE: ProxHTTPSProxyMII: Development
What can be said about this?
https://github.com/urllib3/urllib3/issues/1678
Add Thank You Quote this message in a reply
Sep. 25, 2019, 03:49 AM
Post: #251
RE: ProxHTTPSProxyMII: Development
What versions of Python, OpenSSL, and urllib3 are you using?

vladns https://github.com/urllib3/urllib3/issues/1678 Wrote:has a setting in its work logic that disables the verification of any certificate. And it all works, only on urllib3 version below 1.25.


At https://pypi.org/project/urllib3/ I see

Quote:1.25.5 (2019-09-19)
Add mitigation for BPO-37428 affecting Python <3.7.4 and OpenSSL 1.1.1+ which caused certificate verification to be enabled when using cert_reqs=CERT_NONE. (Issue #1682)

Issue #1682

sbstp https://github.com/urllib3/urllib3/issues/1682 Wrote:In requests, I need to disable certificate verification because we use self signed certificates. I create a Session object and set its verify property to False, which used to worked perfectly fine until urllib3 1.25.4.
Add Thank You Quote this message in a reply
Sep. 25, 2019, 08:25 PM (This post was last modified: Sep. 25, 2019 08:47 PM by vlad_s.)
Post: #252
RE: ProxHTTPSProxyMII: Development
Python 3.7.3, OpenSSL 1.1.1c 28 May 2019, urllib3 at the time of the test 1.25-1.25.3

Honestly, I didn’t really understand quotes about corrections in urllib. I just updated the version of this library, and restarted ProxHTTPSProxyMII and received the above result for a self-signed certificate, and it was (ip address of the web server) with the exception:
Code:
[SSL No-Verify]
192.168.2.1
With urllib3 1.25-1.25.3, this section stopped working.
For this to work, do you need to either downgrade Python3, OpenSSL, and use urllib3 1.25-1.25.3?
Add Thank You Quote this message in a reply
Sep. 26, 2019, 02:44 AM (This post was last modified: Sep. 26, 2019 02:56 AM by JJoe.)
Post: #253
RE: ProxHTTPSProxyMII: Development
Quote:affecting Python <3.7.4 and OpenSSL 1.1.1+

I think this says,

'affecting Python before version 3.7.4 that use any versions of OpenSSL 1.1.1'

(Sep. 25, 2019 08:25 PM)vlad_s Wrote:  For this to work, do you need to either downgrade Python3, OpenSSL, and use urllib3 1.25-1.25.3?

*If* your problem is fixed by their solution, updating urllib3 to v 1.25.5 or Python to to 3.7.4 should fix it.
However, their problem came with urllib3 1.25.4.
Your problem came with 1.25-1.25.3.

vlad_s Wrote:And it all works, only on urllib3 version below 1.25.

The quick solution is to revert to 1.24.

I'm using the exe version for Windows XP.
I think, it works as well as the network does.

To chase this, I would have to install Python 3.7...

I can't promise a quick solution.

Which version of Python do we want to use?
Add Thank You Quote this message in a reply
Sep. 26, 2019, 05:25 PM (This post was last modified: Sep. 26, 2019 05:27 PM by vlad_s.)
Post: #254
RE: ProxHTTPSProxyMII: Development
With current versions of Python and OpenSSL, my problem manifests itself, it seems that on all or many versions of urllib3 above 1.24.3. Yesterday I updated urllib3 to the latest available (1.25.6) pip3 package manager, the problem remains. Ubuntu already has a built-in Python 3.6, I updated it to package distribution manager 3.6 to 3.7. As a result, the latest version is 3.7.3, this is from the python3.7 - 3.7.3-2 ~ 18.04.1. Formally, this system is believed by Python to be listed as 3.7. If the current version is now 3.7.4, then you need to keep this in mind, as most likely 3.7.3 will upgrade to 3.7.4 in Ubuntu.
Add Thank You Quote this message in a reply
Sep. 26, 2019, 05:54 PM
Post: #255
RE: ProxHTTPSProxyMII: Development
I updated Python to 3.7.4, urllib3 to 1.25.6, the problem remains.
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: