ProxHTTPSProxyMII: Development
|
Jun. 11, 2014, 05:14 AM
Post: #31
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
0.8a should fix the urllib3 problem.
You already help a lot. Does that mean I could be free from fixing problems of the script for the next few days? |
|||
The following 1 user says Thank You to whenever for this post: defconnect |
Jun. 11, 2014, 05:22 PM
Post: #32
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 11, 2014 05:14 AM)whenever Wrote: Does that mean I could be free from fixing problems of the script for the next few days? Ahhh! The silver lining to my clouds. (Jun. 11, 2014 05:14 AM)whenever Wrote: 0.8a should fix the urllib3 problem. It does. The Proxomitron shows unclosed connections accumulating. Do you see the same? When I open mail.yahoo.com there is a GET to "sb.scorecardresearch.com/p?" the response is a 302 to "sb.scorecardresearch.com/p2?" that hangs and does not close. After I use the Proxomitron to abort the connection, the browser again requests "sb.scorecardresearch.com/p?". Got to go. |
|||
Jun. 12, 2014, 01:11 AM
(This post was last modified: Jun. 12, 2014 01:12 AM by whenever.)
Post: #33
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 11, 2014 05:22 PM)JJoe Wrote: The Proxomitron shows unclosed connections accumulating. Do you see the same? I saw that too. Just had not got time to look into it. The hang happened between Proxomitron and the rear server, right? If so, I think it is a problem similar to http://prxbx.com/forums/showthread.php?t...7#pid17607 What if you change line 102 from: Code: if r.status == 304: to: Code: if r.status in (302, 304): |
|||
Jun. 12, 2014, 02:47 AM
(This post was last modified: Jun. 12, 2014 03:05 AM by JJoe.)
Post: #34
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 12, 2014 01:11 AM)whenever Wrote: That fixed some. Changing it to Code: if r.status in (204, 302, 304): fixed the rest. Hope it was ok to zip and upload as ProxHTTPSProxy 0.8b and it is very quick! Should anybody still have problems with https at yahoo... In your Proxomitron's bypass list there may be an entry like Code: # Bypasses for Yahoo messenger (ok to remove if you don't use it) When this entry matches the rear ProxHTTPSProxy server may be bypassed. Yahoo may fail to load and the browser may report a 'redirection error'. This may happen after submitting login info. Also, I have been using a very simple cfg, "TestProxHTTPSProxy1.cfg", to help find the source of problems. There are a few simple filters and no lists. Code: ## |
|||
Jun. 13, 2014, 09:42 AM
Post: #35
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 12, 2014 02:47 AM)JJoe Wrote: Nice job! 0.8c added colorful output. It depends on module colorama. Code: c:\Python34\Scripts>pip install colorama |
|||
Jun. 16, 2014, 02:37 AM
Post: #36
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 13, 2014 09:42 AM)whenever Wrote: 0.8c added colorful output. It depends on module Ahhh, that is better and green. I was still seeing hangs at yahoo on "content-length: 0" responses. Including hangs on 200 responses. So I commented out Code: # Proxomitron hang for 304 response if we don't close the connection and added Code: # Hang between Proxomitron and Rear Proxy for "content-length: 0" response if we don't close the connection Hangs are resolved and connections closed. However, things seem slower and I'm seeing EOF errors with each "content-length: 0" response Code: [21:18:55] "GET https://login.yahoo.com/config/verify?.done=https%3a//us-mg.mail.yahoo.com/neo/b/launch%3f.rand=2 HTTP/1.1" 302 0 Code: [21:19:22] "GET https://us-mg.mail.yahoo.com/neo/b/dimensions?noFlush&width=1349&height=658 HTTP/1.1" 200 0 Is my code correct? I see 'Content-Length' mentioned in proxytool.py but it will take some more study before I completely understand it. |
|||
Jun. 16, 2014, 08:38 AM
Post: #37
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 16, 2014 02:37 AM)JJoe Wrote: "data" is request body and most of the time it is None provided the http method is not POST. Your code disable the connection reuse for most of the time, that might be the cause of the slow down. "r.data " is response body and "self.body_length" is its length. The code could be changed to Code: if self.body_length == 0: (Jun. 16, 2014 02:37 AM)JJoe Wrote: I'm seeing EOF errors with each "content-length: 0" response I am not sure if it helps but according to http://stackoverflow.com/questions/14102...-occurred, Let's give it a test by changing line 146 from Code: sslparams = dict(cert_reqs="REQUIRED", ca_certs="cacert.pem", ssl_version="SSLv23") to Code: sslparams = dict(cert_reqs="REQUIRED", ca_certs="cacert.pem", ssl_version="TLSv1") (Jun. 16, 2014 02:37 AM)JJoe Wrote: I see 'Content-Length' mentioned in proxytool.py but it will take some more study before I completely understand it. Persistent connection needs 'Content-Length' value to to indicate the transfer-length of the message-body. proxytool.py updates this header with the right value. |
|||
Jun. 16, 2014, 10:00 PM
Post: #38
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 16, 2014 08:38 AM)whenever Wrote: Your code disable the connection reuse for most of the time, Yep... I added a flag to check. I think the slowdown was mostly an internet connection issue, however. (Jun. 16, 2014 08:38 AM)whenever Wrote: "r.data " is response body and "self.body_length" is its length. And I wondered where whenever found this... And then I wondered how I missed it. Closing on "self.body_length == 0:" has fixed my sign in problem. Sometimes I see a stall. When this happens the Proxomitron's log will show Code: Socket Error 10053 for request flush (Jun. 16, 2014 08:38 AM)whenever Wrote: Let's give it a test by changing line 146... I am still seeing the errors but TLSv1 is quicker and appears to use less cpu. Time to go again. Have fun. |
|||
Jun. 17, 2014, 10:06 AM
Post: #39
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 16, 2014 10:00 PM)JJoe Wrote: It seems Proxomitron hadn't expected that we closed the connection. Let's be a little more polite by changing line 119 of proxytool.py from Code: # Reuse the connection if the Client asks that to Code: # Connection management |
|||
Jun. 18, 2014, 01:26 AM
(This post was last modified: Jun. 18, 2014 01:28 AM by JJoe.)
Post: #40
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 17, 2014 10:06 AM)whenever Wrote: Let's be a little more polite by changing line 119 of proxytool.py from I just saw and logged it, after changing line 119. Maybe some context will help. Some of these problems seem to happen only while web page filtering is enabled. For testing, I have added another instance of the Proxomitron. Looks like FrontProxHTTPSProxy>firstProxomitron>secondProxomitron>RearProxHTTPSProxy. firstProxomitron only logs, no user filters. secondProxomitron may filter. A request for the yahoo mail page after signing in looks like: Code: ********firstProxomitron********* Summary: secondProxomitron user header filter changes "Accept-Encoding" header from "gzip,deflate,lzma,sdch" to "gzip, deflate" ** 3 Socket Error 10053 for request flush **, this does not always happen. "Content-Encoding", "Content-Length" and Connection headers are removed but not by user filters. This is what I remember and expected. "Transfer-Encoding: chunked" header is added, as expected. Not shown: When the secondProxomiton is not filtering, only the Connection header is missing in firstProxomitron log window. So could the switch to "Transfer-Encoding: chunked" be a problem? I'll be looking at \k next. |
|||
Jun. 18, 2014, 11:25 AM
(This post was last modified: Jun. 18, 2014 12:29 PM by whenever.)
Post: #41
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 18, 2014 01:26 AM)JJoe Wrote: Maybe some context will help I mean being polite because it is not a necessary. In HTTP/1.1 you could close a connection without sending "Connection: close" first. (Jun. 18, 2014 01:26 AM)JJoe Wrote: So could the switch to "Transfer-Encoding: chunked" be a problem? I don't so. It's the normal behavior of a filtering proxy. (Jun. 18, 2014 01:26 AM)JJoe Wrote: ** 3 Socket Error 10053 for request flush **, this does not always happen. When that happen, does it break the web page? I saw you mentioned it after applying the "if self.body_length == 0: " fix. Do you see it before that? v0.8d is attached to make sure we are on the same boat. It should solve the EOF errors problem. |
|||
Jun. 18, 2014, 01:45 PM
Post: #42
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 18, 2014 11:25 AM)whenever Wrote:(Jun. 18, 2014 01:26 AM)JJoe Wrote: ** 3 Socket Error 10053 for request flush **, this does not always happen. No. It does eventually load. ProxHTTPSProxy does work. I have been using it. Thank you! (Jun. 18, 2014 11:25 AM)whenever Wrote: I saw you mentioned it after applying the "if self.body_length == 0: " fix. Do you see it before that? I don't remember. I will check. Have fun. |
|||
Jun. 18, 2014, 05:56 PM
Post: #43
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 18, 2014 01:45 PM)JJoe Wrote:(Jun. 18, 2014 11:25 AM)whenever Wrote: I saw you mentioned it after applying the "if self.body_length == 0: " fix. Do you see it before that? No, I do not see it. When not using the fix, the browser hangs before I would expect to see the socket error alert. The URL is the result of a redirect. Directly requesting the URL does not cause an alert. I am not sure the alert is a problem. I may need to find another connection or ISP to test with. (Jun. 18, 2014 01:26 AM)JJoe Wrote: I'll be looking at \k next. I do believe \k is causing Code: ---------------------------------------- These exceptions go away after removing \k from my filters. The socket error remains, however. (Jun. 18, 2014 11:25 AM)whenever Wrote: v0.8d is attached to make sure we are on the same boat. It should solve the EOF errors problem. My ProxHTTPSProxy's log window is exception and error free, for now. |
|||
Jun. 19, 2014, 04:05 AM
Post: #44
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 18, 2014 05:56 PM)JJoe Wrote: I am not sure the alert is a problem. I may need to find another connection or ISP to test with. I saw "Socket Error 10053 for request flush" too, even its self.body_length is > 0, so that connection was not flagged to be closed by our code, and it later got its response normally. So I am not sure how it is related to the "if self.body_length == 0" fix, but I did see it only after applying that fix. (Jun. 18, 2014 01:26 AM)JJoe Wrote: I do believe \k is causing Those exceptions are expected for \k. I would leave it as is since it was raised beyond the scope of our code. JJoe, would you like to write a new readme and update the instructions on configuring the script to work with Proxomitron? I would look into building an exe version. |
|||
Jun. 19, 2014, 04:13 AM
Post: #45
|
|||
|
|||
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 19, 2014 04:05 AM)whenever Wrote: JJoe, would you like to write a new readme and update the instructions on configuring the script to work with Proxomitron? Yes, I'm collecting info for that now. I may start another thread. You might consider another name for this new proxy. First, I want to retest the socket error. I still had ProxHTTPSProxy in the chain. Then I need to go to bed. |
|||
« Next Oldest | Next Newest »
|