ProxHTTPSProxyMII: Development

[GunGunGun]

Okay, thank you.

I have another question, I dont know if i'm wrong or not but seem ProxHTTPSProxy download speed is "much more" slower than just browser download speed, I know that ProxHTTPSProxy have to decode content of the page with cert key and then send to Proxomitron, and then re-encode the page again and then send to browser, but I think problem is not that, seem ProxHTTPSProxy download content one-by-one instead multi-at-a-time ? Example the program will download html of the page then download one-by-one image, on-by-one css and so on and that make ProxHTTPSProxy slow ? I don't know if that is right or not but if it is right I hope you will apply some tweak to ProxHttpsProxy, many thank!

Quote: I prefer to keep this program as simple as possible. I think Proxomitron and Privoxy can do URL blocking better.

The reason that I really want ProxHTTPSProxy can do more than that is I want ProxHTTPSProxy can exclude rar, zip, image file (have big file size) and only decode html, when I download document or something like this on https site like https://doc.google.com, download window will not appear instantly because rar, zip file that have big file size and take really long time to fully downloaded from ProxHTTPSProxy, so it is better if we can exclude rar, zip, image content!

[whenever]

(Oct. 03, 2014 02:40 PM)GunGunGun Wrote:  I think problem is not that, seem ProxHTTPSProxy download content one-by-one instead multi-at-a-time ? Example the program will download html of the page then download one-by-one image, on-by-one css and so on and that make ProxHTTPSProxy slow ?

For each browser request connection, ProxHTTPSProxyMII will start a new thread to handle it, and those threads work simultaneously. In theory it shouldn't slow down the browser.

(Oct. 03, 2014 02:40 PM)GunGunGun Wrote:  download window will not appear instantly because rar, zip file that have big file size and take really long time to fully downloaded from ProxHTTPSProxy, so it is better if we can exclude rar, zip, image content!

You either have to wait for the download window to appear (the ProxHTTPSProxy style) or wait for the download progress bar to finish (the browser style). If you use Firefox, you could use the Foxyproxy extension to exclude rar, zip from feeding them to ProxHTTPSProxy.

The hostname bypass works before SSL starts. For URL bypass we have the get into the SSL tunnel. I will consider your suggestion when I get time.

[herbalist]

I've been using Proxomitron with Tor by launching it with SocksCap. IMO, Proxomitron offers superior protection against many of the attacks being used against TorBrowser. I'd like to add ProxHTTPSProxy to the chain. Would it be possible to add support for Socks5 to the server facing side so that it could be used with Tor? If not, would it be possible to add command line switches that allow you to start each proxy separately? This way, the server facing side could be started via SocksCap.

[GunGunGun]

(Oct. 05, 2014 09:18 AM)whenever Wrote:  

(Oct. 03, 2014 02:40 PM)GunGunGun Wrote:  I think problem is not that, seem ProxHTTPSProxy download content one-by-one instead multi-at-a-time ? Example the program will download html of the page then download one-by-one image, on-by-one css and so on and that make ProxHTTPSProxy slow ?

For each browser request connection, ProxHTTPSProxyMII will start a new thread to handle it, and those threads work simultaneously. In theory it shouldn't slow down the browser.

(Oct. 03, 2014 02:40 PM)GunGunGun Wrote:  download window will not appear instantly because rar, zip file that have big file size and take really long time to fully downloaded from ProxHTTPSProxy, so it is better if we can exclude rar, zip, image content!

You either have to wait for the download window to appear (the ProxHTTPSProxy style) or wait for the download progress bar to finish (the browser style). If you use Firefox, you could use the Foxyproxy extension to exclude rar, zip from feeding them to ProxHTTPSProxy.

The hostname bypass works before SSL starts. For URL bypass we have the get into the SSL tunnel. I will consider your suggestion when I get time.

Thank you, and about ProxHTTPSProxy portablility, I hope you add an ability to ProxHTTPSProxy can have OpenSSL dll in the same folder, have to install OpenSSL in System folder is a pain, I think ProxHTTPSProxy only need libeay32.dll and ssleay32.dll and msvcr70.dll (optional) to work ?

I have tried to place OpenSSL dll and ProxHTTPSProxy 1.0 in the same folder but dont work like older version of Prox.

Proably OpenSSL can make portable anyway, I ve tried to extract OpenSSL 1.0.1 light using UniExtractor, and then create a folder OpenSSL-Win32 in C:\ and copy all content to that folder, work well but a little bit complex because I need to write a Launcher to do that at least.

[JJoe]

(Oct. 06, 2014 06:25 AM)GunGunGun Wrote:  I have tried to place OpenSSL dll and ProxHTTPSProxy 1.0 in the same folder but dont work like older version of Prox.

My ProxHTTPSProxyMII's config.ini has this entry

Code:

[OPENSSL]
PATH = C:\OpenSSL-Win32\bin\openssl.exe

Did you modify PATH to match?

(Oct. 06, 2014 06:25 AM)GunGunGun Wrote:  Proably OpenSSL can make portable anyway, I ve tried to extract OpenSSL 1.0.1 light using UniExtractor, and then create a folder OpenSSL-Win32 in C:\ and copy all content to that folder, work well but a little bit complex because I need to write a Launcher to do that at least.

IIRC, my OpenSSL installation exe allowed me to choose the OpenSSL directory and whether some files would be added to the System folder or OpenSSL's binaries (/bin) directory.

I have not tried it but it all looks "portable".

HTH

[JJoe]

(Oct. 05, 2014 03:30 PM)herbalist Wrote:  Would it be possible to add support for Socks5 to the server facing side so that it could be used with Tor?

config.ini contains

Quote:# Socks proxy support
# https://github.com/shazow/urllib3/pull/284

[GunGunGun]

(Oct. 07, 2014 12:59 AM)JJoe Wrote:  

(Oct. 06, 2014 06:25 AM)GunGunGun Wrote:  I have tried to place OpenSSL dll and ProxHTTPSProxy 1.0 in the same folder but dont work like older version of Prox.

My ProxHTTPSProxyMII's config.ini has this entry

Code:

[OPENSSL]
PATH = C:\OpenSSL-Win32\bin\openssl.exe

Did you modify PATH to match?

(Oct. 06, 2014 06:25 AM)GunGunGun Wrote:  Proably OpenSSL can make portable anyway, I ve tried to extract OpenSSL 1.0.1 light using UniExtractor, and then create a folder OpenSSL-Win32 in C:\ and copy all content to that folder, work well but a little bit complex because I need to write a Launcher to do that at least.

IIRC, my OpenSSL installation exe allowed me to choose the OpenSSL directory and whether some files would be added to the System folder or OpenSSL's binaries (/bin) directory.

I have not tried it but it all looks "portable".

HTH

I've tried to modify OpenSSL path inside the config.ini file, but seem it have no effect, I copied all OpenSSL file to my Prox folder, and then modify OpenSSL config.ini, then I rename my OpenSSL-Win32 folder from my C drive, I cannot even open ProxHTTPSProxy, rename again then work.

[whenever]

@herbalist, before urllib3 supports socks proxy, you can use Privoxy or Polipo to turn socks5 proxy into https proxy.

@GunGunGun, the exe build of ProxHTTPSProxy doesn't deal with libeay32.dll and ssleay32.dll directly. It just uses the openssl command line to create certificates. If you could make a portable OpenSSL to run below command, then I can package it into ProxHTTPSProxy.

Code:

openssl req -x509 -nodes -days 3652 -newkey rsa:2048 -subj "/C=CN/O=ProxHTTPSProxy/OU=OpenSSL Command Line/CN=ProxHTTPSProxy CA" -extensions v3_ca -keyout private.key -out CA.crt

Attached is the version 1.1 that supports URL bypass. Please test.

[GunGunGun]

The bypass zip, rar feature seem quite nice, thank you.
I will try to make a Launcher for OpenSSL, because OpenSSL just need to copy it to C:/OpenSSL-Win32 so a simple Launcher that copy OpenSSL to that folder before Prox starts is okay.

PS: Merry Christmas!

[mas]

Hi, first of all thanks whenever for ProxHTTPSProxy and JJoe for instructions on how to use it - finally I managed to make it work on my desktop. I'm using Proxomitron primarily to serve as ad-blocker for my mobile devices - they just connect to my desktop and receive clean content without using resource-hogging and ineffective adblockers for android. The question is this assuming I can only set one proxy for my connection, so currently I can either filter http (if set proxo's port) or https (if set ProxHTTPSProxy's front, http in this case wouldn't work at all) - how do I tune either proxo or ProxHTTPSProxyv (or both) to handle both http and https on android devices accessing from local network? Any advice would be highly appreciated, TIA!

[whenever]

@mas, I will make a new version to handle both http and https.

[mas]

Thanks for your work, that would be awesome!

[GunGunGun]

(Oct. 14, 2014 03:03 AM)whenever Wrote:  @mas, I will make a new version to handle both http and https.

Thank you, would be cool, I also have some idea that I think really helpful:

- About the domain blocking feature of ProxHTTPSProxy, I think better if we can block url using full RegEx (or at least block domain + path or path alone like Privoxy can), probably better because if we use Privoxy, Proxomitron to block then we still have to fully download that url and its content through ProxHTTPSProxy, and then ProxHTTPSProxy send them to Privoxy/Proxomitron, there will be a little bit slow and the ad is not blocked anyway because we still waste our bandwidth download the ad. Probably the feature will not blur Privoxy or Proxomitron feature anyway, because the webpage filtering feature is the core of Privoxy, Proxomitron and that is why we need ProxHTTPSProxy to decode content from SSL page then send it to Privoxy/Proxomitron to do the post-process. And some website like Youtube always spawn junk request when we are watching video, so block all those junk speed up video download speed too, I think nice to have full url blocking feature like Proxomitron or Privoxy.

Example this is how Privoxy block domain and path and both:
Domain: .google.com
Path: /ads - Block something like example.com/ads
Both: .google.com/ads - Block google.com/ads/abcxyz/mnopq
RegEx: .google.com/.*?ads - Block google.com if contains ads, example google.com/abcxyz/mnopq/ads/123123123.js
yahoo.com/.*?ads - Block yahoo.com if contains ads, example yahoo.com/abcxyz/mnopq/ads/123123123.js
*ads*.com - Block any domain .com that contains ads
*ads*.org - Block any domain .org that contains ads

- I think better if we can edit config.ini and then the config take effect immdiately without restart ProxHTTPSProxy

- And seem ProxHTTPSProxy increase its RAM a lot after long time use, seem ProxHTTPS still use RAM to cache content ? I remember that in the past, ProxHTTPSProxy's RAM have been increased to about 250-300MB RAM, I restart ProxHTTPSProxy and RAM Usage dropped to 12-15MB. I don't know if there is something like memory leak, caching or maybe Python itself problem..

Here is my picture, 1st is 70MB RAM and 2nd is 15MB after restart. And a picture about Privoxy that I've run for a week, only 7MB RAM, I think the way Privoxy works is the same as ProxHTTPSProxy work, download content and then send to next hop.

Thank!
Best regards,

[whenever]

version 1.1a handles both http and https.

@GunGunGun, Proxomitron blocks URL without downloading the content. I think Privoxy works that way too. So there is no need for ProxHTTPSProxy to do that again, and I don't think I can do that better than Proxomitron.

I see the memory leak too. On my machine it even stops to accept new connections when the RAM goes up to 60MB. ProxHTTPSProxy doesn't cache web content. for now I have no idea how it happens.

[GunGunGun]

(Oct. 16, 2014 01:08 AM)whenever Wrote:  version 1.1a handles both http and https.

@GunGunGun, Proxomitron blocks URL without downloading the content. I think Privoxy works that way too. So there is no need for ProxHTTPSProxy to do that again, and I don't think I can do that better than Proxomitron.

I see the memory leak too. On my machine it even stops to accept new connections when the RAM goes up to 60MB. ProxHTTPSProxy doesn't cache web content. for now I have no idea how it happens.

Hi whenever, I know that Privoxy or Proxomitron can block url but if I remember corrrectly, both software can only see domain of the url and only block at domain level, cannot do anything with path level, example:

https://google.com/123456/ads

If I use Privoxy (I set my browser HTTPS proxy to Privoxy 127.0.0.1:8118), there is no way block if google.com contains ads using .google.com/.*?ads, but domain level use .google.com work because the path path is encrypted by server side so Privoxy (maybe Proxomitron) will see something like this https://google.com/malwkmaldwmdklamwdlkz= instead https://google.com/123456/ads, in this article, Privoxy's author explained: http://www.privoxy.org/faq/misc.html

Quote:4.15. How can Privoxy filter Secure (HTTPS) URLs?

Since secure HTTP connections are encrypted SSL sessions between your browser and the secure site, and are meant to be reliably secure, there is little that Privoxy can do but hand the raw gibberish data though from one end to the other unprocessed.

The only exception to this is blocking by host patterns, as the client needs to tell Privoxy the name of the remote server, so that Privoxy can establish the connection. If that name matches a host-only pattern, the connection will be blocked.

And because we set our HTTPS Proxy of our browser to 127.0.0.1:8079 is ProxHTTPSProxy, so probably I think ProxHTTPSProxy will download all content and then send to Privoxy/Proxomitron so I think we will still have to waste our bandwidth ?