Enable HTTPS Filtering: Chain ProxHTTPSProxyMII to Privoxy (OS X Tutorial)

[Faxopita]

ProxHTTPSProxyMII (click link to learn more)
Created to provide modern nag-free HTTPS connections for an HTTP proxy.



Credits & Acknowledgements: whenever, JJoe, Cattleya, GunGunGun and other contributors to ProxHTTPSProxyMII

Installation Steps

Step 1: if not previously done, install the Command Line Tools (click link to learn more); run in Terminal (begin with sudo if necessary):

Code:

xcode-select --install

If you'd like to learn more about this step, visit this excellent tutorial.

Before installing Homebrew, make sure you're not using C shell. To know which one Terminal app. is currently using, input

Code:

echo $SHELL

If you see /bin/csh, then change shell by simply typing bash. Changing shell to Bash—or any other that's not C—is only necessary to allow Brew installation using Ruby.

Step 2: install Homebrew from http://brew.sh or visit this GitHub page.

If you encountered an issue during the installation, install the full Xcode package and try again.

After installing Homebrew, perform these commands:

Code:

sudo chown $(whoami):admin /usr/local
brew doctor
brew update

Above, it is not recommended to run chown command with option -R for security reasons. However, if you run into tons of problems–such as being not able to upgrade packages installed via Homebrew–then sadly, in very last resort, you use the option -R. It is worth mentioning that ClamXav, for example, does not appreciate–for very legitimate reasons–the use of this recursive mode from the /usr/local/ folder.

Note that the first command above—beginning with sudo—has to be run after every permission repair. If not, Homebrew will run into trouble during manual package upgrade.

Step 3: Preparing for ProxHTTPSProxy installation. Run these commands:

Code:

brew install python3 --with-brewed-openssl
pip3 install colorama
pip3 install urllib3
pip3 install pyOpenSSL
pip3 install PySocks

Step 4: obtain ProxHTTPSProxyMII Python scripts from the “ProxHTTPSProxyMII: Development” thread.

Step 5: add ProxHTTPSProxy's CA.crt to OS X Keychain! If using Firefox, use it to import this certificate through its advanced preferences: show Firefox certificates and visit Authorities tab, then import…

Step 6: set HTTPS Proxy to 127.0.0.1:8079 instead of 127.0.0.1:8118 (the port number used by Privoxy); see OS X Network Preferences, under Proxies tab. You may equally input this command in Terminal (replace Wi-Fi by Ethernet or "Thunderbolt Ethernet" if necessary):

Code:

sudo networksetup -setsecurewebproxy Wi-Fi 127.0.0.1 8079

Step 7
Add these lines to user.filter file:

Code:

CLIENT-HEADER-TAGGER: tagger4https
[email protected]^.*Tagged:.*ProxHTTPSProxyMII.*FrontProxy.*$@$[email protected]

Add these lines to user.action file:

Code:

{ +client-header-tagger{tagger4https} }
/
{ +forward-override{forward 127.0.0.1:8081} }
TAG:.*?ProxHTTPSProxyMII

Step 8: open config.ini file—inside ProxHTTPSProxyMII folder. Locate ProxAddr and set port to 8118, thus replacing 8080.

Code:

ProxAddr = http://localhost:8118

Step 9: just be in your ProxHTTPSProxyMII folder and Launch ProxHTTPSProxy with the following command:

Code:

./ProxHTTPSProxy.py

If this doesn't work:

Code:

Python3.X ProxHTTPSProxy.py

(if, for example, Python version 3.X has been installed)

Python binaries should be located inside…

Code:

/usr/local/bin/

Only for those who want to go a little deeper… Add above path to environment variable $PATH and rehash executables:

Code:

hash -r

Alternatively, you can type:

Code:

/usr/local/bin/python3.X ProxHTTPSProxy.py

(replace X with your version number)

You're done!

Note 1: for existing ProxHTTPSProxyMII's installation (before version 1.4), you must…

1. Upgrade to the latest urllib3 version :
   
   Code:
   

   pip3 install -U urllib3
   urllib3/1.14 is known to work well with ProxHTTPSProxyMII. So, if you have a problem with latest urllib3 release:
   
   Code:
   

   pip3 uninstall urllib3
pip3 install urllib3==1.14
2. Install PySocks:
   
   Code:
   

   pip3 install PySocks

Note 2: use ProxHTTPSProxyMII's config file to set exceptions under [SSL Pass-Thru]; make a backup of it for later upgrade.

Note 3: ProxHTTPSProxyMII folder can be located anywhere on your system. Just make sure you have a fairly quick access to config.ini

Note 4: for more info on ProxHTTPSProxyMII, visit http://www.proxfilter.net/proxhttpsproxy/index.html, ProxHTTPSProxyMII thread and GitHub page.

Note 5: activating temporary root access on a regular user account.

1. Log on to your Administrator account.
2. Open Terminal and execute: sudo visudo (and input your admin password)
3. Under # User privilege specification, add this: your_regular_username ALL=(ALL) ALL
   

You might want to visit this page as well: https://support.apple.com/en-gb/HT204012

Note 6: if you intend to use Homebrew's upgrade command, it might be worth to check this article first. This one is interesting as well! System Integrity Protection in OS X El Capitan could cause some issues.

-–—

Minuscule donations are always appreciated…

Code:

BTC --> 34WKogWorDoReJ2MSxw8rTsrGD87VMAPJY
BCH --> 1AXwyMdtMFZktZPvXScC58ESUZXptmjvge
DASH -> XusJsETR6PwDnG4Gde7cvGeRhXzUJFSxtD
ETH --> 0xb829FA99AA9AB31C32590dbc88B837bC5D91453e
ETC --> 0x059F128357331c346Ad2E23F95a4639beC3f0b3a
LTC --> MK7vxk93A1M6HHAYT38W8NPJSb8zANqCia
ZEC --> t1JNCuxdZEWUPBQiAzxZPUMqb4BM87sxs9H
DOGE -> DBPAUuCaez4JYGobAn4RHNNhFXwa9u1W6N
STRAT > SgG6jAHuxQfzW1QBaWyQRVdCdSq514BcyM

[cattleyavns]

Great job!

[diem]

Hi Faxopita,

Many thanks for detailing the steps for OS X installation - I'd like to try it out. Could you confirm whether these are the correct uninstallation steps though? I've not been able to find uninstall instructions elsewhere on this forum, and they're a pre-requisite for me; I need to know how to walk backwards before I'm prepared to walk forwards

1. Delete the folder in which ProxHTTPSProxyMII has been placed (am I correct to assume that all the files for the program are in that one folder and that no config is placed elsewhere?).
2. Delete the ProxHTTPSProxyMII cache - (does it cache? Where is this located?)
3. Uninstall the Python installed by Homebrew (does doing so automatically also uninstall the modules that pip3 installs?)

Any other steps I've missed?


Many thanks again for your efforts on getting this tool working on OS X,

Ian

[Faxopita]

Hello Diem,
thanks for your interest in enhancing the power of Privoxy with the fabulous ProxHTTPSProxyMII! Below, the steps I would carry out in order to clean up everything.

Step 1: Uninstalling Homebrew—with all installed packages.

1. Run this command in Terminal:
   
   Code:
   

   rm -rf /usr/local/Cellar /usr/local/.git && brew cleanup
2. Then, visit Homebrew's FAQ and look for Section How do I uninstall Homebrew?; running the ruby command in Terminal is enough.
   

This step will delete any trace of Python version as well as the pip command.

Step 2: If necessary, remove any python folder under path…

Code:

/usr/local/lib/

Step 3: Restore the original group and file permissions for the following folder:

Code:

sudo chmod 0755 /usr/local/
sudo chgrp wheel /usr/local/

Step 4: Finally, you can just drag the folder ProxHTTPSProxyMII—in which you were presumably running JJoe's Python script—to the trash can.

I hope that helps.

[JJoe]

(Oct. 12, 2015 10:22 PM)Faxopita Wrote:  Finally, you can just drag the folder ProxHTTPSProxyMII—in which you were presumably running JJoe's Python script—to the trash can.

Whoa! whenever is the author.

[Faxopita]

My apologies to whenever. Anyway, you're both the stars of the forum.

I'm just a “facilitator”.

[Faxopita]

For those who have not managed to properly configure ProxHTTPSProxyMII on OS X yet—and were too shy to let me know—my most sincere apologies. I've forgotten to mention Step 7… Please, try again. Don't get discouraged! Tested successfully on Yosemite and El Capitan.

[Faxopita]

Make ProxHTTPSProxy Start Automatically at Session Login

Setting up the Dæmon

1. Retrieve the attached archive to this post.
2. Create the following directory:
   
   Code:
   

       mkdir -p /usr/local/var/log/ProxHTTPSProxy/
   (this to allow you view ProxHTTPSProxy output)
3. Copy ProxHTTPSProxy.sh script to your favourite script folder.
4. Modify the script so it can locate both your own Python binary and ProxHTTPSProxy folders; currently set are my own ones.
5. Copy com.prxbx.ProxHTTPSProxy.plist to ~/Library/LaunchAgents/
6. Edit the above .PLIST file and replace [INSERT_ABSOLUTE_PATH] by the path—in absolute terms—leading to the script ProxHTTPSProxy.sh
7. Service should start automatically at next login, but you can run it now:
   • launchctl load com.prxbx.ProxHTTPSProxy.plist
   • To stop the service: launchctl unload com.prxbx.ProxHTTPSProxy.plist
   • To verify the service is loaded: launchctl list | grep "prxbx"
     

View ProxHTTPSProxy Output in Your Browser

1. Create the following directory:
   
   Code:
   

       mkdir -p /usr/local/var/log/Privoxy2GoTTY/
   (just in case, some logs to help isolate an issue)
2. Install GoTTY via Homebrew:
   
   Code:
   

       brew update && brew install gotty
3. Copy gotty.http.requests.plist to folder ~/Library/LaunchAgents/
4. In the .PLIST file, make the necessary change as per [INSERT_ABSOLUTE_PATH].
5. Copy Privoxy2GoTTY.sh to your script folder. Currently set to listen on Port 8082; amend if necessary.
6. Service should start at next login, but you can run it now: launchctl load gotty.http.requests.plist
7. Open a browser and access http://127.0.0.1:8082
   

Installation works under regular user session; no need to use sudo. If a password is being asked, abort and perform the following commands:

Code:

sudo chown $(whoami):admin /usr/local
brew doctor
brew update

Above, it is not recommended to run chown command with option -R for security reasons. However, if you run into tons of problems–such as being not able to upgrade packages installed via Homebrew–then sadly, in very last resort, you use the option -R. It is worth mentioning that ClamXav, for example, does not appreciate–for very legitimate reasons–the use of this recursive mode from the /usr/local/ folder.

-–—

Minuscule donations are always appreciated…

Code:

BTC --> 34WKogWorDoReJ2MSxw8rTsrGD87VMAPJY
BCH --> 1AXwyMdtMFZktZPvXScC58ESUZXptmjvge
DASH -> XusJsETR6PwDnG4Gde7cvGeRhXzUJFSxtD
ETH --> 0xb829FA99AA9AB31C32590dbc88B837bC5D91453e
ETC --> 0x059F128357331c346Ad2E23F95a4639beC3f0b3a
LTC --> MK7vxk93A1M6HHAYT38W8NPJSb8zANqCia
ZEC --> t1JNCuxdZEWUPBQiAzxZPUMqb4BM87sxs9H
DOGE -> DBPAUuCaez4JYGobAn4RHNNhFXwa9u1W6N
STRAT > SgG6jAHuxQfzW1QBaWyQRVdCdSq514BcyM

[Faxopita]

View Privoxy Status in OS X Menu Bar

1. In all scripts, replace "Thunderbolt Ethernet" by Wi-Fi if necessary. Other alternative: complete with configuring other device ports at the same time.
2. I did not find any better solution than activating Privoxy logging. If issues in detecting Privoxy status, unable debug modes 512, 4096, 8192 or try any one of those.
3. It might be wise to empty the log file once in a while… For example, at system startup with a cron job:
   
   Code:
   

   @reboot echo $(grep "toggle?\(mini=y&\)\?set=\(enable\|disable\)" /private/var/log/privoxy/logfile.log | tail -1) > /private/var/log/privoxy/logfile.log
   (command to empty the log file but the latest Privoxy status)
   
   Note: you might need to claim the ownership of the log file if you choose to add the cron job as a regular user. Make sure Privoxy has write access to the log file too! Otherwise, it will crash.
   

Install BitBar via Homebrew Cask: brew cask install bitbar

Amend [script folder] in those scripts:

• Privoxy_Status.5s.sh (menu bar item refreshed every 5 sec.)
• HTTPS_Port_Status.1m.sh (menu bar item refreshed every one min.)
  

Both located inside your BitBar folder.

-–—

Caution with Script `Privoxy_Status.sh`

If you're not in the habit of clearing Privoxy's logfile once in a while—you may want to keep it for later examination—I suggest you “rotate” it daily in order to reduce CPU usage every time the command grep is called—every x seconds. Highly recommended since using such a command-line utility cyclically on a logfile of 100's of MB (past a million entries) can have a serious impact on your CPU during a few seconds. Not good if you're on battery then!

In my case, I decided to have Privoxy's logfile content flushed out daily to some other file located inside the userland part of the system; you can create a new cron job—`crontab -e` in Terminal—and add this line:

Code:

0 20 * * * cat /private/var/log/privoxy/logfile.log >> /usr/local/var/log/Privoxy/logfile.bak && echo $(grep "toggle?\(mini=y&\)\?set=\(enable\|disable\)" /private/var/log/privoxy/logfile.log | tail -1) > /private/var/log/privoxy/logfile.log

If you just want to clear Privoxy's logfile content at 8 p.m. daily:

Code:

0 20 * * * echo $(grep "toggle?\(mini=y&\)\?set=\(enable\|disable\)" /private/var/log/privoxy/logfile.log | tail -1) > /private/var/log/privoxy/logfile.log

(if every hour, replace `20` by `*`)

Same job but at reboot time only:

Code:

@reboot    echo $(grep "toggle?\(mini=y&\)\?set=\(enable\|disable\)" /private/var/log/privoxy/logfile.log | tail -1) > /private/var/log/privoxy/logfile.log

To list your cron jobs: `contrab -l`.

-–—

Minuscule donations are always appreciated…

Code:

BTC --> 34WKogWorDoReJ2MSxw8rTsrGD87VMAPJY
BCH --> 1AXwyMdtMFZktZPvXScC58ESUZXptmjvge
DASH -> XusJsETR6PwDnG4Gde7cvGeRhXzUJFSxtD
ETH --> 0xb829FA99AA9AB31C32590dbc88B837bC5D91453e
ETC --> 0x059F128357331c346Ad2E23F95a4639beC3f0b3a
LTC --> MK7vxk93A1M6HHAYT38W8NPJSb8zANqCia
ZEC --> t1JNCuxdZEWUPBQiAzxZPUMqb4BM87sxs9H
DOGE -> DBPAUuCaez4JYGobAn4RHNNhFXwa9u1W6N
STRAT > SgG6jAHuxQfzW1QBaWyQRVdCdSq514BcyM

[Faxopita]

Hello,

I have revised the tuto.

(Jan. 01, 2017 09:56 PM)GORTOP Wrote:  Hi

Make ProxHTTPSProxy Start Automatically at Session Login.
It is not clear 3 and 4 points.

[Faxopita]

Allow Privoxy to Block All Requests if VPN Turned Off!

Some VPN apps offer the option to block Internet access if, for example, they happen to be disabled accidentally. I don't have this option, so I decided to create a tiny script to do that job (to some extent) with the help of Privoxy. The script automatically blocks all HTTP/HTTPS requests if your VPN is down. To do so, it simply uncomments these lines if TUN device is not returned in ifconfig output:

Code:

# { +block{VPN is not running. Turn it on first!} }
#  /

Be warned that the script will not deal with layers other than HTTP/HTTPS. So, you might still be able to use, for example, SMTP or FTP if your VPN is down. Any exception rule under `{ -block }` put after the above enforced block action will still allow for Internet access even though your VPN is down; useful to let you at least connect to your VPN, by the way…

If you are interested, then this post is for you, but first: retrieve the attached archive to this post and back up immediately your `user.action` file!

Design choice: the script does its own little job only if the hidden file `.AllVPN` is present in your Home directory. It's a kind of a switch. It's a choice I made, but you can certainly modify the script so it doesn't take account of this switch! So, for now, create it with your favourite Terminal:

Code:

cd `echo $HOME`
touch .AllVPN

Amend your own `user.action` file.

Add this section at the end of your `user.action` file:

Code:

# { +block{VPN is not running. Turn it on first!} }
#  /

  { -block }
# However, you have to whitelist your VPN provider here…

Also, be warned the script requires you to have write permissions on `user.action` file; in Terminal `cd` to your Privoxy directory, then…

Code:

sudo chmod 664 user.action
sudo chown `echo $USER`:admin user.action

Amend script AllVPN.

1. The script uses GNU sed; I got it with Homebrew. Perhaps you'd like to use the version installed with the base system. If so, you might find the need to tweak the corresponding syntax in the script.
   
   Code:
   

   brew info gnu-sed
brew home gnu-sed
brew install gnu-sed
2. Allocate your own paths to `pri_path` and `sed_path`.
3. Copy AllVPN.sh script to your script folder.
   

Setting up the dæmon.

1. Copy privoxy.vpn.bridge.plist to ~/Library/LaunchAgents; it's the one attached to your Home directory.
2. Edit the above .PLIST file and replace [INSERT_ABSOLUTE_PATH] by the path—in absolute terms—leading to the script `AllVPN.sh`.
3. Service should start automatically at next login, but you can run it now; `cd` first to `~/Library/LaunchAgents`, then…
   • launchctl load privoxy.vpn.bridge.plist
   • To stop the service: launchctl unload privoxy.vpn.bridge.plist
   • To verify the service is loaded: launchctl list | grep "bridge"
     

-–—

Minuscule donations are always appreciated…

Code:

BTC --> 34WKogWorDoReJ2MSxw8rTsrGD87VMAPJY
BCH --> 1AXwyMdtMFZktZPvXScC58ESUZXptmjvge
DASH -> XusJsETR6PwDnG4Gde7cvGeRhXzUJFSxtD
ETH --> 0xb829FA99AA9AB31C32590dbc88B837bC5D91453e
ETC --> 0x059F128357331c346Ad2E23F95a4639beC3f0b3a
LTC --> MK7vxk93A1M6HHAYT38W8NPJSb8zANqCia
ZEC --> t1JNCuxdZEWUPBQiAzxZPUMqb4BM87sxs9H
DOGE -> DBPAUuCaez4JYGobAn4RHNNhFXwa9u1W6N
STRAT > SgG6jAHuxQfzW1QBaWyQRVdCdSq514BcyM