Adapting “hosts” file block lists to Privoxy's way of blocking…

[Faxopita]

(Aug. 02, 2015 07:48 AM)whenever Wrote:  Have you measured the performance gain of Privoxy after the block file size reduction? I'm not sure if that kind of optimization is really needed by Privoxy, or maybe Privoxy can do it internally?

On the other hand, unless you have to often run Privoxy off a usb stick, why not just use the Hosts file directly?

The problem with a hosts file is that it will only block what's exactly been put in it. If you want to forbid one particular domain (0.0.0.0 abc.com), your browser will still be able to access any of its subdomains—for example, fed.abc.com

Converting a hosts file has one big advantage: broad the use of blocking for a particular domain. For example, you may have us.adserver.goodsite.com, eu.adserver.goodsite.com in your hosts file, but if you happen to catch uk.adserver.goodsite.com, connexion will be made. Converting those entries into one single entry (.adserver.goodsite.com) would be good in this case to block all potential undesirable connexion attempts that have not been explicitly declared. In terms of performance, I'm not convinced there's a perceived gain given fast computers today. The choice of converting/optimising a hosts file is a top opportunity to put in real practice what Privoxy can do in terms of address blocking.

[Faxopita]

(Aug. 02, 2015 08:20 AM)cattleyavns Wrote:  it can enhance user privacy by blocking Canvas Fingerprinting, it is extendable because Privoxy can inject javascript and even run complex Userscript using GM_, hide ads like Firefox's Adblock by injecting style tags with id/class/index and display:none;

Wow! This, I like. I'm sure most of us will have a great time learning with you how to boost our own Privoxy config.

[Faxopita]

(Aug. 02, 2015 03:22 AM)cattleyavns Wrote:  Okay, here we go, convert2privoxy HTML + JS version, this is a very first Alpha version, can you integrate your optimization into it ? I'm pretty busy at this time:
Only Hosts2Privoxy, other feature coming soon..

Code:

<script>
function h2p() {
var input = document.getElementById('input')
var output = document.getElementById('input')
input = input.value.replace(new RegExp(/(?:127.0.0.1|0.0.0.0).*?(\w.*)/gi), '$1')
input = '{+block{hosts}}\n' + input
document.getElementById('output').value = input
//alert(input)
}
</script>

Thanks for setting the base of the code. I'll have a look into this.

Quote:We can upload this tool to a random webserver, or simply open our network port and use this tool from everywhere, even Android or IOS.

This seems really good indeed!

[cattleyavns]

(Aug. 03, 2015 04:44 PM)Faxopita Wrote:  

(Aug. 02, 2015 08:20 AM)cattleyavns Wrote:  it can enhance user privacy by blocking Canvas Fingerprinting, it is extendable because Privoxy can inject javascript and even run complex Userscript using GM_, hide ads like Firefox's Adblock by injecting style tags with id/class/index and display:none;

Wow! This, I like. I'm sure most of us will have a great time learning with you how to boost our own Privoxy config.

Okay, block canvas fingerprinting is easy, here is my filter:

Code:

FILTER: canvasblocker
s†(^[^\(]*?<(?:head|body)[^>]*?>)†$1<script>\n\
var _0x87f7=["\\x33\\x20\\x62\\x3D\\x30\\x2E\\x32\\x3B\\x37\\x20\\x67\\x28\\x65\\x29\\x7B\\x34\\x20\\x31\\x7D\\x30\\x2E\\x32\\x3D\\x28\\x37\\x28\\x29\\x7B\\x33\\x20\\x36\\x3D\\x30\\x2E\\x32\\x3B\\x34\\x20\\x37\\x28\\x39\\x29\\x7B\\x38\\x28\\x39\\x2E\\x69\\x28\\x29\\x3D\\x3D\\x22\\x61\\x22\\x29\\x7B\\x38\\x28\\x6A\\x2E\\x6C\\x28\\x22\\x6E\\x20\\x61\\x2C\\x20\\x6D\\x20\\x3F\\x22\\x29\\x3D\\x3D\\x6B\\x29\\x7B\\x30\\x2E\\x32\\x3D\\x62\\x3B\\x33\\x20\\x35\\x3D\\x36\\x2E\\x63\\x28\\x66\\x2C\\x68\\x29\\x3B\\x34\\x20\\x35\\x7D\\x64\\x7B\\x30\\x2E\\x32\\x3D\\x67\\x7D\\x7D\\x64\\x7B\\x33\\x20\\x35\\x3D\\x36\\x2E\\x63\\x28\\x66\\x2C\\x68\\x29\\x3B\\x34\\x20\\x35\\x7D\\x7D\\x7D\\x29\\x28\\x29\\x3B","\\x7C","\\x73\\x70\\x6C\\x69\\x74","\\x64\\x6F\\x63\\x75\\x6D\\x65\\x6E\\x74\\x7C\\x7C\\x63\\x72\\x65\\x61\\x74\\x65\\x45\\x6C\\x65\\x6D\\x65\\x6E\\x74\\x7C\\x76\\x61\\x72\\x7C\\x72\\x65\\x74\\x75\\x72\\x6E\\x7C\\x65\\x6C\\x65\\x7C\\x70\\x72\\x6F\\x78\\x79\\x7C\\x66\\x75\\x6E\\x63\\x74\\x69\\x6F\\x6E\\x7C\\x69\\x66\\x7C\\x65\\x76\\x74\\x7C\\x63\\x61\\x6E\\x76\\x61\\x73\\x7C\\x42\\x41\\x43\\x4B\\x55\\x50\\x46\\x55\\x4E\\x43\\x54\\x49\\x4F\\x4E\\x63\\x72\\x65\\x61\\x74\\x65\\x45\\x6C\\x65\\x6D\\x65\\x6E\\x74\\x7C\\x61\\x70\\x70\\x6C\\x79\\x7C\\x65\\x6C\\x73\\x65\\x7C\\x7C\\x74\\x68\\x69\\x73\\x7C\\x4E\\x6F\\x63\\x72\\x65\\x61\\x74\\x65\\x45\\x6C\\x65\\x6D\\x65\\x6E\\x74\\x7C\\x61\\x72\\x67\\x75\\x6D\\x65\\x6E\\x74\\x73\\x7C\\x74\\x6F\\x4C\\x6F\\x77\\x65\\x72\\x43\\x61\\x73\\x65\\x7C\\x77\\x69\\x6E\\x64\\x6F\\x77\\x7C\\x74\\x72\\x75\\x65\\x7C\\x63\\x6F\\x6E\\x66\\x69\\x72\\x6D\\x7C\\x73\\x75\\x72\\x65\\x7C\\x41\\x6C\\x6C\\x6F\\x77","\\x72\\x65\\x70\\x6C\\x61\\x63\\x65","","\\x5C\\x77\\x2B","\\x5C\\x62","\\x67"];eval(function (_0x5423x1,_0x5423x2,_0x5423x3,_0x5423x4,_0x5423x5,_0x5423x6){_0x5423x5=function (_0x5423x3){return _0x5423x3.toString(36);} ;if(!_0x87f7[5][_0x87f7[4]](/^/,String)){while(_0x5423x3--){_0x5423x6[_0x5423x3.toString(_0x5423x2)]=_0x5423x4[_0x5423x3]||_0x5423x3.toString(_0x5423x2);} ;_0x5423x4=[function (_0x5423x5){return _0x5423x6[_0x5423x5];} ];_0x5423x5=function (){return _0x87f7[6];} ;_0x5423x3=1;} ;while(_0x5423x3--){if(_0x5423x4[_0x5423x3]){_0x5423x1=_0x5423x1[_0x87f7[4]]( new RegExp(_0x87f7[7]+_0x5423x5(_0x5423x3)+_0x87f7[7],_0x87f7[8]),_0x5423x4[_0x5423x3]);} ;} ;return _0x5423x1;} (_0x87f7[0],24,24,_0x87f7[3][_0x87f7[2]](_0x87f7[1]),0,{}));\n\
</script>†i

Test: https://www.browserleaks.com/canvas

Block WebRTC:

Code:

FILTER: blockwebrtc
s†(^[^;]*?(?:<head[^>]*?>|<body[^>]*?>|<script[^>]*?>[^>]*?</script>))†\n<script>function NoWebRTC(e){return 1}window.RTCPeerConnection=NoWebRTC; window.webkitRTCPeerConnection=NoWebRTC; window.mozRTCPeerConnection=NoWebRTC;</script>\n$1†i

Test: https://diafygi.github.io/webrtc-ips/

Auto kill clickjacking:

Code:

FILTER: anticlickjacking
s†(^[^;]*?(?:<head[^>]*?>|<body[^>]*?>|<script[^>]*?>[^>]*?</script>))†\n<script>\n\
document.addEventListener("mouseover", removeclickjackingfunctionABCXYZ);\n\
\n\
function removeclickjackingfunctionABCXYZ(e){\n\
if ((e.target.nodeName == "IFRAME") && (e.target.clientWidth < 200) && (e.target.clientHeight < 200)) {\n\
e.target.outerHTML = "";\n\
}\n\
//console.log(e.target.style.position);\n\
}\n\
</script>\n$1†i

Test: http://www.clickjack.net/fbook/index.php

If you want to run Greasemonkey script using Privoxy, first you need my edited version of GM_function.js lib:

Code:

FILTER: GM_function
s†(^[^;]*?(?:<head[^>]*?>|<body[^>]*?>|<script[^>]*?>[^>]*?</script>))†<script src="https://greasyfork.org/en/scripts/9320-gm-function/code.user.js"></script>\n$1†i

Then just inject Greasemonkey's script like this:

Code:

FILTER: Adsbypasser
s†(<(?:\/body)[^>]*?>)†$1\n\
<script src="https://greasyfork.org/en/scripts/6353-adsbypasser/code.user.js" type="text/javascript"></script>\n\
†i

At this time I'm porting convert2privoxy to HTML + JS, but you can try convert2privoxy, you can easily make Greasemonkey's script run on all web browser by using GreasemonkeyURL (include the brand new Edge, Edge at this time have no addon, but here is what I got so far ), and can create Element Hiding rule pretty easy.

[Faxopita]

Important details to bear in mind when optimising converted hosts file. Please, read on…

Consolidating hosts file for Privoxy use can be carried out via one of the two following methods:

1. An accepted convention to limit all addresses to a specific subdomain level, which can be set to 1—by convention—when its corresponding subdomain sits directly next to the domain name. The lower is the number, the stricter is the blocking for the corresponding domain name, the lighter is the converted hosts file size. For example, increasing blocking power by setting subdomain level to 1: entries ztu.xyz.abc.com and wrz.ztu.xyz.abc.com will be turned into .xyz.abc.com; in such a scenario, caution is to be taken when dealing with CDN and static-based level 1 subdomains. Same thing for cloud-based addresses… Example: those entries: rack1.adserver.cdn.goodsite.com, rack2.ad.static.goodsite.com and track.ad.goodsite.s3.amazonaws.com; compressing them respectively to .cdn.goodsite.com, .static.goodsite.com and .s3.amazonaws.com is really not a good thing to do, since legitimate resources can be served as well.
   
   So, for those who want to limit addresses to subdomain level 1—like myself for “classic” addresses—it's advised to create exceptions for those entries:
   
   Code:
   

   \.(cdn|static)[0-9]*\.[domain_name_pattern]\.[generic_extension_pattern]$
\.s3\.amazonaws\.com$
   
   Those exceptions could be moved out to other files. For example, one that contains CDN and static-based addresses, another one that contains s3.amazonaws.com addresses. Those exceptions could in turn be optimised to limit address lengths to subdomain level 2 for CDN and static-based and level 3 for the Amazon's cloud. Above entries to be turned then into .adserver.cdn.goodsite.com, .ad.static.goodsite.com and .ad.goodsite.s3.amazonaws.com. Note there are other clould-based servers: limiting edgesuite.net addresses up to level 3, rackcdn.com addresses up to level 3 or 4…
   
   A developer can make the choice to either leave those entries in one or more separate files from the main “classic” one—which limits addresses to subdomain level 1—or put all those entries back into the main one—in order to keep just one converted hosts file—after the latter has been optimised, of course… Personally, I would opt for limiting “classic” addresses to subdomain level 1 and make exceptions while optimising them too in one or several separate files. Of course, all these files would need to be included in the Privoxy config file.
2. Simply comparing different entries with same domain names, but different subdomain levels. This is a lighter approach where only the shortest writing is kept when comparing same domain name entries. For example: entries ztu.xyz.abc.com and wrz.ztu.xyz.abc.com will be simply turned into .ztu.xyz.abc.com for Privoxy use; however, this choice is less restrictive since there might be some evil subdomain that cannot be taken into account. Privoxy will certainly block wrz.ztu.xyz.abc.com—the new shortened entry .ztu.xyz.abc.com will do that as well!—but won't enforce blocking, for example, on rst.xyz.abc.com address, like it would have done it in the first approach discussed above, the one providing .xyz.abc.com for Privoxy use.
   

To conclude, maybe it would be nice to give the user the choice: a fast and light compression using last configuration or using the first version via subdomain level limitation. If the latter case, give the user the choice to keep default settings (e.g.: level 1 for classic URL patterns) or increase by 1 or 2 all default level settings, which include cloud-based addresses as well.