Adapting “hosts” file block lists to Privoxy's way of blocking…

[Faxopita]

Cattleyavns add-on to convert hosts entries into Privoxy language is a nice tool to do the conversion quickly.

For those who want to reduce the resulting file size, the script from post #3's download link could be a potential alternative. Sure, it can be improved by any brighter mind reading this post and I hope it will happen.

The purpose of this script (hpHosts Maintainer) is to consolidate the block list in order to obey to the Privoxy's blocking style. So, I thought it might be good to remind what this style is about…

Imagine those entries in a hosts file (we forget localhost for simplicity):

Code:

abc.com
www.abc.com
tuv.xyz.def.com
xyz.def.com
def.com
www23.ghi.com
jkl.mno.pqr.com
mno.pqr.com

The “hpHosts Maintainer” script will turn the above entries into:

Code:

.abc.com
.def.com
.ghi.com
.mno.pqr.com

Privoxy would accept the entries as such from the hosts file (Cattleyavns), but would provide the exact same blocking actions as the above consolidated version created by the “Maintainer” script.

Now, the script goes a little further into simplifying entries. Imagine those entries in a hosts file:

Code:

www.zulu
www.com
mars.jupiter.saturn.uranus.neptune.com
jupiter.saturn.uranus.neptune.com
saturn.uranus.neptune.com
jupiter.uranus.neptune.com
mars.uranus.neptune.com

The “hpHosts Maintainer” script will turn the above entries into:

Code:

.zulu
.www.com
.uranus.neptune.com

“zulu” is not a known TLD. No problem, then. However, “www.com” cannot be turned into “.com” (a well-known TLD)—otherwise, all “.com” domains will be blocked; so, the script just prefixes with a dot. The good thing of prefixing with a dot, is that not only the bad “www.com" website will be blocked, but also any of its subdomains. Visit http://hosts-file.net/default.asp?s=www.com

Finally, I have made the deliberate choice to limit blocking to the first subdomain of the website/server. However, brighter minds of this forum may not like this way of proceeding. That's why the script also moves out cloud-based addresses to another file so they're not touched by URL compression. For example: “.tracker.not_a_bad_side.wordpress.com” will not be compressed to “.not_a_bad_side.wordpress.com”. In the above example, if “neptune.com” is not a cloud-based server, then the resulting entry is “.uranus.neptune.com”. The script compressed five “neptune.com”-related hosts entries into just one. The rational being that if “mars”, “jupiter” and “saturn” are bad, it can mean that “uranus” is bad as well. Then, we could assume that “neptune” is bad too, but for generic purpose, the script leave the URL down to the first subdomain.

One last thing. hosts file entries such as:

Code:

pub.casino-making-money.com
chocolate.article.diet.com
article.diet.com
viagra.pill.com

will be turned by the script into:

Code:

.casino-making-money.com
.diet.com
.pill.com

No need to go as far as the subdomain for these “scam”-related websites.

One of the reasons why I wanted to give away my script is to make it available for all platforms, not only for unix-based. Anyone wanted to work on it, improve it and make it a binary version or Python-based is welcome. If you do so, just append my member name next yours for future celebrity… Just in case!