Post Reply 
Andrew's Security Filter(s) v5.62 (May 10, 2009)
Jun. 13, 2008, 03:31 PM
Post: #46
RE: Andrew's Security Filter(s) v5.55 (June 11, 2008)
Try loading the attached "test.cfg" into Proxomitron ("File => Load Config File").

I'm thinking some of the other filters might be conflicting with the Javascript display code. I've tested this test.cfg file fine with Firefox, Opera and IE, and they all work =\


Attached File(s)
.cfg  test.cfg (Size: 7.97 KB / Downloads: 766)
Visit this user's website
Add Thank You Quote this message in a reply
Jun. 13, 2008, 07:30 PM
Post: #47
RE: Andrew's Security Filter(s) v5.55 (June 11, 2008)
still the same , may be this is Java`s version problem. I have version 6 update 5 installed on my system.

mystics
Quote this message in a reply
Jun. 14, 2008, 02:45 AM
Post: #48
RE: Andrew's Security Filter(s) v5.55 (June 11, 2008)
Looks like a lot has been happening with this filter while I've been incommunicado.

Kye-U Wrote:I wonder... in case there are pages that don't have either of the closing tags (</body>|</html>), should I just match the end of the page with (^*?) and then insert the scripts?

Maybe some version of this could work:
Code:
| (</body > </html > (^*?)|<html > (^*?)|(^*?))$STOP()
(
($TST(andrew=yes)|$TST(bypassing=1))
$SET(1=\r\n\r\n<script type="text/javascript" src="http://local.ptron/andrew.js"></script>\r\n
<script type="text/javascript">\r\n
prx_a_array($GET(scriptn), $GET(noscriptn), $GET(iframen), $GET(appletn), $GET(embedn), $GET(objectn), $GET(noappletn), $GET(noembedn), $GET(noobjectn), "$GET(sep)", "$GET(bypassing)", "$GET(origlink)");\r\n
</script>\r\n
</body>\r\n
</html>\r\n
)
)

Kye-U Wrote:Thanks for the code, I've always wondered how to match the end of a page

Your welcome. Smile!

"Kye-U Wrote:Upon closer analysis, on one page, there was an <embed> tag nested inside an <object> tag. This just means that the whole <object ... </object> tag is killed, along with it the nested <embed> tag.

Object tags can be nested too. Sad

I've been wondering about this:
Is there a need to match tags inside allowed scripted?
Seems to me that if you trust a script, you might as well let it rip without any matches by this filter.

Matching tags in js can be difficult at best.
There's lots of ways for scripts to inject Dom nodes that can be quite difficult to detect.

If it's desirable to disable certain tags inside a script, maybe it would be more effective just to break the tags.
Use a list to replace "embed" or "object" with "foo;".
That should also foil methods like document.createElement("tagname") or similar methods.
It would probably also break document.write statements that had the replacement tagname.
Sadly, this technique wouldn't catch encoded tag names however.

This might even break html object|embed|applet tags even when not in a script.
You'd probably still have to match html tags such as iframe or noscript with a $NEST().

I haven't tried this yet, just thinking out loud.

z12
Add Thank You Quote this message in a reply
Jun. 16, 2008, 02:55 AM
Post: #49
RE: Andrew's Security Filter(s) v5.56 (June 15, 2008)
Guest Wrote:still the same , may be this is Java`s version problem. I have version 6 update 5 installed on my system.

mystics

I must admit, I'm out of ideas =\
If it still didn't work even with a "fresh", standalone config set, then there's something that's blocking the Javascript from loading, somehow.

z12 Wrote:Seems to me that if you trust a script, you might as well let it rip without any matches by this filter.

...

If it's desirable to disable certain tags inside a script, maybe it would be more effective just to break the tags.
Use a list to replace "embed" or "object" with "foo;".

Very valid point =] I guess once you allow a tag/tags, then they shouldn't be removed. By renaming the tags instead of removing the entire code from the page, I'd imagine the filtering speed would be increased (since the byte limit would be 10 bytes or so, instead of 32767).

My thinking is, if a tag needs to be obfusticated via Javascript, it's either an advertisement or a nuisance.
Visit this user's website
Add Thank You Quote this message in a reply
Jun. 16, 2008, 03:33 AM
Post: #50
RE: Andrew's Security Filter(s) v5.56 (June 15, 2008)
I worked on a filter that replaces tag names only, but it doesn't seem to work that well xD The filter below replaces offending tags with "foobar", but if you go on http://www.cnn.com, you'll see that it doesn't remove the contents of script tags.

I tried replacing foobar with div style="display:none;" and even tried to comment out the entire code with <!-- and -->, but only remembered that any nested allowed tags would be hidden.

Good test code would be <object><embed></object>. (with <embed> tags being allowed)

Code:
[Patterns]
Name = "Andrew's Security Filter - new method test"
Active = TRUE
URL = "(^$TST(a_bypass=*.ALL.*))$TYPE(htm)\9($TST(\9=\8(\&|\?)trust=bypass)$SET(bypassing=1)$SET(origlink=\8)|$TST(\9=*\?*)$SET(sep=\q\&)|(^$TST(\9=*\?*))$SET(sep=\?)|)$SET(scriptn=0)$SET(noscriptn=0)$SET(appletn=0)$SET(noappletn=0)$SET(objectn=0)$SET(noobjectn=0)$SET(embedn=0)$SET(noembedn=0)$SET(iframen=0)"
Limit = 11
Match = "("
        "(^$TST(bypassing=1))"
        ""
        "("
        "<"
        "("
        "script$TST(a_script=1)$TST(($GET(scriptn)+)=$LST(Count)|*)$SET(scriptn=$GET(i))"
        "|noscript$TST(a_noscript=1)$TST(($GET(noscriptn)+)=$LST(Count)|*)$SET(noscriptn=$GET(i))"
        "|applet$TST(a_applet=1)$TST(($GET(appletn)+)=$LST(Count)|*)$SET(appletn=$GET(i))"
        "|object$TST(a_object=1)$TST(($GET(objectn)+)=$LST(Count)|*)$SET(objectn=$GET(i))"
        "|embed$TST(a_embed=1)$TST(($GET(embedn)+)=$LST(Count)|*)$SET(embedn=$GET(i))"
        "|iframe$TST(a_iframe=1)$TST(($GET(iframen)+)=$LST(Count)|*)$SET(iframen=$GET(i))"
        "|noapplet$TST(a_noapplet=1)$TST(($GET(noappletn)+)=$LST(Count)|*)$SET(noappletn=$GET(i))"
        "|noobject$TST(a_noobject=1)$TST(($GET(noobjectn)+)=$LST(Count)|*)$SET(noobjectn=$GET(i))"
        "|noembed$TST(a_noembed=1)$TST(($GET(noembedn)+)=$LST(Count)|*)$SET(noembedn=$GET(i))"
        ")"
        "$SET(1=<foobar)"
        "|"
        "</"
        "("
        "script$TST(a_script=1)"
        "|noscript$TST(a_noscript=1)"
        "|applet$TST(a_applet=1)"
        "|object$TST(a_object=1)"
        "|embed$TST(a_embed=1)"
        "|iframe$TST(a_iframe=1)"
        "|noapplet$TST(a_noapplet=1)"
        "|noobject$TST(a_noobject=1)"
        "|noembed$TST(a_noembed=1)"
        ")"
        ">"
        "$SET(1=</foobar>)"
        ")"
        "$SET(andrew=yes)"
        ")"
        ""
        "|(^*?)$STOP()"
        "("
        "($TST(andrew=yes)|$TST(bypassing=1))"
        "$SET(1=\r\n\r\n<script type="text/javascript" src="http://local.ptron/andrew.js"></script>\r\n"
        "<script type="text/javascript">\r\n"
        "prx_a_array($GET(scriptn), $GET(noscriptn), $GET(iframen), $GET(appletn), $GET(embedn), $GET(objectn), $GET(noappletn), $GET(noembedn), $GET(noobjectn), "$GET(sep)", "$GET(bypassing)", "$GET(origlink)");\r\n"
        "</script>)"
        ")"
Replace = "\1"

EDIT:

This one comments out the entire block of code (included any nested allowed tags =[):

Code:
[Patterns]
Name = "Andrew's Security Filter - new method test (comment out)"
Active = TRUE
URL = "(^$TST(a_bypass=*.ALL.*))$TYPE(htm)\9($TST(\9=\8(\&|\?)trust=bypass)$SET(bypassing=1)$SET(origlink=\8)|$TST(\9=*\?*)$SET(sep=\q\&)|(^$TST(\9=*\?*))$SET(sep=\?)|)$SET(scriptn=0)$SET(noscriptn=0)$SET(appletn=0)$SET(noappletn=0)$SET(objectn=0)$SET(noobjectn=0)$SET(embedn=0)$SET(noembedn=0)$SET(iframen=0)"
Limit = 11
Match = "("
        "(^$TST(bypassing=1))"
        ""
        "("
        "<"
        "("
        "script$TST(a_script=1)$TST(($GET(scriptn)+)=$LST(Count)|*)$SET(scriptn=$GET(i))"
        "|noscript$TST(a_noscript=1)$TST(($GET(noscriptn)+)=$LST(Count)|*)$SET(noscriptn=$GET(i))"
        "|applet$TST(a_applet=1)$TST(($GET(appletn)+)=$LST(Count)|*)$SET(appletn=$GET(i))"
        "|object$TST(a_object=1)$TST(($GET(objectn)+)=$LST(Count)|*)$SET(objectn=$GET(i))"
        "|embed$TST(a_embed=1)$TST(($GET(embedn)+)=$LST(Count)|*)$SET(embedn=$GET(i))"
        "|iframe$TST(a_iframe=1)$TST(($GET(iframen)+)=$LST(Count)|*)$SET(iframen=$GET(i))"
        "|noapplet$TST(a_noapplet=1)$TST(($GET(noappletn)+)=$LST(Count)|*)$SET(noappletn=$GET(i))"
        "|noobject$TST(a_noobject=1)$TST(($GET(noobjectn)+)=$LST(Count)|*)$SET(noobjectn=$GET(i))"
        "|noembed$TST(a_noembed=1)$TST(($GET(noembedn)+)=$LST(Count)|*)$SET(noembedn=$GET(i))"
        ")\7"
        "$SET(matched=yes)"
        "$SET(1=<!-- <[\7])"
        "|"
        "(<!--|-->)$TST(matched=yes)"
        "|"
        "</"
        "("
        "script$TST(a_script=1)"
        "|noscript$TST(a_noscript=1)"
        "|applet$TST(a_applet=1)"
        "|object$TST(a_object=1)"
        "|embed$TST(a_embed=1)"
        "|iframe$TST(a_iframe=1)"
        "|noapplet$TST(a_noapplet=1)"
        "|noobject$TST(a_noobject=1)"
        "|noembed$TST(a_noembed=1)"
        ")\7"
        ">"
        "$SET(1=<[/\7]> -->)"
        "$SET(matched=)"
        ")"
        "$SET(andrew=yes)"
        ")"
        ""
        "|(^*?)$STOP()"
        "("
        "($TST(andrew=yes)|$TST(bypassing=1))"
        "$SET(1=\r\n\r\n<script type="text/javascript" src="http://local.ptron/andrew.js"></script>\r\n"
        "<script type="text/javascript">\r\n"
        "prx_a_array($GET(scriptn), $GET(noscriptn), $GET(iframen), $GET(appletn), $GET(embedn), $GET(objectn), $GET(noappletn), $GET(noembedn), $GET(noobjectn), "$GET(sep)", "$GET(bypassing)", "$GET(origlink)");\r\n"
        "</script>)"
        ")"
Replace = "\1"
Visit this user's website
Add Thank You Quote this message in a reply
Jun. 16, 2008, 01:58 PM
Post: #51
RE: Andrew's Security Filter(s) v5.56 (June 15, 2008)
"Kye-U Wrote:The filter below replaces offending tags with "foobar", but if you go on http://www.cnn.com, you'll see that it doesn't remove the contents of script tags.

Yeah, that doesn't work so good. Smile!
How about using something like this match/replace code to break script tags:
Code:
[Patterns]
Name = "Break Script Tags"
Active = TRUE
Limit = 256
Match = "<script[^>]+>$SET(1=<script type="foo">//)"
Replace = "\1"

I tried it a cnn, seems to work ok with firefox.
I didn't see any js window variables in the dom (except for my js).
Didn't even get any js errors either.
Haven't tried this with any other browsers though.

For object and embed tags, replacing the tag name should break them.

Or, the iframe and object tags could be matched with $NEST().
From what I've noticed, the inner html for these tags usually has a fairly small byte limit.

Another alternative could be something like this:
Code:
Match: (<(iframe|object))\1
Replace: <textarea style="display:none">\1

Match: (</(iframe|object)[^>]+>)\1
Replace: \1</textarea>

This converts the inner code to hidden text.
Just need to watch out for the style attribute quote mark if matching in js.

I quit using html comments in replacement text.
Html comments aren't neseted, the first -- encountered terminates the comment.
If there's a comment inside the injected comment, the injected comment is terminated.
This usually results in html code being rendered.

I'm not sure about the need to match noscript.
I have found the noscript tag to be useful when js is disabled.
If js is enabled, they don't do anything anyway.

As for the other "no" tags that are being matched, I've never seen them.
What browser understands those tags?
Firefox just ignores the noobject tag on your test page and displays the inner hello!.

At any rate, it seems that other filters could match annoying html inside the no[a-z] tags.

As a side note about your test page, I wonder what link.com thinks of this? Smile!
Code:
+++GET 4704+++
GET /to/bad.js HTTP/1.1
Host: www.link.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14
Accept: */*
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Referer: http://www.link.com/
If-Modified-Since: Mon, 16 Jun 2008 12:41:46 GMT
Connection: keep-alive

+++RESP 4704+++
HTTP/1.1 404 Not Found
Date: Mon, 16 Jun 2008 12:41:46 GMT
Server: Apache/2.2.0 (Unix) mod_ssl/2.2.0 OpenSSL/0.9.7a PHP/4.4.2
Content-Length: 207
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
Last-Modified: Mon, 16 Jun 2008 12:41:46 GMT

z12
Add Thank You Quote this message in a reply
Jun. 16, 2008, 11:59 PM
Post: #52
RE: Andrew's Security Filter(s) v5.56 (June 15, 2008)
I've updated both test pages! I hope link.com didn't mind being an innocent bystander!

I'm currently playing around with the filters, and took out noembed, noobject, and noapplet. I left noscript in there because many websites seem to have webbug/ad images nested inside (see Washington Post, or The Star).

I've made it so that iframes and noscripts are converted into textareas. Object, applets and embed are changed to "foo". (Thanks for the tip about using textareas! I was worried about webbugs loading, but I just remembered that textareas turn any nested code into plaintext, and renders them useless!)

EDIT: Found a 484-byte long <script> tag here; increased the byte-limit in my test filter from 256 to 512 Smile!
Visit this user's website
Add Thank You Quote this message in a reply
Jun. 17, 2008, 02:24 AM
Post: #53
RE: Andrew's Security Filter(s) v5.56 (June 15, 2008)
Mike and Kye-U;

Been following along, enjoying the ride.........Wink

Don't forget that in some cases, the "<script...." you see in a page's source code didn't necessarily arrive in that form. Mike's comment about the DOM reminded me that you often see the write method used to create things like this:

<document.write "<scr" + "ipt>"...... etc.>

Fortunately, dbug.. will show this coding correctly. (Mike, is this what you meant by the tag sometimes being encoded?) And Kye-U, I think you once wrote a filter to ferret out this kind of behavio(u)r, didn't you?

This might also be the reason why other filters fail to work as expected, with no seeming logic for said failure. Whistling



Oddysey

I'm no longer in the rat race - the rats won't have me!
Add Thank You Quote this message in a reply
Jun. 17, 2008, 11:48 AM
Post: #54
RE: Andrew's Security Filter(s) v5.56 (June 15, 2008)
Kye-U Wrote:EDIT: Found a 484-byte long <script> tag here; increased the byte-limit in my test filter from 256 to 512

Yeah, 256 is too small.

Oddysey Wrote:Don't forget that in some cases, the "<script...." you see in a page's source code didn't necessarily arrive in that form
...snip...
<document.write "<scr" + "ipt>"...... etc.>

In order for the code example you cited to work, the outer html script tag must work.
If you disable the html script tags, nothing in the script will work.

Oddysey Wrote:Mike, is this what you meant by the tag sometimes being encoded?

I was thinking of the String.fromCharCode() and String.charCodeAt() javascript methods.
Although trying to text match variations of your example can also be a real pain.

z12
Add Thank You Quote this message in a reply
Jun. 19, 2008, 01:51 PM
Post: #55
RE: Andrew's Security Filter(s) v5.56 (June 15, 2008)
So far version 5.56 seems very good to me. I have no problems editing the whitelist anymore.
It does seem quite a bit more efficient, and it feels somewhat faster.
I think the new whitelist method is an excellent way of doing it.
I like the new layout, non intrusive and efficient.
All in all great work imho Smile!

Only "problem" i have encountered(as im aware of), i dont have the onmouse over info box anymore. Ive only got a small box/square.
The filters are direct copy paste form the original post and ive done no editing(other than the whitelist). I think it was working fine in 5.55 but i cant recall exact.


Attached File(s) Image(s)
   
Add Thank You Quote this message in a reply
Jun. 23, 2008, 01:47 AM
Post: #56
RE: Andrew's Security Filter(s) v5.56 (June 15, 2008)
Try going to http://local.ptron/andrew.js , and refresh it (an alternative is clearing your browser's cache).

I have a rough idea of what v4.57 would be like, which doesn't include the option to remove noapplet, noobject and noembed tags, and uses z12's replacement method (simply breaks tags instead of removing the entire code from the page, improving speed).

So far, so good Smile! Currently testing it out as much as I can.

Oddysey, looking for embed tags in scripts would be too complicated, I think, as I'd have to write a filter that somehow intercepts and detects javascript functions and events at the browser level (as NoScript for Firefox does, I presume). Perhaps it can be done Smile! But at the moment, I don't have the knowledge required to write such a script =[
Visit this user's website
Add Thank You Quote this message in a reply
Jun. 23, 2008, 08:51 AM
Post: #57
RE: Andrew's Security Filter(s) v5.56 (June 15, 2008)
Yeah, refreshing doesnt work, neither does cleaning cache etc.
And just for the record it must have been some version before 5.55 as that one is the same.
Just did a "clean" test in a vmware session with xpsp3 installed. Using IE6 and whatever java version is bundled, added andrew only(no other filters) and i got the same result except the onmouseover box is a bit more rectangular.
So it shouldnt be anything on my system interfering. Sorry i dont have any more usable info. Let me know if theres anything i can do.

Its no big deal to me though, the filter is alive and kicking so.... Smile!
Add Thank You Quote this message in a reply
Jun. 29, 2008, 08:21 PM
Post: #58
RE: Andrew's Security Filter(s) v5.56 (June 15, 2008)
Here's an interesting page:
http://ghill.customer.netspace.net.au/embiggen/

It creates and loads a script via an "onclick" attribute.
This script then creates and loads other scripts.

All in All, the script is pretty cool.
The script converts "TinyURLs" to regular urls.
I'm thinking of adapting it for use with my config.

Anyway, it does point out the weakness of matching only html script tags.

I considered a few different ways of disabling html event attribute js.
In the end, I decided to just rename those attributes.
The main drawback is there can be a lot of matches in the log window.

FYI, here's a modified version of a filter I posted earlier:
Code:
[Patterns]
Name = "Break Script Tags 3.0"
Active = TRUE
URL = "$TYPE(htm)(^local.ptron)"
Limit = 1024
Match = "<script[^>]+>"
        "$SET(bScript=1)"
        "$SET(1=<script type="foo">/*)"
        "|"
        "</script >"
        "$SET(bScript=)(^?)"
        "|"
        "(\son[a-z]+)\0="
        "(^$TST(bScript=1))"
        "$SET(1=\0not=)"
Replace = "\1"

The bScript variable, as used above, prevents matching inside script tags.
It's an attempt to keep the log window from filling up with useless matches.

This version is faster:
Code:
[Patterns]
Name = "Break Script Tags 3.1"
Active = FALSE
URL = "$TYPE(htm)(^local.ptron)"
Limit = 1024
Match = "<script[^>]+>"
        "$SET(1=<script type="foo">/*)"
        "|"
        "(\son[a-z]+)\0="
        "$SET(1=\0not=)"
Replace = "\1"
But I'm not sure the speed difference would be noticeable.
Personally, I like version 3.0 better.

z12

Edit:
Looks like I've got to re-think this attribute match/replace.
One of my other filters matched on an anchor tag before this one.
This allowed an onmouseover to get by without being replaced.
Add Thank You Quote this message in a reply
Jul. 09, 2008, 12:47 AM
Post: #59
RE: Andrew's Security Filter(s) v5.57 (July 8, 2008)
@usr: I've just posted v5.57, I hope it solves the issue!

@z12: Wow, I've never seen that before (I didn't know it was possible)! I've included \son[a-z]+=$TST(a_script=1)$SET(1= foo=) in v5.57. Thank you for all your help! I'm going to continue testing to see if this breaks any pages. I notice that it matches with Sidki's filters (see the "Embiggen" link on http://ghill.customer.netspace.net.au/embiggen/)
Visit this user's website
Add Thank You Quote this message in a reply
Jul. 10, 2008, 01:05 PM
Post: #60
RE: Andrew's Security Filter(s) v5.57 (July 8, 2008)
Kye-U Wrote:@usr: I've just posted v5.57, I hope it solves the issue!
Unfortunately, no such luck.
Ive tried v5.57 on three different pcs and a wmvare virtual machine(all xpsp3, only filter added is andrew) with the same result.
Wonder if its sp3 related in some way. I find it strange though, im the only one experiencing this.
Ill poke around some and get back to you.
Thanks for looking into it, and thanks for the update Smile!

Edit:Tested in vmware xp no service pack, IE 6, fresh proxo 4.56 installation with default filters and andrew, same result.
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: