Post Reply 
Base: Speeding up ad-list
Mar. 15, 2009, 09:47 PM
Post: #16
RE: Base: Speeding up ad-list
(Mar. 15, 2009 08:35 PM)sidki3003 Wrote:  Very inventive idea! I'm curious about your results. Smile!

I don't know if that applies to your case, but for the situations where i'm using this $TST / $ADDLST routine, i had to do a $WESC when adding, as well as an end-of-string test, like:
Code:
(^$TST((\0\5)=$LST(Mem-ScriptSrc)))$ADDLST(Mem-ScriptSrc,$WESC(\0\5)(^?))

For the (^?) I guess \0\5 is a full address? If yes, i don't need it here because the adlist have keywords like /ads/, http://advert, etc... and they are in the middle of the address, so are followed by more chars
About the $WESC of course it applies Sidki! Updating Wink)))
Add Thank You Quote this message in a reply
Mar. 16, 2009, 03:15 AM
Post: #17
RE: Base: Speeding up ad-list
Something off topic, besides reloading proxomitron, is there a way to destroy the content of a memory list when I want to reuse it for another purpose?
Add Thank You Quote this message in a reply
Mar. 16, 2009, 10:17 AM
Post: #18
RE: Base: Speeding up ad-list
Whenever, i learned about the existence of memory lists two days ago reading some documents from the Sidki's config. BTW I'm interested in searching for a method to pass a variable from one connection to another. In that way i could write a HEADER filter wich would block offsite scripts. If a memory list could be cleaned with a command it would be helpful for this...
Add Thank You Quote this message in a reply
Mar. 16, 2009, 03:26 PM
Post: #19
RE: Base: Speeding up ad-list
lnminente: Yep, those match a full address.

whenever: I don't know of any.

(Mar. 16, 2009 10:17 AM)lnminente Wrote:  BTW I'm interested in searching for a method to pass a variable from one connection to another. In that way i could write a HEADER filter wich would block offsite scripts.

You could test the current document domain against the original (*real*) referrer. Maybe you only need to modify log -> block in "Content-Type: 6c Log Third Party Scripts". (I'm currently developing "Set-Cookie: 2d Block Third Party Cookies", based on the same principle. Smile! )


Quote:If a memory list could be cleaned with a command it would be helpful for this...

You could probably automate config reloading via the "load" URL command, although i doubt that this is a practical approach, also because it would kill all open connections.
Add Thank You Quote this message in a reply
Mar. 16, 2009, 08:08 PM
Post: #20
RE: Base: Speeding up ad-list
Bad new, the intermediate Mem-Adlist didn't give any benefits with the AdList. The good new is proxomitron does a very good work processing the 18000 AdLinks using so few CPU from my Athlon 2'8Ghz.

The test:
I went to a page with many adlinks and extracted all of them.
I copied these 9 adlinks two hundred times inside a clean html page, having 18000 adlinks and resulting an HTML file of 2'88 megabytes.

In the picture we can see two CPU load graphics, upper is firefox, down is proxomitron. The left cpu load is without using the Mem-Adlist, the right cpu load is using the Mem-Adlist.

Conclusion: no benefits while using hashed keywords from the list ( http://local.ptron/.pinfo/lists/Adlist ) Later did the same, replacing all the adlinks to get matched for keywords don't hashed, and more of the same, no benefits... I think the "mem-cache method" is something to have in mind for other methods, but not for lists of keywords hashed.

Note: The second cpu load in firefox is bigger, the reason is i didn't reboot firefox. I made more experiments taking this in mind but the results of the cpu load of proxomitron was the same.


(Mar. 16, 2009 03:26 PM)sidki3003 Wrote:  lnminente: Yep, those match a full address.
Fine, so i can forget about (^?), you know how strange is the proxomitron matching sometimes hehe

(Mar. 16, 2009 03:26 PM)sidki3003 Wrote:  Maybe you only need to modify log -> block in "Content-Type: 6c Log Third Party Scripts". (I'm currently developing "Set-Cookie: 2d Block Third Party Cookies", based on the same principle. Smile! )

You could probably automate config reloading via the "load" URL command, although i doubt that this is a practical approach, also because it would kill all open connections.
I will take a look as soon as i can Sidki. I'm currently blocking all the Set-cookie from ad-sources, and would like to complement it with 3rd party too Smile!

Code:
[HTTP headers]
In = TRUE
Out = FALSE
Key = "Set-Cookie: Block in ad sources {ln}090314 (out) TEST"
Match = "(?*)\1$URL(((\w)\2|*((^(http|ftp)://|//)(http|.|/)\w)\2)$TST(\2=(.|/|)($LST(Adlist))\9*)) $ADDLST(debug,Set-Cookie: Block in ad sources\t$DTM(Y/M/D H:m:s't)\t$GET(uDom)\t$GET(uExt)\t$GET(ContentType)\t$GET(Resp)\t$GET(ContentLength)\t\u\t\9\t\1\t\3)"
Replace = "$LOG(R$DTM(c),Set-Cookie: Remove in ad source (\9): \1 )"

By the way i'm testing now for some days and by now is working fine, to remove Location in Ad sources, maybe would be better just to block all non 200 responses... (learning about responses these days Smile! )


Attached File(s)
.jpg  CPU-load with 18000 links. First without mem list. Second with.jpg (Size: 268.74 KB / Downloads: 378)
Add Thank You Quote this message in a reply
Mar. 17, 2009, 12:42 AM
Post: #21
RE: Base: Speeding up ad-list
interesting!!!

um, why did the private bytes DROP in the second screen-cap (ie, "with" the mem-list)?
Add Thank You Quote this message in a reply
Mar. 17, 2009, 12:17 PM
Post: #22
RE: Base: Speeding up ad-list
mmmhhh also in the first graphic of proxomitron there is a small drop just when the CPU load ends, it could be the text wich is parsing

By the way it was with Windows XP SP3, Process Explorer 11.11 from SysInternals, Firefox 3.0.7 (to menction 3.1b3 goes very well)
Add Thank You Quote this message in a reply
May. 27, 2009, 01:04 AM
Post: #23
RE: Base: Speeding up ad-list
That's the code i'm currently using as a function in the functions list, it will be included in the next base config.

Code:
#Function ADLIST:
#Input: [Url to test if is an adsource];
#Output: Truth if match with adlist. \9 will be the portion of URL matched in the Adlist list.
#Explain: It parses quickly URLs looking for adkeywords contained in the list Adlist.
#lnminente 2009 04 23
ADLIST;\0;(^?)            $TST(\0=((\w)\0|[^?]++((^(http|ftp)://|//)(http|.|/)\w)\0)$TST(\0=(.|/|)($LST(Adlist))\9*))
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: