SSL woes.....

[Oddysey]

Hard to believe it, but I'm actually out of the driver's seat on this one.

Seems like I finally buckle down and get with the program, meaning that I've finally jumped to 4.5j (really!), and it was only for one purpose - to be able to use SSL on a particular HTTPS page. If it weren't for that one page, I'd still be sitting pretty with Naoko 4.4.

Well, now I've tried every SSL set of files I can find, from 0.9.3 on up, and still no go. The page in question is Yahoo's log-in page, necessary before I can read my mail or get into most of my groups. And it's not so much a matter of "security", it's more for dress layout - all Yahoo pages are ugly to start with, and full of invasive crap underneath the hood. Proxo to the rescue, or at least, until they went to HTTPS a few weeks ago. Now, Proxo doesn't seem to be able to cut through the crap.

Whyzat?

And yes, I've followed all the instructions I can find here, and on sidki's site and http://www.proxomitron.info. (I must say, if I were a beginner, I'd think that those two are pretty sparce on the user-friendliness. Kye-U has it down much better, IMHO.) They all say pretty much the same thing, and I still get the same results - no joy.

The log window says that the request is going out fine, and coming back in unfiltered, presumably because a server-side redirect brought back an HTTPS page. I'm not so much concerned as I am puzzled.

The Odd has been stumped. Step right up and get your ya-ya's out, then offer up some valid help, eh? Please?

Thanks.


Oddysey

[z12]

Hi Oddysey

Oddysey Wrote:I've finally jumped to 4.5j

It's about time you made the jump to 4.5j.

I got to ask the obvious, are you sure you have "checked" Use SSLeay/OpenSSL on proxo's "Config" --> "HTTP" window?

As far as Yahoo goes, don't use it so can't help you out there.

Mike

[Oddysey]

Mike;

z12 Wrote:Hi Oddysey

Oddysey Wrote:I've finally jumped to 4.5j

It's about time you made the jump to 4.5j.

I got to ask the obvious, are you sure you have "checked" Use SSLeay/OpenSSL on proxo's "Config" --> "HTTP" window?

As far as Yahoo goes, don't use it so can't help you out there.

Mike

Yeppers, done did that. In fact, under Naoko 4.4, I had to go in and manually set the value to TRUE in the config file - the program refused to let me check the box. Not sure why, but after that, it seemed to work just fine. But after Yahoo's upgrade, 4.4 no longer worked (if it ever did).

This is the only site I access that uses HTTPS, so I may not have ever had it truly working, that's true. But now that I need it, it sure would be nice if it would just cooperate and do what it's supposed to do.

I remembered a few minutes ago, that when I used to hit certain HTTPS sites under 4.4, I got the warning about out-of-date certificates. Would that happen anyway, if the SSL stuff wasn't working? Seems to me that it wouldn't. The first time I hit on that particular error, I tried disabling Proxo, and that helped - no more expired cert warning. That led me to believe that SSL was working under 4.4..... or am I all washed up here?

Now, under 4.5j, I don't get expired cert warnings, but I have to admit, I got the new certs file too, so everything I'm likely to hit may already be up-to-date. Anyone wanna give me a test site where I'd be sure to hit HTTPS/SSL, and possibly an expired cert warning?


Oddysey

[JJoe]

Oddysey Wrote:Anyone wanna give me a test site where I'd be sure to hit HTTPS/SSL,

https://grc.com/x/ne.dll?bh0bkyd2

With SSL enabled,
Domain name mismatch warnings.
Proxomitron Log entries like:
+++SSL:GET 562+++
+++SSL:RESP 562+++

Without, the Log should show lines like
SSL Pass-Thru: CONNECT https://grc.com:443/

[ProxRocks]

JJoe Wrote:https://grc.com/x/ne.dll?bh0bkyd2

With SSL enabled,
Domain name mismatch warnings.
Proxomitron Log entries like:
+++SSL:GET 562+++
+++SSL:RESP 562+++

Er, I have SSL enabled and am getting the +++SSL:(GET|RESP) xyz+++'s but I am not getting any domain name mismatch warnings (unless there's some security setting I overlooked that may be "why" I'm not getting such a warning)...

[JJoe]

ProxRocks Wrote:but I am not getting any domain name mismatch warnings (unless there's some security setting I overlooked that may be "why" I'm not getting such a warning)...

Aren't you using HalfSSL? I'm assuming Oddysey isn't.
Otherwise and assuming that you haven't turned off notification, IE should warn you about a "mismatch".
Server Certificates And Such and maybe an unknown cert.

[z12]

ProxRocks Wrote:I am not getting any domain name mismatch warning

I certainly am.

Another possiblity, though remote, is yahoo is returning a mime-type with application/xhtml. By default, proxo won't filter that:

Code:

+++GET 217+++
GET / HTTP/1.1
Host: www.w3.org
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060601 Firefox/1.5.0.4
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://www.w3.org/

+++RESP 217+++
HTTP/1.1 200 OK
Date: Tue, 17 Oct 2006 16:45:41 GMT
Server: Apache/1.3.37 (Unix) PHP/4.4.3
Content-Location: Home.xhtml
TCN: choice
P3P: policyref="http://www.w3.org/2001/05/P3P/p3p.xml"
Cache-Control: max-age=1
Last-Modified: Tue, 17 Oct 2006 16:45:41 GMT
ETag: "4534effc;42380ddc"
Accept-Ranges: bytes
Content-Length: 32327
Content-Type: application/xhtml+xml; charset=utf-8; PrxMsg=Filtering Forced

Mike

[Oddysey]

JJoe;

In no particular order.......

Per your suggestion above, here's an excerpt of my log:
New Message Log Window....

+++GET 3008+++
GET / HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/x-shockwave-flash, application/pdf, */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
Host: mail.yahoo.com
Cookie: (deleted, not necessary)
Connection: keep-alive

+++RESP 3008+++
HTTP/1.1 302 Found
Date: Tue, 17 Oct 2006 22:43:29 GMT
P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV"
Location: https://login.yahoo.com/config/login_verify2?&.src=ym
Pragma: no-cache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html
+++CLOSE 3008+++

+++GET 3009+++
GET /x/ne.dll?bh0bkyd2 HTTP/1.1
Accept: */*
Referer: http://www.grc.com/x/ne.dll?bh0bkyd2
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
Host: http://www.grc.com
Connection: keep-alive

+++RESP 3009+++
HTTP/1.1 302 Object Moved
Location: https://www.grc.com/x/ne.dll?bh0bkyd2
Content-Type: text/html
Pragma: no-cache
P3P: CP="NOI DSP COR NID NOR"
Set-Cookie: temp=52nf1gs4tsqjc; path=/; domain=www.grc.com
Set-Cookie: perm=svp5seoc5qzui; path=/; domain=www.grc.com; expires=Mon, 01-Jan-2046 00:00:00 GMT
Connection: close
Content-Length: 160
Server: GRC Custom Hybrid NanoProbe Engine/1.57 (experimental)
+++CLOSE 3009+++

Note that I opened a fresh browser session, so the cache was completely empty, and it was the only one to be open at the time.

I first accessed Yahoo, then GRC, as you can see. No warnings, no Proxo statements about SSL, nada. Also noteworthy, I can refresh the page as often as I want, and no additional activity shows in the log. That's strange, at least to me.

My browser (IE 6.1) is set to not give warnings on switching between secure and insecure pages, is that what you mean? I also check for site certificates, and for revoked ones too (with warnings turned on). Nothing of that sort ever comes up, frivolously I mean. I do have one site where I'm the administrator (SYSGOD), and it hounds me with a expired certificate every time I log in, so the 'warning' mechanism is working..... at least I think it is.

Yahoo doesn't seem to be messing with mime-types (thanks, ProxRocks), although they do collect a list of supported mime-types, but I don't see them doing anything with that on the log-in page. Perhaps later in the session.....

And finally, to the best of my knowledge, I'm not using halfSSL. If the setup procedure for that is different, then perhaps I should try it?

Ooops, even more finally..... While messing around as I write this, I see that the log window will reflect the loading of SSL as you enable it from Proxo's config screen, if you do so during a session (as I did for testing everyone's suggestions here). I went back and re-ran every test, no difference.

Anyone got any Rogaine handy?


Oddysey

[JJoe]

Oddysey Wrote:New Message Log Window....

Have you set IE to use a proxy for https? Nothing in the log show https going through the Proxomitron.

Oddysey Wrote:My browser (IE 6.1) is set to not give warnings on switching between secure and insecure pages, is that what you mean?

Nope.

[Oddysey]

JJoe;

JJoe Wrote:Have you set IE to use a proxy for https? Nothing in the log show https going through the Proxomitron.

Give the man a fat ceee-gar!

Don't know what possessed me to forget something like this, but then again, I never had use for it before now.

All pages are filtered just fine, thank you very much. Now to update my Proxo certificate, and the last "warning alert" should disappear into the night.

Tha's why we have this forum/board/community. Even I can benefit, right?


Oddysey

[Oddysey]

Six months later.......

The saga continues.

One day, I'm fussing over my Ipod's USB driver (it wouldn't disconnect gracefully), so I pulled it all out, endeavoring to start over. The next thing I knew, I went to IE, and all the troubles I described at the beginning of this thread were right back with me!

First things first, I double-checked to see that my browser hadn't reverted on me.... nope, it's still set to use Proxo as a proxy on the correct port. The log definitely shows https activity, supposedly all is in order..... The mime-types don't appear to have been modified by Yahoo...... Proxo seems to still be the 4.5j version, and the SSL files appear to be intact - at least, Proxo isn't barking at me about the wrong version or anything, and I don't get any bad certification warnings......

Next, I should report that I can use Proxo's URL commands to modify this somewhat. By inserting the 'https..' url command after the double back-slashes (and making sure that the protocol is set to http, not https), I can make most of my filters work, but strange things happen here, too. I have two filters set to block the vast majority of crap. The second one sweeps out all the detritus from the bottom of the page, and that works great. But the top half is working only partially.

So I looked at the log, and neither of them are working at all. ??? What's happening is that the Kill Nosey Javascript filter is working just fine, and it's removing junk like there's no tomorrow. But even though it's set to Multi=True (and so are my other two), no other filters get exercised. Now I'm really stumped - - why would one filter work on an HTTPS page, and not another one? And particularly when said cleanup filters worked like a charm for over 6 months (see all previous messages, this thread).

And if you're curious, no, I haven't added or edited any filters for at least 5 or 6 weeks, it's been awhile since I've felt 'The Need'. The config file is just fine, and even if it were, I have a backup of that, tucked away. Which I tested anyway, just to be thorough about it all.

I'm about at my wit's end here. Any further ideas?


Oddysey

[JJoe]

Oddysey Wrote:I have two filters set to block the vast majority of crap. The second one sweeps out all the detritus from the bottom of the page, and that works great. But the top half is working only partially.

So I looked at the log, and neither of them are working at all. ??? What's happening is that the Kill Nosey Javascript filter is working just fine, and it's removing junk like there's no tomorrow. But even though it's set to Multi=True (and so are my other two), no other filters get exercised. Now I'm really stumped - - why would one filter work on an HTTPS page, and not another one?

I'd assume the filter(s) isn't matching. Has the page's URL or code changed?
What happens, if you remove the URL Match?
Enable "HTML Debug info" in the Log window and reload the problem page. Is some other filter matching first and 'hiding' or removing code.

[Oddysey]

JJoe Wrote:

Oddysey Wrote:I have two filters set to block the vast majority of crap. The second one sweeps out all the detritus from the bottom of the page, and that works great. But the top half is working only partially.

So I looked at the log, and neither of them are working at all. ??? What's happening is that the Kill Nosey Javascript filter is working just fine, and it's removing junk like there's no tomorrow. But even though it's set to Multi=True (and so are my other two), no other filters get exercised. Now I'm really stumped - - why would one filter work on an HTTPS page, and not another one?

I'd assume the filter(s) isn't matching. Has the page's URL or code changed?
What happens, if you remove the URL Match?
Enable "HTML Debug info" in the Log window and reload the problem page. Is some other filter matching first and 'hiding' or removing code.

No changes can be detected in either the page code or the URL itself. It's been HTTPS protocol since just before my first complaint, back in September. Removing the URL match from the filter has no effect.

I did the "Enable HTML Debug info" on the log window, and reviewed the page in depth. The filter just doesn't kick in at all, and ordinarily, it would be the first or second thing to hit. I tried moving it to the bottom of the pile, to the middle (where it sat originally, it's an old filter), and now to the top. In all three cases, no difference...... no filtering takes place.

No other filter hits near the <body .... tag on the page. Further down, under the best conditions, the Kill Nosey Javascript filter hits, several times. But even then, both that and my filters are set for Multi=TRUE.

A portion of the HTML Debug info page contents:

Code:

.......
</style>
<![endif]-->

</head>
<body id="yregtml">
<div id="yregwp">
<!-- begin header -->
<table id="yregmst" width="750" cellpadding="0" cellspacing="0" border="0"><tr valign="top">
<td width="100%"><table width="100%" cellspacing="0" border="0"><tr valign="top">
<td width="1%"><img src="https://a248.e.akamai.net/sec.yimg.com/i/us/nt/ma/ma_mail_1.gif" alt="Yahoo! Mail" width=196 height=33 border=0>
</td>

..... lots of additional junk here .....

<font size=1 color=red>[Sign up for Yahoo!]</font></a></center></p>
</div>
<h2>Already have a Yahoo! ID?</h2>
<p>Sign in.</p>

<fieldset>
<legend>Login Form</legend>

My filter that should be getting into the act:

Code:

Name = "Yahoo login page cleaner, semi-auto login"
Active = TRUE
Multi = TRUE
Limit = 8192
Match = "<body*(<p>Sign in.</p>|<h1>Sign in(<br>|) to Yahoo!</h1>)*</legend>"
Replace =  "<body LoadOff="document.forms.login_form.login.value='abcdefghi';document.forms.login_form.passwd.focus()">"

(Sorry, my onLoad-Unloader filter forces the Match line to read "LoadOff", but I'm sure you all understand that my real code says onLoad. )

This removes junk, inserts my user name, then sets focus to the password field. (Call me anal, but I don't store my password anywhere on my machine, not even in a filter. )

When all goes well, like it used to, you would ordinarily see the word MATCH in red, right between the </head> and the <body tags. Never more, just like the Raven sez.

Here's a bit from my Log screen:

Code:

+++GET 1869+++
GET /config/login_verify2?&.src=ym HTTP/1.1
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
Host: login.yahoo.com
Cookie: B=d7o3i9d2o3e9r&b=3&s=pi; LYC=l_v=0&l_lv=10&l_s=zxxy0qs535q3utxxzurrz5t3x0xu0xxr&l_um=0_0_1_0_0; U=mt=b.d0PZ2MhYom6jXqLn8sgmBgIFZqEww.s3fV&ux=UYv8FB&un=cgcsl8v2no1d6
Connection: close
BlockList 1869: in CookieList, line 32

+++SSL:RESP 1869+++
SSL cipher TLSv1 AES256-SHA (256 bits)
HTTP/1.1 200 OK
Date: Tue, 13 Mar 2007 22:34:56 GMT
P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV"
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html
BlockList 1869: in LoaderList, line 22
<start> 1869: Suppress all JavaScript errors
<start> 1869: Stop browser window resizing
<start> 1869: Stop status bar scrollers
<start> 1869: Stop JavaScript Timers
BlockList 1869: in CookieList, line 32
Match 1869: Kill anti-cache meta tags
Match 1869: Kill anti-cache meta tags
Match 1869: replace 'funny' quote marks
Match 1869: Kill Nosey JavaScripts
Match 1869: Hide Browser's Version from JS
Match 1869: Hide Browser's Version from JS
Match 1869: Kill Nosey JavaScripts
Match 1869: replace 'funny' quote marks
Match 1869: replace 'funny' quote marks
Match 1869: replace 'funny' quote marks
BlockList 1869: in AdKeys, line 14
Match 1869: Banner Blaster (limit text)
Match 1869: replace 'funny' quote marks
Match 1869: replace 'funny' quote marks
Match 1869: kill Yahoo's new mail crap at bottom
Match 1869: Kill add-on JavaScripts
Match 1869: Kill add-on JavaScripts
<end> 1869: Restore pop-ups after a page loads
+++CLOSE 1869+++

Looks to me like the SSL overlays are working. Or am I missing something?

Notice that several other filters work just fine. Hmmmmm....

I disabled the filters shown above, and no change. But of course, if I disable Proxo entirely, then everything works as advertised. Can you see how I'm getting really cheesed off at this behavior?

I have over 200 filters currently running, though many of them are site specific. Please, don't make me disable the remainders one-by-one, that'll take until next week! I'd like to return to somewhat more normal operations sometime before I grow a beard...... oh, wait..... err, never mind.

What next, oh Great Ones?


Oddysey

p.s. I see I almost forgot to explain...... The first thing just after the SSL:RESP, you see "Blocklist 1869: in Loaderlist, line 22". That's a site-specific list to bypass the onLoad-unloader filter, which I leave enabled at all times.

[ProxRocks]

Oddysey Wrote:...detritus...

lol...
i get the feeling i'm not the only one in here that took Latin in high school...

[JJoe]

Oddysey Wrote:My filter that should be getting into the act:

Code:

Name = "Yahoo login page cleaner, semi-auto login"
Active = TRUE
Multi = TRUE
Limit = 8192
Match = "<body*(<p>Sign in.</p>|<h1>Sign in(<br>|) to Yahoo!</h1>)*</legend>"
Replace =  "<body onload="document.forms.login_form.login.value='abcdefghi';document.forms.login_form.passwd.focus()">"

Code changes aren't always obvious. Could be added space for formatting.
Some sites like to add some space characters around tags. Some pages add space inside the tags.
The size can change. So

Code:

[Patterns]
Name = "Yahoo login page cleaner, semi-auto login test"
Active = TRUE
Limit = 10000
Match = "<body*(<p> Sign in. </p>|<h1> Sign in (<br>|) to Yahoo! </h1>)*</legend>"
Replace = "<body onload="document.forms.login_form.login.value='abcdefghi';document.forms.login_form.passwd.focus()">"

If this doesn't work in your cfg, you could load Scott's, enable ssl filtering, and test it there.
Otherwise...
Ahhh there it is. The code I see is 8683 bytes.
Increase your filter's limit to 10000 or use the one I posted.

wfm
HTH