Gmail Technical Rant

[Graycode]

Just wanted to vent at some idiots who work for Google.

Recently I tightened down some timing rules in my proxy. One of the rules determined how long, after receiving all HTTP headers and any initial data, the proxy should wait for subsequent data that it expects. For example, if the Content-Length specifies 300 bytes and we've gotten 200 so far, how long should it wait for the remaining? Earlier I was defaulting to 30 seconds ... but decided that was too generous and made the default 15 seconds. In reality, 1 or 2 seconds would be generous enough in most cases.

So today I'm using Gmail and notice the proxy's log shows rejections of invalid server content, and the flow of the browser's TCP connections isn't happening quite like it should. It turns out that the Gmail API is sending Chunked content ... with each chunk spaced about 25 seconds apart! By having changed the default to 15 seconds, the waiting timed out and the proxy flared up with an error response (to help protect browsers from garbage).

It's happening when Gmail's AJAX repeatedly talks to their servers when the Gmail page is logged in.
The URL for these looks like: (I repalced some data portions with "..." for posting here)
mail.google.com/mail/channel/bind?at=...&VER=6&it=...&RID=rpc&SID=...&CI=1&AID=44&TYPE=xmlhttp&zx=...

Below is some examples. In this debug session, time of day is logged (these samples started 4:03PM + 35 seconds + 28 milliseconds), and the size of socket data read from the server. Also, anyone who believes in standards should ignore the non-compliant (empty) ETag headers.

================================
This first one shows 27 seconds between each part of the chunked data.

160335.28 SvrData Read Size=293
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Thu, 19 Feb 2009 22:02:42 GMT
Content-Type: text/plain; charset=utf-8
ETag:
Transfer-Encoding: chunked
X-Content-Type-Options: nosniff
Server: GFE/1.3

15
18
[[45,["noop"]
]
]

160402.09 SvrData Read Size=32
15
18
[[46,["noop"]
]
]

0

================================
Next one, 25 seconds between data chunks:

160402.31 SvrData Read Size=293
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Thu, 19 Feb 2009 22:03:10 GMT
Content-Type: text/plain; charset=utf-8
ETag:
Transfer-Encoding: chunked
X-Content-Type-Options: nosniff
Server: GFE/1.3

15
18
[[47,["noop"]
]
]

160427.68 SvrData Read Size=27
15
18
[[48,["noop"]
]
]

0

================================
Next one, 25 seconds between data chunks:

160427.96 SvrData Read Size=293
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Thu, 19 Feb 2009 22:03:35 GMT
Content-Type: text/plain; charset=utf-8
ETag:
Transfer-Encoding: chunked
X-Content-Type-Options: nosniff
Server: GFE/1.3

15
18
[[49,["noop"]
]
]

160452.18 SvrData Read Size=32
15
18
[[50,["noop"]
]
]

0

==================================

Doing that in AJAX just doesn't seem appropriate to me. Use of HTTP/1.1 chunking for it seems like a violation of something (common sense maybe). Locking up some of a browser's Max-Connections with this method is certainly "Un-friendly" to users.

I'm a bit angry, so I've decided to just block those AJAX requests for now. If blocking keeps new email activity from showing when a browser's tab is on Gmail then I'll just click on a button when I want to know.

Why am I ranting here? Well if Proxo tries to grab the whole response data, then it's going to be a long wait. And someone might care to know about long-lived TCP connection resources being used for it. Or maybe not.

[sidki3003]

Once i'm logged into GMail, i always have (exactly) one open connection. When i killed it, it got immediately restored.

I thought it was a keep-alive ping (to check if window still open, just one window, etc.). I've seen that in other Web2 apps too - Y!Mail for one. XHR also to do the restore. Maybe "chunked" to save on opening new connections?

[whenever]

Yes, there are always active connections shown on Proxomitron main window when gmail is open. Now I understand it better.

@Graycode, how is your proxy going? When will it be public?

[ProxRocks]

i'm using "basic HTML" for my Gmail viewing and i don't have any active connections showing, never have...

goes for IE7 shells and Opera...
didn't check Firefox...


are you guys getting this "nuance" if you view via "basic HTML" ???

[sidki3003]

I didn't check, but most likely not, because Google would hardly use XMLHttpRequest in the basic interface.

[Graycode]

(Feb. 20, 2009 06:25 AM)sidki3003 Wrote:  Maybe "chunked" to save on opening new connections?

Maybe, but perhaps they didn't think it through. While the server response comes in delayed chunks, the client-side AJAX isn't waiting between each round. It pumps another request almost immediately after the previous one completes. The AJAX should just be just polling requests with setTimeout() intervals, but apparently here they've chosen to burden their server-side resources with delay intervals.

The Transfer-Encoding: header is hop-by-hop, intermediate processes like proxies may un-encode or re-encode that as they see fit. Gmail may have a proxy-like server front end doing the chunking, and that might be where their delay is implemented.

The server should be sending a normal (complete) response, and let client browsers & proxies manage the re-usable connections in the usual efficient fashion. Hopefully they'll fix it some day.

Today I'm using a server data timeout of just a few seconds (having blocked the problematic Gmail URL). So far I haven't encountered any other sites that hold connections hostage with a server-side partial-response delay.

(Feb. 20, 2009 06:56 AM)whenever Wrote:  @Graycode, how is your proxy going? When will it be public?

Thanks for asking. The proxy works extremely well, but bringing it to the public took a hit as "The Economy" happened. A lot of preperation remains that requires funding to accomplish. If you could please fix "The Economy" I'd really appreciate that!

[ProxRocks]

(Feb. 20, 2009 06:59 PM)Graycode Wrote:  If you could please fix "The Economy" I'd really appreciate that!

i SECOND the motion on the floor...
motion carried...

first 100 days sets the four-year stage...

[sidki3003]

Here is another variant:
http://picasaweb.google.com/lh/ajaxlog?p...5544002577

Although it's a normal 204 reply (no content, no content-type), it keeps the connection open here for exactly 1 minute. So, while browsing a Picasa member directory, i end up with a bunch of open connections.

The main pages OTOH come as "Transfer-Encoding: chunked". Quite often, and after a long delay, i see this instead of the rendered HTML:

Code:

0

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Date: Mon, 23 Feb 2009 17:20:58 GMT
X-Content-Type-Options: nosniff
Cache-control: private, must-revalidate, max-age=0
Server: GFE/2.0
|.*.URL-ID: http://picasaweb.google.com/Page.Error/VideoUploadTest
Transfer-Encoding: chunked

1


393
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
<snip>

Thus far i got above garbage only, when there was at least one open "ajaxlog" connection. So maybe it isn't related to the transfer-encoding, but i don't know...

[Graycode]

(Feb. 23, 2009 06:03 PM)sidki3003 Wrote:  Although it's a normal 204 reply (no content, no content-type), it keeps the connection open here for exactly 1 minute. So, while browsing a Picasa member directory, i end up with a bunch of open connections.

It's a bit surprising that Google is retaining the connection for that long. I think the use of long-term persistence may become less frequent.

As you mentioned, the 204 shouldn't have any content.

Code:

HTTP/1.1 204 No Content
Set-Cookie: _rtok=_TZCpJWNfBg1; Path=/
Set-Cookie: S=photos_html=cgJ3-InHjgaIXlwt69mDUw; Domain=.google.com; Path=/; HttpOnly
Date: Mon, 23 Feb 2009 18:16:20 GMT
X-Content-Type-Options: nosniff
Server: GFE/2.0

The server isn't explicitly specifying its Connection preference, so for HTTP/1.1 it defaults to being re-usable for subsequent requests. But then the questions become: For how long should the server remain connected, and for how long should the client proxy | browser hold on to it? For each, now many of those connections should be retained?

Some servers don't seem to be configured for particular persistent timeouts, it's an area that web software has been paying more attention to lately. I believe Apache still needs a resource (either a thread or process) for each persistent connection that's waiting for possible re-use. As more client browsers and speed-hungry users configure themselves for more connections & longer persistence, the servers now have to defend their available resources more.

IE version 8 has increased its number of persistent connections to each server. Opera used to be the most agressive at it. Mozilla's has also been "tweaking" their connection limits to push servers harder, and they've alreay responded to what IE 8 will do. I also noticed just the other day that a default Firefox installation was sending "Keep-Alive: 300" to Servers, indicating its intent of 5 minutes of persistence for every connection.

This weekend's new release of Privoxy now tries to retain server connections for several minutes indepenently of each client-side connection. I forget whether their default is 180 seconds or 300 for everything, and that proxy doesn't seem to be looking for or honoring any timeout that a server might specify in its Keep-Alive: header.

Some people who recommend increasing a browser's max-connections are really just demanding that servers defend themselves more & more. Examples include the sticky thread and related FAQ at http://www.dslreports.com/forum/mozilla

When a proxy is used, most browsers aren't bothering to abide by their "max-connections-per-Server" guidelines. They're leaving that completely up to the proxy and instead just oversee their total browser "max-connections". So, by using a proxy it's even more likely to have a lot of connections to a particular server if nothing's managing that aspect.

The battle rule is that clients & servers will play that game until the other side closes the connection. "I'm the fastest browser" does things that tries to defeat "I'm the most stable server".

I think recent Apache distributions are limiting the server's keep-alive persistence to 15 seconds by default.

This site's Apache uses 5 seconds, and after that amount of quiet time the server really does close the connections.

Code:

Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

Some of that was mentioned in http://prxbx.com/forums/showthread.php?tid=639&page=5

Sorry I can't tell you how Proxo manages connections. If you'd like I could describe how my proxy does it, but that isn't necessarily going to be helpful to Proxo.

(Feb. 23, 2009 06:03 PM)sidki3003 Wrote:  The main pages OTOH come as "Transfer-Encoding: chunked". Quite often, and after a long delay, i see this instead of the rendered HTML:

I haven't seen that one before. It looks like the server connection was re-used, and the first "0" was the trailing part of a previous request's chunked response. That bogus header having the error URL sure implies it's another Google server screw-up mixing multiple responses together.

[sidki3003]

Thanks for the explanations.
That bogus incoming "URL-ID" header got inserted by Proxomitron, for debugging purposes.

I've just made a little experiment:
- loaded: http://picasaweb.google.com/Page.Error/VideoUploadTest (can be any other member directory)
- got above discussed open connection to the 204
- immediately refreshed: http://local.ptron/.pinfo/urls/ (local.ptron is the alias for Proxomitron's web interface)

The local.ptron page only got loaded once the 204 connection was closed (i.e. after 60 seconds).
2 out of 3 local.ptron refreshes ended up with described plain-text garbage instead of the rendered page.

So it could also be Proxomitron (or Fx3's proxy handling), not correctly dealing with the unclosed 204.

[Graycode]

Your experiment is very interesting.

With the local.ptron being a completely different host name, it's obvious they should not have been mixed up to result in the plain-text garbage. On the surface that appears to be a Proxo bug. Firefox is probably be re-using its connection to the proxy after the 204, but Proxo does not realize the 204 was all done and that the browser's new communication is a completely different request.

Can a Proxo header filter safely provide "Connection: Close" back to the browser for every 204 response that doesn't also have a "Content-Length: 0" header? That should keep the browser from trying to re-use its connection to Proxo, thereby circumventing the problem. If the browser then initiates closure, Proxo would probably also stop waiting for more (that will never arrive) from the 204.

I'm guessing you wouldn't encounter the problem with this other URL used by Gmail:

http://www.google.com/setgmail?zx=

Code:

HTTP/1.1 204 No Content
Content-Type: text/html; charset=UTF-8
Set-Cookie: PREF=ID=...:TM=...:LM=...:GM=1:S=...; path=/; domain=.google.com
Date: Tue, 24 Feb 2009 17:13:39 GMT
Server: gws
Content-Length: 0

The above provides explicit "Content-Length: 0" while the picasaweb server doesn't do that. Here, Proxo would probably pick up on the zero length and realize the 204 response is a done deal.

[sidki3003]

(Feb. 24, 2009 07:07 PM)Graycode Wrote:  Can a Proxo header filter safely provide "Connection: Close" back to the browser for every 204 response that doesn't also have a "Content-Length: 0" header? That should keep the browser from trying to re-use its connection to Proxo, thereby circumventing the problem. If the browser then initiates closure, Proxo would probably also stop waiting for more (that will never arrive) from the 204.

I think getting the matching tests right - RESP 204 and Content-Length != 0 - should be no problem. However, there are a few headers that can't be modified, and - i *think* "Connection" is somewhere in between. I did my tests years ago, so i'm not sure anymore, but if true, it would mean that even Scott's own "Connection: Close all connections (In+Out)" filter is 50% defunct, after introducing pipelining support in 2002.

Anyway, i'll re-check this. Also, i'll test another idea (redirecting 204s). I'll probably post a follow-up later.

Quote:I'm guessing you wouldn't encounter the problem with this other URL used by Gmail:

Yep -- Zero problem there!

[sidki3003]

No, constructing an incoming "Connection: close" header doesn't work.
The tricky (and confusing) thing is that Prox' Log window is indeed showing that added incoming header, but doesn't send it.

Since Firebug is useless for such special cases, i tested with SocketSniff (by incredible Mr. Nir Sofer).
I only tested with Proxomitron's "Local connections persist..." option turned on (default).
I was testing at http://picasaweb.google.com/lh/ajaxlog?page=

SocketSniff:

Code:

HTTP/1.1 204 No Content
Date: Wed, 25 Feb 2009 09:18:56 GMT
X-Content-Type-Options: nosniff
Server: GFE/2.0
Cache-Control: public, max-age=86400
Transfer-Encoding: chunked

Proxomitron's Log window:

Code:

HTTP/1.1 204 No Content
Date: Wed, 25 Feb 2009 09:18:56 GMT
X-Content-Type-Options: nosniff
Server: GFE/2.0
Cache-Control: public, max-age=86400
Connection: close

What does work is redirecting the 204. I tend to use a quick local 400 reponse ( http://local.ptron/nirvana ) for such cases.

Code:

[HTTP headers]
In = TRUE
Out = FALSE
Key = "Content-Type: 6d Block 204s w/o Content-Length:0     9.02.25 (In) TEST3"
URL = "$RESP(204)"
Match = "(^$IHDR(Content-Length:( ) 0))\0&$RDIR(http://local.ptron/nirvana)"
Replace = "\0"

My problem right now is that i just know what RFC2616 says. I don't have a clear idea about the purpose (and real life use) of 204 replies.

Could the content of a 204 response ever (or theoretically) be parsed by the browser?
Right now, my config is inspecting (i.e. filtering) every response with a missing content-type header (also because this scenario activates content sniffing in Firefox and Opera). This also includes most 204s.

Should i modify the content-length test in the filter to only redirect if the Content-Length header is missing?

[whenever]

http://books.google.com/books?id=3AjrYKTmimsC&pg=PA429

Quote:204 No Content
A status code and header are given in the response, but there is no entity body in the reply. On receiving this response, browsers should not update their document view. This is a useful code for an imagemap handler to return when the user clicks on useless or blank areas of an image.

[lnminente]

Hi guys, i noticed also about "No content" responses when writing the filter for youtube high quality. Since then, i began to log all these responses to know something about its use. For one month i'm blocking the requests wich produces the 204 response from google and new banners from youtube. Most of them are blocked with these expresions:

Code:

*google.(com|es)\/(csi\?v=|url\?sa=)*
*(google.com|doubleclick.net)/s?ns=(yt|vss)\&docid=*
http://s2.youtube.com/s\?ns=*
http://video-stats.video.google.com/s\?ns=*

And are used to send google wich link i clicked, and to receive adclips in youtube (this last i think, i don't remember very well)


EDIT:
Just to menction we can kill the incoming 204 responses from the picassa's link using this filter

Code:

[HTTP headers]
In = TRUE
Out = FALSE
Key = "URL :I_1.3 Log and kill No content {ln}090225"
URL = "$RESP(204*)"
Replace = "$ADDLST(debug,I_1.3 Log and kill No content\t$DTM(YMD H:m:s't)\t$GET(uDom)\t$GET(uExt)\t$GET(ContentType)\t\u\t\9\t\1\t\3) $LOG(R$DTM(c),I_1.3 No content in \u ) \k"

The 204 responses are converted to 00 with 0 content lenght in http://local.ptron/.pinfo/urls/