Posts Tagged IE7

Domain-Sharding and SSL

Hi there!

Some time ago I noticed some weird behaviour on our website so I thought it would be a good idea to revisit this issue. Part of the above mentioned website is a selfcare area, where our customers are able to view their bill, to change the product options etc. etc. This area requires a login and so the login itself, as well as the subsequent pages are secured by SSL.

SSL and Performance Optimization was for quite some time a very difficult beast to debug, but fortunately with Pat Meenans WebPageTest this was solved at least for IE. In the old days you could use HTTPWatch for example, to see the HTTP interaction within the SSL tunnel, but you were unable to view the TCP Connection flows. With Microsofts Visual Roundtrip Analyzer you were able to see the TCP Connection flows, but you were unable to see the HTTP Conversation within the SSL Tunnel. Pat’s tool was the first (maybe still is the only one) that allowed to see both, the TCP Connection flow as well as what happens within the SSL Tunnel.

So, back to our page. In one of our former optimization iterations we decided to do some domain sharding. With this we wanted to improve performance especially for IE6 and IE7 users, as these Browsers only open up 2 TCP Connections per domain. Resulting in a performance limit, that is often lower than what your bandwidth is able to provide.

We implemented that in a way, that we could follow another performance rule, which is “Make static content cookie-free”. So we set up another Domain,, which we used for the above mentioned domain sharding, as well as keeping most of the content cookie-free.

There are some reasonable concerns regarding domain sharding in conjunction with SSL, as you not only have an additional DNS lookup and TCP handshake, but also an additional SSL handshake, which can be quite time-consuming. But we were pretty confident, as our customers are solely based in small Germany with a RTT of ~50 ms, that the benefit of 4 connections would outweigh the impact of that additional DNS lookup and TCP/SSL Handshake.

So, when I did a run with, the result in IE7 looked like this:

Hmm… that was weird… As you can see, you have some 2 connection waterfall behaviour, but for the first 7 objects, it seems like the TCP / SSL handshake is closed after each object, and re-established for the next one. Now THAT is much more painful, than what we would have expected…

So I was almost picking up the phone to call the Webserver Admin, that probably the Connection: Keep-alive Headers were not set correctly in Apache, but I decided first to check the HTTP Headers. And they looked like this:

Request Headers:

GET /selfcare/content/staticcode/tls/kundencenter/1025300/2010-06-23-09-49-15/ui.all.css HTTP/1.1
Accept: */*
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; PTST 2.257)
Connection: Keep-Alive

Response Headers:

HTTP/1.1 200 OK
Date: Thu, 13 Jan 2011 07:46:20 GMT
Server: Apache
Expires: Fri, 14 Jan 2011 07:45:33 GMT
Cache-Control: public, max-age=3628800
Content-Language: de-DE
Vary: Accept-Encoding
Content-Encoding: gzip
Age: 47
Content-Length: 5122
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/css;charset=UTF-8

As you can see, BOTH ends have a “Connection: Keep-Alive Header” using HTTP1.1. So why in the hell was the connection closed???

So I fired up Wireshark to look at the raw packet flow. Wireshark normally does not allow me to see the content within the SSL Tunnel, but in this case it didn’t matter, as I only wanted to see, WHO is closing the connection.

So it was ME (Or my machine) which was closing the connection, not the Server. As you can see in the trace, I am the one sending the TCP FIN.

So far, so bad.

Thinking about a solution / workaround I was wondering: Only the CSS files are affected. The Headers are correct and the same as the other assets, which are downloaded later on from the same domain. And with these assets I see a perfect persistent connection… Maybe it simply is due to being hosted on another domain?

Setting up another test page I dropped the idea of domain sharding, and used just one domain. The domain, where the base page resides. And the result was:

And the issue is gone… Meanwhile the Time-to-Render has improved from 2.2 to 1.6.

As it seems, the buggy TCP Connection behaviour seems to show up, when you are delivering CSS files VIA SSL from a different domain, than the base page.

I did a further test (waterfall coming soon), where I have shifted only the CSS files back to the domain, where the basepage resides. Result is

As you can see, the connection was kept-alive during the CSS downloads. BUT, later on, when the CSS-Images were downloaded from the sharded domain, suddenly these CSS-Images showed the Connection: Close behaviour!

Now, which IE Versions are affected? IE 7, as you can see above. IE 6 as well, IE 8 and IE 9 are not also affected. (Though I have to check with IE 8 and IE 9, if the issue would only be visible, if I add a seventh CSS file in the HEAD. But then you have another issue anyway). With IE 8 and IE 9, though, the issue is a lot less impacting, as you will be hit ONLY if you have more than 6 Stylesheets referenced in HEAD. Something which you should avoid anyway.

Firefox is not affected.

Finally: Who should care (If my observations/assumptions are true. Feel free to comment)?

– If it is not SSL, you have no problem.

– If you have inlined CSS, you have no problem.

– If your CSS files and CSS Images are on the same domain as the base page, you have no problem.

– If you have just a single CSS file via SSL in the HEAD, it is cacheable and your customers have a rather low latency, you shouldn’t worry too much.

– BUT, if you have multiple CSS files, or different CSS files on each page, which are on a different domain, then it might be an issue. More so, if they are not cacheable, and even more so, when your customers DO have high latency. Remember, NO RENDERING, until all CSS files from the HEAD are received! Blank Page!

So, even though there seem to be quite a lot of  conditions that have to be met, my assumption is that quite a few pages might be hit by this issue. I coincidentally saw that on Gateway’s site for example (CSS Images via SSL on a sharded Domain. Look at the bottom of the waterfall and the corresponding HTTP Headers). And it sounds reasonable. When you follow some of the performance best practices (Domain sharding, Make static assets cookie-free, Use a CDN) and are using SSL, you probably run automatically in this scenario.

Finally a big thank you to Thomas G., Daniel G. and Michael S. who supported me in this analysis!




I got in contact with Microsofts IE Team via their blog and indeed, they confirmed the above mentioned behaviour. So this is not a unique fail case with our domain, but actually a bug in IE, present in versions 6 to 9. In the same response they said they are looking to fix this in an upcoming IE version.

See the last 2 comments:

Comments (2)

Strange IE behaviour with SSL connections

Hi there!

We stumbled upon a strange SSL/CCS Issue with IE6 and IE7 when we were tuning our homepage Alice. Found was this issue with Pat Meenans Pagetest and HTTPWatch. I am a little bit puzzled and still have not drawn conclusions yet. So here is what happened. For those not familiar with Pats Tool: Dark green is DNS Lookup, Orange is TCP Connection Setup, pink is SSL Handshake, Green is Time-to-first-Byte, and Blue is Data being received.

As the page is done via a Content Management System, we picked one page, which is pretty much representative how all the pages are constructed. The page we picked was this Rechnung, as it is delivered via SSL like almost all of the pages, but does not yet need authentication.

We first had a page with tons of assets. Something like 15 Javascripts in the HEAD, 30 CSS Files in the HEAD, and 20 Images and 20 CSS Images.  The page is normally delivered via SSL, so the initial Connection View of it looked like this:

Clearly you can see it starts to download the CSS and Javascript files on the two sockets IE7 allows per domain. Also you can see between 1.5s and 5.5s the blocking behavior of the Javascript files. Even though having two connections, it doesn’t  use them in parallel.

At roughly 5 seconds it has completed that, starts to render, and downloads all the images. So far so bad.

So we started to optimize and first concatenated all the 30 CSS files to just 5, one being IE7fixes.CSS. This gave us some improvement. You can see that a lot less CSS objects get fetched at the beginning. (The first 4 items are actually newly introduced Redirects due to an Single-Sign-On Service. Yikes.)

But you can still see the painful blocking behavior of the Javascript files.

So what we did then was to concatenate the Javascript files and introduce a second (static) domain, were we would put all the static content. Apart of the Basepage and the Javascript file, everything should reside there, Cookie-free.

And the result is this:

And this seriously puzzled me.  What we see here are at

Line 1 the Redirects and the download of the Basepage.

Line 2 download of the concatenated Javascript file, which is still on the first domain.

Line 3-7 A wonderful waterfall of two connections, with a full TCP and SSL Handshake for EACH CSS FILE…

Line 6-7 The last two SSL Connections are then reused for fetching all the images.

So what is so special about these CSS Files??? HTTP Headers are the same for all of them. Both, Request and Response Headers are HTTP/1.1 with the Header Connection: Keep-alive. And as you can see, for the images it works fine. Why not for the CSS assets?

I did a Wireshark trace and saw, that it is actually me (the Browser) who closes the connection. After receiving the data, I am sending a TCP FIN,ACK. Why does it do so?

So I tested a little bit more and found out, that this behavior is visible with IE6 on XP and IE7 on XP and Vista.

Tests with IE8 on XP showed a totally different picture. Here I got:

Now this is fine. You can see in

Line 1 and 2 that two sockets are used for the Redirects, the basepage, the Javascript and the favicon.

Line 3-4 Opening up 2 new connections to the second, static, domain to fetch the CSS files

Line 5-8 Opening up 4 more connections reaching the maximum limit of 6 connections per domain to fetch all the other assets.

THIS is how I expected the whole thing from the beginning. So I wonder what is going on with IE 6 and IE 7…?

As a final note I tried to fetch the page with IE7 via plain HTTP, no SSL. This is to some degree possible, but leaves me even more puzzled. Here it is:

Line 1 fetches the Basepage and the Javascript file via plain HTTP

Line 2 are the SSO Redirects via HTTPs

Line 3-4 are the CSS files, where it reuses the connection via Plain HTTP, and later on some images

Line 5-6 is fetching two CSS Images via SSL. These connections are NOT reused.

Line 7 fetches some more images

Line 8-13, 15 again a perfect two-connections per domain waterfall, where again CSS Images are pulled via SSL, NOT reusing these connections.

Line 14, 16 some more Images a fetched via plain HTTP, reusing the connection

So we have here with Plain HTTP a reuse of the connections for the CSS Files, the CSS Images though, fetched via SSL, show again the pattern of NOT reusing the connection.

Currently my assumption is, that IE6 and IE7 show this broken behavior, if the following is true:

MULTIPLE CSS Files, served from a DIFFERENT Domain, when fetched via SSL.

A final test would be, to put the CSS files (maybe even the CSS Images?) back on the first domain, and see what happens.

I searched a lot, but was unable to find any known bug for this. If this is indeed a bug and I am not totally misleaded with this analysis.

Any comments/help would be highly appreciated, if this is a known bug, a mistake by myself, or a workaround exists.


P.S.: In case you wonder: Some of the traces were taken from the UK, some from the US. So do not wonder about the different times to fully download etc.

Comments (3)