Ingress and Egress

Last week someone from British Telecom/BT came round to boost my networking, running Fibre to the Cabinet and then re-enabling the existing Copper-to-the-Home from there.

As the graph shows, it's got a lot faster than the virgin cable: download has gone from 12.7 (vs. a promised 20 Mb/s) to ~54-55 Mb/s, a 4x improvement, while upload has gone from a throttled 2Mb/s to 15 Mbits -7x. That 7x upload speed is that I was really looking -both the ADSL and Cable offerings are weak here, with sky being the worst at 0.8Mb/s, which is pretty atrocious. The cable modem offering also suffered from collapsing under load in the evening, especially when the students were back (i.e. this time of year). I don't have that problem any more.

Now I can not only download things, I can upload them. In fact, this network is now so fast that you can see other problems. As an example, the flickr uploader used to crawl through each photo upload. Now it sprints up -so much so that the per-photo fixup at the end becomes the pause in the progress bar, not a minor detail.

Its on the downloads though, where problems arise -problems down to TCP and HTTP. HTTP likes to open a new connection on every GET/POST/PUT/HEAD/whatever operation. TCP has throttling built in to stop it flooding the network. Part of that throttling is slow-start: rather than streaming data at the full rate claimed by the far end, TCP slowly ramps its window size based on the acknowledgements coming back to it. Acknowledgements that depend on the far end getting back to the remote host -and hence the round trip time -not the bandwidth. Even though my bandwidth has improve in both directions, the distance to remote servers and the number of hops is roughly the same -only now that slow start is visible.

Take an example: the NetFlix progress bar at the start of a video. It begins, slowly filling up. Suddenly half way along it picks up speed and fills the rest of the bar in 1 second, compared to the 4-5 seconds for the first half.

What I am seeing there is latency in action.

It shows the real difference between 100Mb/s LAN and WAN connections at a sizeable fraction of that. 100MB/s LAN isn't too bad for pushing data between two boxes adjacent to each other -and ramp up time is neglible. Over a distance, its latency and round trip times that make short-lived TCP operations -of which HTTP GETs are a key example- way slower than they need to be.

Google have a paper discussing this and arguing for increasing the initial window size. For those of us with long-but-fat-pipes, this makes sense. I don't know about all those mobile things though.