Reading: Computer Architecture -a Quantitative Approach

I've just managed to finish what I've been intermittently reading since April: Patterson and Hennessey, Computer Architecture -a Quantitative Approach. I'd forgotten what a profound book it was -it was just something that had been sitting on a shelf at work for over a decade, and after I'd brought it home, decided to read through it all. I started it on a four day family + borrowed child break where the lack of network connectivity would force me to engage in conversation or read -I managed to get halfway through the book. Since then, patchy.

As of today -I've re-read all of the 1995 edition, and am about to read the 2007 version that I got as an eBook.

Everyone who writes programs should read Patterson and Hennessey from time to time.

• It explains the concepts and reasoning behind the decisions that have been implemented in modern microprocessors. 
• It stops you pretending a computer just "is". It's gates made out of transistors, units built from the gates. Logic -not magic. And if you don't understand logic, you're in the wrong career, as your if() statements won't work.
• Instruction pipelines, caching, read misses, write misses and the like make it clear the performance costs of various software decisions. Once you move to languages higher level than C/C++ it's harder to deal with those consequences, but even knowing about them is useful.
• The bit on queue theory is relatively easy to understand.
• The exercises in each chapter are things you can do in your head or a bit of paper, and they keep your brain busy in a way that debugging failed unit tests doesn't. 
• The section on multiprocessing, including cache-coherency and relaxed consistency models have serious consequences for applications designed for multiprocessing -and are both an adjunct and an extension to the Java memory model that most Java programmers are oblivious too, and those people who think they understand -don't. I count myself in the "don't" category since Berlin Buzzwords -more on that another day. 

One of the fascinating things is that a lot of the work on multiprocessor & multicomputer systems in the 1980s and early 90s is a precursor to datacentre design -and hence the applications that run on them. There are some big changes

• The individual nodes in a 1980s multicomputer weren't 12-16 hyperthreaded core servers with 64+GB of Non Uniform (NUMA) memory. These are shared-bus MPU machines on their own,
• Various interconnect options were tried above and beyond Ethernet with its spanning tree routing and oversubscription issues. Crossbar switches, store and forward, other ideas.
• They didn't care about failures of anything other than disks. Well they cared- but treated them as disasters, not something to keep an eye on -and not something for the OS and application to deal with.

Some of the reasoning there does help put datacentre-scale computing into its perspective as an extension of the earlier systems, from the big supercomputers to VAXcluster systems -and knowing that history helps you see things better.

A truly excellent book.

[Artwork: Sepr at the Bearpit]

HDP released -and what it took to get there

This week the Hortonworks Data Platform has launched -to ship on Friday. I've been working with it for a while, and I'm really excited about the launch and seeing that it's getting to real people's hands.

Most of the coverage will be about features, the management console, other things for enterprise users -like availability. I'm going to look at another aspect of the product: the time and effort put in by the QA people.

The Hadoop stack is open source, and it's designed to run on everything from a few machines to very large scale clusters. Some of the design decisions only make sense at the large scale. For example, tasks are pushed out to task trackers when they check in for heartbeats or task completion events -because in a large cluster, 1000+ servers checking in generate enough traffic that pushing work out this way is the only way to stop the JobTracker overloading. This design can lead to a worse startup time in very small clusters (tip: decrease the heartbeat interval), but it's something that had to be done for the scale-up. There's all the bugs that only show up at scale -like the need to have datanodes report in on a different port number than DFS client operations, as that stops a namenode overloaded with FS requests from thinking more datanodes are down, leading to a cascade failure of the cluster. Scale problems for the big clusters -which means that nobody else will hit the same problems. And, with the big cluster running the same code as the smaller ones -all the smaller clusters -the majority- get to benefit from the ongoing performance enhancements, scheduler improvements and other features which the teams working with the large clusters can develop.

To use Hadoop in a big cluster, you need to be sure it works at scale -and with the performance to back up the scale. That's a test problem which very few organisations can address. This marks a big difference between these datacentre-scale platforms and the rest of the Apache portfolio. I can build and test my own version of Hadoop to run in small clusters (indeed, I've done just that), but it's not something that anyone can trust to work at any scale.

What's even more important is that it's not just HDFS and the MR layer, it's the entire stack. This is what the QA people have pulled off -testing everything up the stack, at scale. Then, the problems they've reported have been turned into JIRAs, then through the work of the Hortonworks developers and many others in the Hadoop dev community, patches; patches that are now in the Apache codebase.

What the QA team have done then, is help ensure that the Apache releases are qualified to work as a coherent stack even at a scale most people won't encounter. Which mean that the rest of us can be confident that it will work for us too, whether its the HDP product, or just the plain Apache artifacts you pull down in your build process(*).

Which is why, despite all the great stuff that's gone into the release, including stuff of mine that I'm really proud to have covered, I think its the work of the QA people, testing the code on the big 1000+ node cluster, that deserves credit and recognition. They are the people that have given us a stack that works.

(*) I know, there's a small catch there -different configurations- which I'll look at some other time.

[Photo: Saturday afternoon cricket in front of the Clifton Suspension Bridge]