British Airways: Airline of the 1%

We're sitting in SeaTac, joyous airport of the PNW, waiting for a BA flight to the UK that's running an hour late. I'm playing with the manual aperture and exposure settings on the (compact) camera, the rest of the family fiddling with some other consumer electronics.



A "fat family" sits down nearby. I think it's everyone's right to eat as much as they like, but I always feel sorry for the kids in these situations, as they get pushed into a unhealthy lifestyle from birth. What I do worry about is fat families on flights.

If they sit next to you, they end up overrunning your seat, not because they want to, but because they have no choice, not in BA fourth class seating. If they sit in the row in front then even when they sit down the millimetres-thick seat bends back, and once the seat belt light comes off they will hit the recline button, and you yourself have a tenth of a second to hit your own recline button before your kneecaps get crushed. And if they sit behind you, you know they will heave themselves up on the back of your seat in some combined groan and hacking cough, while they trundle off to the toilet. As they will do every twenty minutes for the entire transatlantic flight, as incontinence seems to go with the weight.

Needless to say, I was concerned. We'd checked in and chosen our seats 23h50 minutes earlier, not wanting to pay the £20+ plus premium for the right to choose a seat more than 24h in advance -and now I was worried where these three were going to end up. With only twenty rows in "world traveller" class, the probability of having their group of three adjoining our group of three is about 1 in 30, enough to worry about given the consequences.

Eventually the plane loads, and we stroll down the 747. This is where I get the good news: the father of the three is battened down in business class, not looking too happy as he is forced to squeeze into their biz-class cubbyholes, solid on three sides and so not ideal if you don't fit. The wife and daughter, they are in third class, "world traveller plus", the one where you actually have enough room to open a 12" netbook without fear of the passenger in front hitting the recline button and shattering your LCD. I can carry on back on the rear of the plane.

Eight or so hours later, we arrive. This is after having crossed the Arctic with the cabin crew requesting that everyone pull shut their blinds to stop people seeing the frozen-planet class scenery, instead consuming sub-DIVX quality videos on screens smaller than smart phones, screens driven by hardware "hidden" under every seat, hidden where you bump into it the moment you stretch your legs to avoid your kneecaps getting squashed. As we land, the cabin crew request that the premium travellers return their personal DVD players, finish their last round of golf, get out of the pool, return their champagne glasses. Those of us in the back are asked to take a deep breath then on the count of three everyone should flip their seat to upright.

We get to the airport, and eventually those of us at the rear get to disembark. As we do so, we get to go through world traveller plus, then through business class and up to the main fuselage's first class. All of these seats are empty and show no signs of use apart from those used by the fat family. Certainly they got value for money, but consider this: the rear third of the plane, world traveller, was 100% full, 10 people per row for 20 rows: 200 passengers. The front third of the main fuselage: three people. If the upper floor was equally populated, there may well have been a whole five people in the entire premium section of the plane.

Well, those people got something for their money: space. But with 100% utilisation in the back seating area, it was probably oversubscribed. While the premium areas were effectively wasted. You can't make money if 3/4 of the body of the plane has five passengers.

Yet BA have chosen a seat layout for their aeroplanes that are biased towards premium passengers, especially business class. A seat layout that does not -for the SEA-LHR route- appear to be working.

Which makes me think. BA's business model depends on people with the personal or organisational money to afford to pay the premium for luxuries above and beyond the basic "get from one country to another". Meanwhile, those of us in the back get to experience something a bit like the Occupy Wall Street camp without the leg room and with a cabin crew about a friendly a the NYPD.

BA is the airline of the 1%. They don't make money from the 99%; they have optimised the interiors of their planes for that 1%, while making things worse for the 99% in order to make that 1% feel better about their purchasing decisions. Indeed, I suspect that we in the back aren't there to bring in profit, we are just there to cover a bit of the costs, and to make the premium travellers feel glad about not being there with us. Just as the cabin announcements "we shall give business class travellers their four course meal shortly, with a choice of wines" are there to make us feel bad for having gone economy, the announcements for us are designed to make the premium travellers feel smug "economy class passengers will get a small plate giving them whatever is left in the box of lukewarm edibles".

There you have it. British Airways: the airline of the 1%.

Offline in OR

I am online. I have been offline. In fact I have been offline in Oregon, doing, amongst other things: skiing, on the storm-swept beaches the pacific coast, mountain biking, and, mostly, spending time with friends.


I also spent time growing what could be roughly called a beard, in some half-hearted cargo-cult attempt to stay ready for the cold of some winter skiing -and so hope that more snow arrived.


In the background you can see the child I own embracing his Oregonian birthright by holding an orange chainsaw. It wasn't switched on, but later that afternoon he did go out to the woods and cut down a tree for Christmas.


Just as every French village has its patisserie, every English village a former pub converted into four authentic rural-look flats, every small town in Oregon has a store with a bright orange logo above it: Stihl. They are the official vendors of deforestation tooling in the PNW.

The good news: there are still a lot of trees left -if there is one thing that hits you when you return to the state is just how many trees there are left.



That's probably despite the best efforts of the inhabitants, rather than because.

That peak in the background: Mary's Peak. 4000' of ascent; on-road from the south, off-road from the north, and something I used to train on. Not having something like that within day-ride range makes you lazy, so it's nice to see it again, even if I didn't manage to get back up it.

The price of taking three weeks offline is the vast amount of emails to be flicked through. It's taken two days to clear the backlog and sit down to do something vaguely useful. Time to get busy.

My other computer is a datacentre. A very small one.

Once upon a time every computer that filled a whole room, and many people had to look after it. The people who wanted to use it had to use punched cards; submit work and wait for an indeterminate period of time to see the results of that work.

Then minicomputers came, which were only the size of a cabinet in a room. These could be shared between less people, and with terminals, be more interactive. And from these machines came Unix, time_t, and buffer overflows in sprintf() statements.

It was still shared; you still had to compete with other people for finite resources. This is why when the microcomputer came along, and then the interactive workstation, something profound happened: you could do more. You could write interactive programs and be more productive without the long waits. That was before web browser updates, interactive downloading of emergency flash updates and AV Scanners got in the way of productive work, so desktop workstations were actually useful. Of course, all these machines were not that well connected on their own, so people would run round holding floppy disks, until networking products became available. Ethernet itself dates from the era of the workstation, though apparently it's frame length is partly driven by DEC's need to keep memory costs down, hence its pitiful size today. Ethernet, Netware, email and the like evolved, so now people can send me emails inviting me to phone conferences at 01:00 UK time, invitations that will somehow be tentatively stuck in my shared calendar without my knowledge, then synchronised to my phone, so that it can wake me from my sleep to let me know that I am late for a phone conference that I didn't know about. Truly, we are blessed.

And yet the wheel goes round. What is fashionable again? The Datacentre. A room full of boxes turning electrons into waste heat the way thermodynamic entropy requires, routing many of the through the CPUs in the process, so doing useful things. The first big datacentres used many racks of machines, each box with 1-2 cpus, a couple of 512MB HDDs and 1 GbE between them. Now you could build up a single rack with the same storage capacity and compute power.

But you'd still have to share it. Which is why I'm pleased to show off a little addition to our facility: a very small Hadoop cluster.



These are four SL390s servers in 2 U's of rack; the two bits below are just expansion slots in the 4U chassis.

Each one the same basic node used in one of the top 10 supercomputers, though they have many more units Infiniband interconnect and a total power budget of 1.4MW, which is not something I'd want.

The front of the boxes contain all the interconnects; 2x10GbE on the motherboard, and a management ports hooked up to 100 MbE for ILO management;.. Having the ports at the front is something that Allen W has complained about. It does make sense if your switches work that way, if you can set your hold/cold aisles up so that the fronts are accessible. If your switch has its ports at the back, well, it's "suboptimal"

Round the back: power only. Shared PSUs and fans for a bit more resilience.



The twin-socketed Xeon E-series parts have a relatively low power budget for x86-64 servers, though not in ARM terms; the multiple SFF HDDs you can fit into each unit give pretty good bandwidth and 4TB of storage. If you opt for the 3.5" HDDs your bandwidth drops, but you get 6TB/node.

Then there's the RAM: up to 192 GB/node. These ones have a bit less than that in, but there's still more per core than the entire RAM supply in my house.

From a storage perspective, there's not that much capacity: 16-24 TB. The ratio of store:compute and drive:compute is pretty good though, and as you can also sneak in a GPU, if you have compute intensive work, these four machines make for a nice little setup. And given that those TB of storage don't need to be shared with anyone else, it's not so bad.

This then is capable of storing and working through a reasonable amount of data, building up complex in-memory structures and being as responsive mid-afternoon as it is on a weekend, as nobody else is trying to do stuff on it.

At this scale HDFS makes no sense. You don't have the capacity to handle a server failure; 3X replication is too expensive. Better to RAID everything and NFS cross mount the filesystems.

I know the big cluster people will look at these boxes with bemusement, but think about this

1. It's not the only cluster I have access to. This one is free for me to play with new versions and code on without causing problems.
2. I'm sure the mainframe people didn't think much of minicomputers, minicomputer aficionados looked down on desktop computers, and -as we can see- desktop computers are having do accept the growing functionality of phones and other devices.

This then, is my own personal datacentre.

Alternate Hadoop filesystems

My coverage of Netapp's Hadoop story not only generated a blip in network traffic to the site, but a blip in people from NetApp viewing my LinkedIn profile (remember: the mining cuts both ways), including one Val Bercovici -I hope nobody was too offended. That's why I stuck in a disclaimer about my competence*.


I'm not against running MapReduce -or the entire Hadoop stack- against alternate filesystems. There are some good cases where it makes sense. Other filesystems offer security, NFS mounting, the ability to be used by other applications and other features. HDFS is designed to scale well on "commodity" hardware, (where servers containing Xeon E5 series parts with 64+GB RAM, 10GbE and 8-12 SFF HDDs are considered a subset of "commodity"). HDFS makes some key assumptions

1. The hardware is unreliable --replication addresses this.
2. Disk and network bandwidth is limited --again, replication addresses this.
3. Failures are mostly independent, therefore replication is a good strategy for avoiding data loss
4. Sub-POSIX semantics are all you need --makes handling partitioning easier, locking, etc.
   

The independence of failure is something I will have to look at some other time, the key thing being it doesn't extend to virtualised "cloud" infrastructures" where all your VMs may be hosted on the same physical machine. A topic for another day.

What I will point to instead is a paper by Xyratex looking at running Hadoop MR jobs over Lustre. I'm not actually going to argue with this paper (much) as it does appear to show substantial performance benefits with the combination of Lustre + Infiniband. They then cost it out and argue that for the same amount of storage, the RAIDed LustreFS capacity is less and that delivers long term power savings as well as purchase price.

I don't know if the performance numbers stand up, but at the very least Lustre does offer other features that HDFS doesn't: it's more general purpose, NFS mountable, supports many small files, and, with the right network, delivers lots of data to anywhere in the cluster. Also Eric Barton of Whamcloud bought us beer at the Hillgrove at our last Hadoop get together.

Cost wise, I look at those number and don't know what to make of them. On page 12 of their paper (#14 in the PDF file) and say that you only need to spend 100x$7500 for the number of nodes in the cluster, so the extra costs of Infiniband networking , and $104K for the storage are justifiable, as the total cluster capital cost comes in at half, and the power budget would be half too, leading to lower running costs. These tables I would question.

They've achieved much of the cost saving by saying "oh look, you only need half as many servers!" True, but that's cut your computation power in half too. You'd have to look hard at the overhead imposed by the datanode work on your jobs to be sure that you really can go down to half as many disks. Oh, and each node would gain from having a bit of local FS storage for logs and overspill data, because it costs more in the big servers, and local storage is fine there.

IBM have stood Hadoop up against GPFS. This makes sense if you have an HPC cluster with a GPFS filesystem nearby,  want an easier programming model than MPI --or the ability to re-use the Hadoop++ layers. GPFS delivers fantastic bandwidth to any node in the cluster, it just makes the cost of storage high. You may want to consider having HDDs in the nodes, using that for the low value bulk data, and using GPFS for output data, commonly read data, and anything where your app does want to seek() a lot. There's nothing in the DFS APIs that stop you having the job input or output FS separate from your fs.default.name, after all.
When running Hadoop in AWS EC2, again the S3 FS is a good ingress/egress FS. Nobody uses it for the intermediate work, but the pay by the kilo storage costs are lower than the blockstore rates, and where you want to keep the cold data.

There's also the work my colleague Johannes did on deploying Hadoop inside the IBRIX filesystem:

High availability hadoop november 2010

This design doesn't try and deploy HDFS above an existing RAIDed storage system; it runs location-aware work inside the filesystem itself. What does that give you?

• The existing FS features: POSIX, mounting, etc.
• Different scalability behaviours.
• Different failure modes: bits of the FS can go offline, and if one server fails another can take over that storage
• RAID-level replication, not 3X.
• Much better management tooling.

I don't know what the purchasing costs would be -talk to someone in marketing there- but I do know the technical limits.

• Less places to run code next to the data, so more network traffic.
• Rack failures would take data offline completely until it returned. And as we know, rack failures are correlated
• Classic filesystems are very fussy about OS versions and testing, which may create conflict between the needs of the Hadoop layers and the OS/FS.

Even so: all these designs can make sense, depending on your needs. What I was unimpressed by was trying to bring up HDFS on top of hardware that offered good availability guarantees anyway -because HDFS is designed to expect and recover from failure of medium availability hardware.

(*) For anyone doubts those claims about my competence and the state of the kitchen after I bake anything would have changed their opinions after seeing what happened when I managed to drop a glass pint bottle of milk onto a counter in the middle of the kitchen. I didn't know milk could travel that far.

[Photo: Hillgrove Porter Stores. A fine drinking establishment on the hill above Stokes Croft. Many beers are available, along with cheesy chips]

Attack Surface

I allege that

1. Any program that is capable of parsing untrusted content is vulnerable to exploitation by malicious entities.
   
2. Programs which parse binary file contents are particularly vulnerable due to the tendency of such formats to use offset pointers and other values which the parsers assume are valid -but which "fuzzing" can be used to discover new exploits.
   
3. Programs that treat the data as containing executable content -even within a sandbox- are vulnerable to any exploit that permits code to escape the sandbox, or simply to Denial of Service attacks in which resources in the sandbox such as memory, CPU time and network bandwidth are consumed.
   
4. Recent incidents involving the generation of false certificates for popular web sites have shown that signed web sites and signed content cannot be consistently trusted.
   
5. Any program that can process content downloaded by an HTTP client application is capable of parsing untrusted content, and so vulnerable to exploitation.
   
6. Any program that can process content received in the payload of email messages is capable of parsing untrusted content, and so vulnerable to exploitation.
   
7. Any program that can process delivered in untrusted physical media is capable of parsing untrusted content, and so vulnerable to exploitation.
   

The last of these items concerns me today. The Stuxnet worm has shown how a well-engineered piece of malware can propagate between windows machines, and across an airgap from windows machines to SCADA infrastructure by way of USB infection. The Sony Rootkit Scandal showed that even large business were willing to use this to sneak on anti-copying features on what pretended to be a music CD.

A key conclusion from the CERT advisory on the Sony Incident was Disable automatically running CD-ROMs by editing the registry to change the Autorun value to 0 (zero) as described in Microsoft Article 155217.


I've long disabled autorun on windows boxes, and I also remember testing a long time ago whether or not you could autorun a CD while the screen was locked. Answer: no. Windows waits until the screen is unlocked before an autorun application is executed. This provides some defence against malicious CDs, though not as much as disabling autorun completely.


I don't worry too much about windows security these days because my main machines are Linux boxes, which keep themselves up to date with apt-get update. This keeps browsers, flash, JVMs, OpenOffice current, which reduces their vulnerability. I also keep an eye on the SANS newsfeed to see if there are any 0-day exploits out, and take proactive action if anything gets out into the wild that we are at risk from. I've stopped fielding service requests from family members saying "there is something on my machine -can you clean it up". If I am asked, my response is "I will reformat your HDD and install ubuntu on it". FWIW, I have done this for one relative and it works really well.


I am confident that I do keep my machines locked down, and know that the primary vulnerability on an Linux system on which I do open source development is that I implicitly have to trust everyone who has commit rights to every application that I use on it.


If there is one issue I have with Linux is that it is over-paranoid. For example, once the screen is locked I can't suspend the system. This means that I can't suspend the laptop until I unlock it and hit the suspend button. Somewhere on launchpad there was a bugrep about that, but people who looked after servers were saying "no!", despite the fact that if I had physical access to the power button, it would be only four-seconds away from a hard power off, and then, unless the system had the CD-ROM, USB and PXE boot disabled, one to two minutes from being my box. It's annoying but a sign that security is taken over-seriously.


Imagine my surprise then, last week, on inserting a CD-ROM from an external source when suddenly the system started grinding away, with a tiny thumbnail of each PDF image appearing one after the other. For that to happen the desktop had to have scanned through the list of files, handed each PDF file off to some parser and got a 64x64 graphic back. It wasn't the wasted effort that concerned me -it was the way it did this without me even asking.


I did some research and discovered an article on the topic. Essentially the Linux desktop standard allows for autostart.sh files to be executed on mounting a media device. The the thumbnailing is something else: nautilus defaults to doing this for local files -and views all files on mounted local media as "local", rather than "untrusted content on an untrusted filesystem". Given that exploits have been found in the thumbnailing code, having Nautilus thumbnail files on an external device is insane.


The best bit: this happens even while the desktop is locked.


There I am, regularly saying rude words because I thought I'd suspended my laptop, but as the screen was locked it's just stayed powered up, slowly killing the battery and/or overheading in a bag. I accepted this inconvenience as I thought it was a security feature for my own good. It turns out that it was irrelevant as all you needed to do was plug in a USB stick with a any .dvi, .png or .pdf file designed to exploit some security hole and my box is 0wned. Which, if done while I am logged in, offers access to the encrypted bits of the box.

There is a workaround -disable all thumbnailing even of local content, as well as the usual "do nothing" action on handling media insertions. I've done that everywhere. Yet what annoys me is this: what were the dev team thinking. On the one hand there is the power management group saying "we won't let people suspend while the desktop is locked for security reasons", while some other people are thinking "let's generate thumbnails whenever new media is inserted". The strength of OSS is that anyone can contribute. The weakness: a lot of them aren't devious enough to recognise what they think of a feature is someone else's route of access to your systems


[Photo: Park Street]

Disclaimer: I reserve the right to not know what I'm talking about

There have been some links to this blog with the implications that either because I am a researcher at HP Laboratiories with commit rights to a number of projects -including Apache Hadoop- then my opinions may have some value.

Not so.

Everything I say is obviously my own opinions, do not reflect those of my employer, may be at odds with corporate strategies of which I am blissfully unaware of, if I do mention corporate plans they may change on a whim, etc, etc.

I do work at HP Labs -which is a fun place to work- but think about what that means. It means that I don't do stuff that goes straight into shipping products. The code I write is not considered safe for humanity. I may talk about it, I may write about it, some may get into the field, but only after rigorous review by people who care about such things.

This makes it a bit like a university, except that I don't teach courses. Cleary I am not considered safe near students either, in case I damage thier minds.

"Oh well", people say, "he's a Hadoop committer -that must count for something?"

It does, but I don't get to spend enough time there to keep up with the changes, let alone do interesting stuff of my own ideas.

Whenever I create an issue and say "I will do this", other people in the project -Arun, Tom, Todd and others will stare at their inbox in despair, with the same expression on their faces that my family adopts when I say "I will bake something".

Because then, yes, I may come up with something unusual and possibly even tasty, but they will know that the kitchen will be left with a thin layer of flour over everything, there will be pools of milk, and whatever went in the oven went into the closest bowl-shaped thing I could reach with one hand, rather than anything designed to be used in an oven, let alone watertight. Even if I didn't use flour or milk.

Bear that in mind before believing anything written here.

[Photo:me somewhere in the Belledone mountains; French Alps. For some reason French villages never like it when people on bicycles come to a halt]

Towards a Topology of Failure

The Apache community is not just the mailing lists and the get togethers: it is the planet apache aggregate blog; this lets other committers share their thoughts -and look at yours. This makes for unexpected connections.

After I posted my comments on Availability in Globally Distributed Storage Systems, Phil Steitz posted wonderful article on the mathematics behind it. This impressed me, not least because of his ability to get TeX-grade equations into HTML. What he did do was look at the real theory behind it, and even attempted to implement Dynamic Programming solution the problem.

I'm not going to be that ambitious, but I will try and link this paper -and the other ones on server failures, into a new concept, "Failure Topology". This is an excessively pretentious phrase, but I like it -if it ever takes off I can claim I was the first person to use it, as I can do with "continuous deployment"

The key concept of Failure Topology is that failures of systems often follow topologies. Rack outages can be caused by rack-level upgrades. Switch-level outages can take out 1* racks and are are driven by the network topology. Power outages can be caused by the failure of power supplies to specific servers, sets of servers, specific racks or even quadrants of a datacentre. There's the also the notion of specific hardware instances, such as server batches with the same run of HDDs or CPU revisions.

Failure topologies, then, are maps of the computing infrastructure that show how these things are related, and where the risk lies. A power topology would be a map of the power input to the datacentre. We have the switch topology for Hadoop, but it is optimised for network efficiency, rather than looking at the risk of switch failure. A failure-aware topology would need to know which racks were protected by duplicate ToR switches and view them as less at risk then single-switch racks. Across multiple sites you'd need to look at the regional power grids, the different telcos. Then is the politics overlay: what government controls the datacentre sites; whether or not that government is part of the EU and hence has data protection rights, or whether there's some DMCA-style takedown rules.

You'd also need to look at physical issues: fault lines, whether the sites were downwind of Mt St Helen's class volcanoes. That goes from abstract topologies to physical maps.

What does all this mean? Well, in Disk-Locality in Datacenter Computing Considered Irrelevant, Ganesha Ananthanarayanan argues that as switch and backplane bandwidth increases you don't have to worry about where your code runs relative to the data. I concur: with 10GbE and emerging backplanes, network bandwidth means that switch-local vs switch-remote will become less important. Which means you can stop worrying about Hadoop topology scripts driving code execution policies. Doing this now opens a new possibility:

Writing a script to model the failure topology of the datacentre.

You want to move from a simple "/switch2/node21" map to one that includes power sources, switches, racks, shared-PSUs in servers, something like "/ups1/switch2/rack3/psu2/node21". This models not the network hierarchy, but the failure topology of the site. Admittedly, it assumes that switches and racks share the same UPS, but if the switch power source goes away, the rack is partitioned and effectively offline anyway -so you may as well do that.

I haven't played with this yet, but as I progress with my patch to allow Hadoop to focus on keeping all blocks on separate racks for availability, this failure topology notion could be the longer term topology that the ops team need to define.

[Photo: sunrise from the snowhole on the (dormant) volcano Mt Hood -not far from Google's Dalles datacentre]