Safety in High Energy Physics and US Visitor paperwork

The NYT article on US immigration and ESTA makes me think that I should publish the safety lecture from the November 2010 Bristol Hadoop workshop, which was hosted by Bristol University Physics Dept. All signs in the slides are from their physics building, except for the last slide.




The last form is genuine it's hosted on the state department as the 1405-0134 form, I've had to fill it in a couple of times. Things I like about it

• I can cross more countries on a one day alpine bike ride than there is room for in the "countries you have visited" section. 
• Giving to a charity is clearly something they don't expect people to do much of -again, space for two or three, and no time limit on how far back you must list your donations.
• In places like Boise, Idaho, having firearms training is something they ought to give visitors, not ask if they have it.

Because form explicitly says "nuclear experience", the HEP folk can get away with saying no. Saying you work with antimatter or neutron beams is not the thing you want to do at an immigration border.

While taking the on-site photos I got cornered by the University site security people for taking pictures with an SLR as it is "what the police warned them about". I didn't point out to them that if I wanted to take photos discreetly I'd use a camera phone or my HD-resolution cycle helmet cam on the bike helmet I'd carry nonchalantly under one arm -as that would only make them think I was planning something. Better to stick to the idea that enemies of the state use SLRs -so anyone with an SLR is potentially an enemy. Anyway, I didn't argue, just sat there, let them look at the photos while verifying that I was visiting the physics dept. At some point the chief minion started talking about deleting one of the paper sign listing how many gigabequerels they had. Ignoring the fact that such info is available online, having that photo deleted would have wasted 15 minutes of my life once I got back home and undeleted it. At least they didn't ask to look at the laptop, as that would have created conflict.

Returning to ESTA, here's something funny about it. There is no online way to see if it expires. Apparently you get a renewal email, but I've never seen one, and last november I tried to see if mine was still valid before flew to the US. I didn't get a reply until after I'd flown out:

Dear Stephen,
I am sorry we were not able to respond to your question sooner. Hopefully you did not have any problems traveling to the US, but please write back if you still need help.

That is -we hope it hadn't expired yet because you'd have been stuffed if it had.

Hadoop in Cloud Infrastructures

People say "should you run Hadoop in the cloud?". I say "it depends".

I think there is value in Hadoop-in-cloud, I talked about doing it in 2010 at Berlin Buzzwords 2010; since then I've had more experience with using Hadoop and implementing cloud infrastructures.

1. If your data is stored in a cloud provider's storage infrastructure, doing the analysis locally is the only rational action. It's that "work near the data" philosophy.
2. If you are only doing some computation -say nightly- then you can rent some cluster time. Even if compute performance is worse, you can just rent some more machines to compensate.
3. You may be able to achieve better security through isolation of clusters (depends on your IaaS vendor's abilities).
4. No upfront capex; fund from ongoing revenue.
5. Easier to expand your cluster; no need to buy more racks, find more rack space.
6. You don't need to care about the problems of networking.
7. Less of a problem of heterogenous clusters if you expand later.

Against that

1. Cost of data storage can only increase at a rate proportional to ingress/retention rates.
2. Cost of cluster time increases at a rate proportional to analysis performed. There is no "spare cluster time" for low priority work.
3. Even if CPU time can scale up, IO rate of persistent data may not.
4. Hadoop contains lots of assumptions about running in a static infrastructure; it's scheduling and recovery algorithms assume this.

Some examples of where Hadoop's assumptions diverge from that of cloud infrastructures:

• HDFS assumes failures are independent, and places data accordingly (Google's Availability in Globally Distributed Storage Systems paper shows this doesn't hold in physical infrastructures, my notion of failure topologies expands on that)
• MR blacklists failing machines, rather than releasing them and requesting new ones.
• Worker nodes handle failure of master nodes by spinning on the hostname, not querying (dynamic) configuration data for new hostnames. Some of the HA HDFS may address that, I'm not tracking it enough.
• Topology scripts are static. I've been slowly tweaking topology logic in 0.23+ but haven't put the dynamicness in there yet (HDFS and MR cache (name->rack) mappings on the assumption that the data is coming from slow to exec scripts, not fast & refreshable in-VM data).
• Schedulers assume #of machines are static, don't allocate and release compute nodes based on demand and with knowledge of cost and quantum of CPU rentals. (I'm not sure quantum is the right term there, I mean the fact that VMs may be rented by the hour, 15 minutes, etc, so your scheduler should retain them for 59 minutes after acquiring them.
• Scheduling doesn't bill different users for their cluster use in a way that is easily mapped to cluster time.

A lot of these are tractable, you just have to put in the effort. The Stratosphere team in Berlin are doing lots of excellent work here, including taking a higher level query language and generating an execution plan that is IaaS aware -you can optimise for fast (many machines) or lower cost (use less machines more efficiently).

In comparison, a physical cluster:

• Offers a lower cost/TB of any corporate filestore to date other than people's desktop computers (which have a high TCO and power cost that is generally ignored), so enables you to store lots of stuff you would otherwise discard.
• Let's you choose the hardware optimised for your current and predicted workloads.
• Has free time for the low priority background work as well as the quicker queries that near-real-time UIs like.
• May be directly accessible from desktops in the organisation (depends on security model of cluster).
• Is easily hooked up to Jenkins infrastructure for execution of work as CI jobs.
• Let's you do fancy tricks like striping of different MR versions across the racks for in-rack locality and different sets of task trackers for foreground vs background work, and different JTs (reduces memory use, cost of failure, etc).
• Is way, way easier to hook up to internal databases, log feeds. To do ETL into your corporate oracle servers, you will need to run something behind the firewall to fetch it off the IaaS storage layer, rather than have your reducers push it to the RDBMS itself.

If you are generating data in house, in house clusters make a lot of sense.
This is why I say "it depends" -it depends on where you collect your data and what you plan to do with it.

As for the way Hadoop doesn't currently work so well in such infrastructures, well the code is there for people to fix. It's also a lot easier to test in-cloud behaviour, including resilience to failure, than it is with physical clusters.

[Photo: Descending Mt Ranier, 2000]

The datacentre is the new laptop

HP has just announced its forthcoming Gen 8 servers. Rather than go on about the usual stuff: CPUs, I/O bandwidth etc., or even the trend to put solid state storage off the PCIx bus, what's interesting to me is this: the servers are explicitly designed to be part of a larger system, a datacentre.

The existing products, they are individual servers you just happen to put into racks, you just happen to hook up to a switch you've stuck at the top of the same rack. The racks may be set up into hot rack/cold rack, but that's mostly a deployment detail the servers don't care about, except in ensuring airflow is good.

This has now changed.

The Datacenter as a computer argued that the software developers need to recognise that a datacentre is the new execution platform, one with mixed availability, limited bandwidth and other concerns that could be ignored before -or at least treated as the special case of "distributed systems", rather than what we have now: "systems". Everything is distributed.

These hardware changes mirror that. Here are some of the new concerns for both the ops team -and the applications themselves.

1. Re-integration of Storage and Computation.
   
2. Availability though replication. Less RAID-style hardware, more
   replication across machines.
3. Inventory tracking -especially for identifying failure points, such as monitoring the history of specific batches of disks. If some appear particularly unreliable, you want to find all of them.
4. Networking: 10 GbE is still a luxury, bonded 2x1 GbE good for availability too. Understanding network failures in data centers shows why ToR switches become the dominant network failure point in a cluster -and from a re-replication perspective, that's not ideal.
   
5. Power management. Beyond just PUE, the metric of datacentre overhead, power consumption in the servers is a big concern.

The new servers then, are designed to live in this world.

Inventory They work out from the rack (don't ask me how, I don't know these things) where they are on it -information that can be propagated to the management tools so that they can be used for inventory tracking.

Networking Lots of ethernet ports. Some slow and inexpensive for management, faster ones for the application.

Power. This work here is something you can point to Chrandrakant Patel in HP Labs for. If you look at his published work, you can see a lot of it is about airflow and cooling in a datacentre. If you can improve that -as the container hosted datacentre pods can do- then your PUE is better. Why instrument the inside of the servers? It ensures that you can keep the hardware within its limits, because you have a better idea of what is going on inside. Every extra degree F, C or K you can take the air up, lets you save a lot of money over time. Yet the risk of overheating -and the cost of doing so- makes this dangerous. Knowing what is happening inside the servers give you more confidence of what's happening.

This is what the new servers enable. Which means that we are going from servers that you stack to servers that are designed to locate themselves in the racks, ideally hosted within a datacentre container that is optimised for airflow and designed to work as close to the limits of temperature as is considered safe based on the information coming out of the servers themselves.

Which is very close to what a laptop does: a box with optimised airflow and fans that come on when they feel it is important, and with a power budget that the system is designed to optimise. The datacentre is the new laptop, at least from a power and cooling perspective.


Now, what about the software? If the datacentre-level application infrastructure can get at the power, topology and network information, it could adapt itself better.

The topology information that the servers can determine could be used to dynamically generate the topology map for the cluster. It is entirely co-incidental that I'm typing this while my new topology patches are being tested in a adjacent console, but those changes (better support for topology sources other than the script runner, ability to dump the current topology) are effectively a precursor. I wouldn't do some fancy integrated java module though -better to have a topology source that just reads a java properties file and by polling for changes, can react to moving topologies. Let the management tooling generate that and it would propagate into HDFS and the RM/MR layer.

Power? If overheating is a problem, that server can be clocked back, which makes it slower. It may be better to actually tell the resource manager that there are less slots on that box, so reducing its actual workload. This could ensure that the work running in the remaining slots doesn't take longer than normal to complete.

Networking? We really need a way to get more information about the network backplane into the application -including the amount of bandwidth currently allocated to applications. Bandwidth can be a precious resource, but right now there is better tooling to manage it in a bittorrent client than there is between applications in a datacentre.

This is a challenge and an opportunity. A challenge: this information needs to be extracted and forwarded to the applications -which then need to act on it. An opportunity -it will make the applications and datacentres work better. Wave goodbye to writing topology scripts that don't work, say hello to being able to move servers around and have them the application infrastructure work out where they are. Worry less about uncontrolled backbone bandwidth use in a shared datacentre; have some policy tooling to manage it across applications. As for power, hope to see the electricity bills decrease.


[Artwork: Sepr on Jamaica Street, Stokes Croft]

Just because you can rewrite your codebase doesn't mean you should have to

The strength of automated driven test suites is that you can verify that all the testable/covered parts of your system still work, even after major changes.

The strength of modern IDEs is at a click of mouse you can find wherever a class or method is used, so go to those places and edit them.

Even so: availability should not imply necessity: life is best when you don't have to use these features unless you really, really want to. Everything should work.

And usually it does. But not last week. A small bug surfaced on tuesday: you got an error marshalling strings with square brackets around, "[]". At first I did the obvious tactic: deny that this could be happening, but after repeated evidence to the contrary I sat down and had a look.

It turns out that the library, json-lib, with the same interface as the java Map interfaces, has an extra "feature" in that whenever you add a string attribute, that string value gets parsed as JSON if it starts with "{" or "[". Which means that parentnode.put("request","4,5") would add the string attribute "request":"4,5" to the parent node, the slight variation parentnode.put("request","[4,5]") would generate the attribute "request":[ 4 , 5]. This is fundamentally different, and challenges any assumption the recipient had about types in marshalled data, as the types varies depending on the contents of the strings being marshalled.

Needless to say, I was unhappy. At least by the point that unhappiness was reached, there were now some tests to work out what was going wrong. Which made fixing it possible, the fix being to always single quote strings being added, such as parentnode.put("request","'[4,5]'"). When the put() operation is invoked, the single quotes are stripped, and the unparsed inner values become the attribute. With careful wrapping of the put operations -and ensuring that only one set of double quotes is placed around every string attribute, the tests passed, everything worked, and a new nightly release went out with the issue marked as closed.

Except the next day, the issues came back. Because it wasn't fixed. Not once you took that node, parentnode, and added it under another node: messagenode.put("payload", parentnode).

Doing that appears to trigger a reparse of every string value. Which means all safely planted array strings end up being reparsed, and converted from strings to JSON arrays.

At this point, the unhappiness level changed from "medium" to excessive. With the new tests replicating this behaviour, and no obvious in-source switch to say "be less helpful", the only solution that seemed 100% likely to work with all possible payloads and orderings of payload construction was to rip out the entire library and replace it with one that did not exhibit the same behaviour.

Which is what I spent Thursday and Friday doing: a complete removal of all uses of json-lib and replacement with Jackson. Which, I not only regret in time wasted, but in the way that the json-lib Java-friendly object model is better than the Jackson "look like the DOM" world view, because the DOM, ubiquitous as it is, is pretty painful. Yes, with experience of the XML parsing world, I can certainly use it -but that did not mean I enjoy it. I've had to rip out and place server side and client side code, other things that are visible to other bits of the system that used the same JSON library. I haven't fixed all that code -but add enough downconvert/upconvert that the code compiles and appears to work.

Four hours over two days writing tests to show this problem existed -then
two full days of repair: switching libraries, backtracking on method calls, changing types and seeing won't compile, running tests. Some extra tests this weekend and then only the merge with two days worth of other peoples changes and some reruns and it's ready to commit. That will leave Jenkins to do the final retest and email, then everyone downstream will have to fix their side of the down- and up- converted code (the conversion methods marked as @Deprecated to make them easy to spot), which will take 1+ hour on monday for a couple of people.

When we look back at this, what will we be able to say?

We can now reliably send strings with square brackets around.

This is one of those weeks that I would never use in a motivational talk for anyone interested in taking software engineering.

[Artwork: 3Dom, "Remember the Future", Richmond Road, Montpelier]

British Airways: Airline of the 1%

We're sitting in SeaTac, joyous airport of the PNW, waiting for a BA flight to the UK that's running an hour late. I'm playing with the manual aperture and exposure settings on the (compact) camera, the rest of the family fiddling with some other consumer electronics.



A "fat family" sits down nearby. I think it's everyone's right to eat as much as they like, but I always feel sorry for the kids in these situations, as they get pushed into a unhealthy lifestyle from birth. What I do worry about is fat families on flights.

If they sit next to you, they end up overrunning your seat, not because they want to, but because they have no choice, not in BA fourth class seating. If they sit in the row in front then even when they sit down the millimetres-thick seat bends back, and once the seat belt light comes off they will hit the recline button, and you yourself have a tenth of a second to hit your own recline button before your kneecaps get crushed. And if they sit behind you, you know they will heave themselves up on the back of your seat in some combined groan and hacking cough, while they trundle off to the toilet. As they will do every twenty minutes for the entire transatlantic flight, as incontinence seems to go with the weight.

Needless to say, I was concerned. We'd checked in and chosen our seats 23h50 minutes earlier, not wanting to pay the £20+ plus premium for the right to choose a seat more than 24h in advance -and now I was worried where these three were going to end up. With only twenty rows in "world traveller" class, the probability of having their group of three adjoining our group of three is about 1 in 30, enough to worry about given the consequences.

Eventually the plane loads, and we stroll down the 747. This is where I get the good news: the father of the three is battened down in business class, not looking too happy as he is forced to squeeze into their biz-class cubbyholes, solid on three sides and so not ideal if you don't fit. The wife and daughter, they are in third class, "world traveller plus", the one where you actually have enough room to open a 12" netbook without fear of the passenger in front hitting the recline button and shattering your LCD. I can carry on back on the rear of the plane.

Eight or so hours later, we arrive. This is after having crossed the Arctic with the cabin crew requesting that everyone pull shut their blinds to stop people seeing the frozen-planet class scenery, instead consuming sub-DIVX quality videos on screens smaller than smart phones, screens driven by hardware "hidden" under every seat, hidden where you bump into it the moment you stretch your legs to avoid your kneecaps getting squashed. As we land, the cabin crew request that the premium travellers return their personal DVD players, finish their last round of golf, get out of the pool, return their champagne glasses. Those of us in the back are asked to take a deep breath then on the count of three everyone should flip their seat to upright.

We get to the airport, and eventually those of us at the rear get to disembark. As we do so, we get to go through world traveller plus, then through business class and up to the main fuselage's first class. All of these seats are empty and show no signs of use apart from those used by the fat family. Certainly they got value for money, but consider this: the rear third of the plane, world traveller, was 100% full, 10 people per row for 20 rows: 200 passengers. The front third of the main fuselage: three people. If the upper floor was equally populated, there may well have been a whole five people in the entire premium section of the plane.

Well, those people got something for their money: space. But with 100% utilisation in the back seating area, it was probably oversubscribed. While the premium areas were effectively wasted. You can't make money if 3/4 of the body of the plane has five passengers.

Yet BA have chosen a seat layout for their aeroplanes that are biased towards premium passengers, especially business class. A seat layout that does not -for the SEA-LHR route- appear to be working.

Which makes me think. BA's business model depends on people with the personal or organisational money to afford to pay the premium for luxuries above and beyond the basic "get from one country to another". Meanwhile, those of us in the back get to experience something a bit like the Occupy Wall Street camp without the leg room and with a cabin crew about a friendly a the NYPD.

BA is the airline of the 1%. They don't make money from the 99%; they have optimised the interiors of their planes for that 1%, while making things worse for the 99% in order to make that 1% feel better about their purchasing decisions. Indeed, I suspect that we in the back aren't there to bring in profit, we are just there to cover a bit of the costs, and to make the premium travellers feel glad about not being there with us. Just as the cabin announcements "we shall give business class travellers their four course meal shortly, with a choice of wines" are there to make us feel bad for having gone economy, the announcements for us are designed to make the premium travellers feel smug "economy class passengers will get a small plate giving them whatever is left in the box of lukewarm edibles".

There you have it. British Airways: the airline of the 1%.

Offline in OR

I am online. I have been offline. In fact I have been offline in Oregon, doing, amongst other things: skiing, on the storm-swept beaches the pacific coast, mountain biking, and, mostly, spending time with friends.


I also spent time growing what could be roughly called a beard, in some half-hearted cargo-cult attempt to stay ready for the cold of some winter skiing -and so hope that more snow arrived.


In the background you can see the child I own embracing his Oregonian birthright by holding an orange chainsaw. It wasn't switched on, but later that afternoon he did go out to the woods and cut down a tree for Christmas.


Just as every French village has its patisserie, every English village a former pub converted into four authentic rural-look flats, every small town in Oregon has a store with a bright orange logo above it: Stihl. They are the official vendors of deforestation tooling in the PNW.

The good news: there are still a lot of trees left -if there is one thing that hits you when you return to the state is just how many trees there are left.



That's probably despite the best efforts of the inhabitants, rather than because.

That peak in the background: Mary's Peak. 4000' of ascent; on-road from the south, off-road from the north, and something I used to train on. Not having something like that within day-ride range makes you lazy, so it's nice to see it again, even if I didn't manage to get back up it.

The price of taking three weeks offline is the vast amount of emails to be flicked through. It's taken two days to clear the backlog and sit down to do something vaguely useful. Time to get busy.

My other computer is a datacentre. A very small one.

Once upon a time every computer that filled a whole room, and many people had to look after it. The people who wanted to use it had to use punched cards; submit work and wait for an indeterminate period of time to see the results of that work.

Then minicomputers came, which were only the size of a cabinet in a room. These could be shared between less people, and with terminals, be more interactive. And from these machines came Unix, time_t, and buffer overflows in sprintf() statements.

It was still shared; you still had to compete with other people for finite resources. This is why when the microcomputer came along, and then the interactive workstation, something profound happened: you could do more. You could write interactive programs and be more productive without the long waits. That was before web browser updates, interactive downloading of emergency flash updates and AV Scanners got in the way of productive work, so desktop workstations were actually useful. Of course, all these machines were not that well connected on their own, so people would run round holding floppy disks, until networking products became available. Ethernet itself dates from the era of the workstation, though apparently it's frame length is partly driven by DEC's need to keep memory costs down, hence its pitiful size today. Ethernet, Netware, email and the like evolved, so now people can send me emails inviting me to phone conferences at 01:00 UK time, invitations that will somehow be tentatively stuck in my shared calendar without my knowledge, then synchronised to my phone, so that it can wake me from my sleep to let me know that I am late for a phone conference that I didn't know about. Truly, we are blessed.

And yet the wheel goes round. What is fashionable again? The Datacentre. A room full of boxes turning electrons into waste heat the way thermodynamic entropy requires, routing many of the through the CPUs in the process, so doing useful things. The first big datacentres used many racks of machines, each box with 1-2 cpus, a couple of 512MB HDDs and 1 GbE between them. Now you could build up a single rack with the same storage capacity and compute power.

But you'd still have to share it. Which is why I'm pleased to show off a little addition to our facility: a very small Hadoop cluster.



These are four SL390s servers in 2 U's of rack; the two bits below are just expansion slots in the 4U chassis.

Each one the same basic node used in one of the top 10 supercomputers, though they have many more units Infiniband interconnect and a total power budget of 1.4MW, which is not something I'd want.

The front of the boxes contain all the interconnects; 2x10GbE on the motherboard, and a management ports hooked up to 100 MbE for ILO management;.. Having the ports at the front is something that Allen W has complained about. It does make sense if your switches work that way, if you can set your hold/cold aisles up so that the fronts are accessible. If your switch has its ports at the back, well, it's "suboptimal"

Round the back: power only. Shared PSUs and fans for a bit more resilience.



The twin-socketed Xeon E-series parts have a relatively low power budget for x86-64 servers, though not in ARM terms; the multiple SFF HDDs you can fit into each unit give pretty good bandwidth and 4TB of storage. If you opt for the 3.5" HDDs your bandwidth drops, but you get 6TB/node.

Then there's the RAM: up to 192 GB/node. These ones have a bit less than that in, but there's still more per core than the entire RAM supply in my house.

From a storage perspective, there's not that much capacity: 16-24 TB. The ratio of store:compute and drive:compute is pretty good though, and as you can also sneak in a GPU, if you have compute intensive work, these four machines make for a nice little setup. And given that those TB of storage don't need to be shared with anyone else, it's not so bad.

This then is capable of storing and working through a reasonable amount of data, building up complex in-memory structures and being as responsive mid-afternoon as it is on a weekend, as nobody else is trying to do stuff on it.

At this scale HDFS makes no sense. You don't have the capacity to handle a server failure; 3X replication is too expensive. Better to RAID everything and NFS cross mount the filesystems.

I know the big cluster people will look at these boxes with bemusement, but think about this

1. It's not the only cluster I have access to. This one is free for me to play with new versions and code on without causing problems.
2. I'm sure the mainframe people didn't think much of minicomputers, minicomputer aficionados looked down on desktop computers, and -as we can see- desktop computers are having do accept the growing functionality of phones and other devices.

This then, is my own personal datacentre.