x86 Parts and Server Farms

Intel have ~85% market share in x86 parts; AMD having the rest. That would seem good except for a quote from that report:

Total MPU shipments decreased 6% quarter-over-quarter in the fourth quarter, well below normal seasonality of down 1% quarter-over-quarter due to slowing PC demand and share loss to tablets. We note microprocessor shipments have now declined sequentially in four of the last five quarters.

On the desktop Intel managed to surf the success of Windows & the corresponding demand for x86 parts, so fund the R&D and the Fabs needed to stay ahead of more space efficient RISC CPU architectures. Intel could bring things out like the P5 superscalar CPU with the Pentium, then with the Pentium Pro introduce the first mass market speculative execution CPU architecture, the P6. For these parts and their successors, performance came before power.

The growing performance of those desktop parts not only killed the non-x86 Unix workstation business, they brought the x86 into the server world, pretty much taking a sharp knife to all the Unix on home-rolled CPU business -only IBM's Power and the SPARC lines remain, though how long SPARC will survive remains to be seen. (I'm ignoring Itanium, it being an Intel part too).

That's why when the Hadoop community talks about "commodity" servers, x86 parts are implicit in there. And with 85% market share, that's a lot of CPUs. Or, if you are in networking business, a lot of potentially 10GbE network ports -as nobody else is interested in them.

Intel are supplying the parts for those clusters then, which means they know the number that is selling, and they know something bad about it: most large clusters ship with only one socket filled up.

Problem: Hadoop is usually IO intensive; RAM is often a limiting factor on work. Fill in both sockets and you need to double the RAM too -but even then, may end up idling for data coming off disk or network.

Problem: On the spreadsheets used to cost out a big cluster -you have not only the rows needed to set the parts for each node (ram, storage, CPU), there's one for power. Fill that second socket and not only do you take a CAPEX hit, your OPEX looks worse as your power budget can jump up. And while x86 parts can crank back their wattage when idle, that says "when that second socket you filled is underused, your power bill isn't so bad", as in "If you still overspent on the CAPEX then your OPEX is slightly better".

On a big cluster, not filling that second socket can save enough money you could add another rack of servers -and the storage that comes with it. Storage is a tangible benefit.

Problem: GPUs are where the cycles are coming from, and in HPC clusters, it's worth compiling your code down to it -and working to make sure your code can compile down -and people are exploring GPUs in Hadoop clusters, especially as YARN delivers more cluster flexibility. Intel's attempts to get into the GPU business have failed -apart from the lower power display chipsets used in laptops when they are in battery mode, so GPUs aren't a revenue stream.

Problem on a big datacentre the buyer gets to negotiate prices. This is at its worst (for Intel, AMD, Dell, HP, SuperMicro, SGI) on the big cloud farms, as it will be a take-it-or-leave-it deal for thousands of nodes, and people like Amazon care about both capital cost of parts, and the operational cost of them. The people doing the negotiating know about bill of material curves (see below), and won't be rushing to by the latest part, not if last quarter's part costs significantly less for not much performance difference.

Problem: virtualisation may reduce demand for servers. The more you move to virtual server hosting, the less you need servers for every individual application. The adoption of virtualisation may have driven demand for new blade systems, but now that servers have consolidated, demand may level out -except for the new clusters, HPC racks and cloud service providers.

Problem: The desktop and laptop market are stagnating: the growing end-user products are smart phones and tablets, and with the exception of MS Surface Professional, Intel have barely a presence. Arm owns that market, and AMD have announced their plans to build server-side ARM parts and chipsets. AMD, with only 15% market share, have little to lose after all.

It's not just the CPU design that matters here, it's storage too. As I discussed in my Hadoop Summit EU talk proposal, Hadoop needs to plan ahead here. Tiered storage, many-core WIMPy parts, that's something Hadoop can adopt, especially if we start the groundwork.

And what's part of the nightly Jenkins build of Hadoop? Hadoop on ARM.

There we have it then: Hadoop clusters becoming a big part of the enterprise datacentre, but without demand for CPU parts compared to virtualisation blades and classic HPC systems. GPUs and parts moving from the desktop and ARM cores from the tablet -all have their eyes on the server market, offering massive CAPEX and OPEX savings. Oh, and the desktop market going away while the tablet and phone market remains elusive.

Yes, if you were the premier vendor of today's desktop & server CPUs, you'd be worrying about these -and thinking "even if the desktop business is going down, how can we increase demand for our parts in a growing Hadoop cluster business?"

That can be done in three ways: getting that second socket filled, making sure the CPUs sold have the latest-and greatest parts, and expanding demand for Hadoop clusters.

There is also the SSD business, which is money on the table as far as Intel are concerned: stuff to take from the HDD vendors. There may be competitors -such as Samsung- but if INTC can do mainboards with SSD hooked off the PCIex bus rather than pretending to be disk, there are opportunities that they can exploit.

Finally: 10GbE networking. This is something you need for the latest massive-storage-capacity servers anyway, unless your cluster has so many machines that the loss of single 36TB server isn't significant and the cost of 10GbE switches is. Lower cost of 10GbE switches would be nice.

If you look at the Intel Distribution for Apache Hadoop, then, you can see two features: a justification for multiple CPU sockets filled with the latest parts (encryption using new AES opcodes), and SSD speedup, though not the explicit tiering that the HDFS developers are starting to talk about.

What's not so obvious is why do their own bundling of this, which means they have to take on the entire QA process -which isn't something I've seen in terms of JIRAs filed- and it leaves CPU demand out of mainstream Hadoop. Unless Intel's Hadoop-related product obtains market share rapidly, I'm not sure why they didn't just try and get the CPU-intensive features in early.
[Photo: 3Dom's "The Cycle"; went up on Stokes Croft over the weekend"]

AWS: why the bias towards US-east?

As MastodonC will point out, Amazon's US-East sites are the most polluting, not just because they have a high CO2 footprint, but because the coal they (and the other east coast) industries burn is polluting in other ways, such as sulphur. It's not as bad as, say, a steelworks (having had relatives living near  Ravenscraig Steelworks I can vouch for this), but as datacentres can be placed near other electricity sources, it's needless.

I intermittently use US-West-2, up in Oregon, where the melting snow creates electricity.



Unfortunately there's an implicit bias in the AWS APIs towards US-East. Where's the default site for S3 Buckets? US-East. Where's the default site for EC2 instances? US-East. What is the default location for EMR jobs? The same -to the extent that the command line clients treats requesting a different site as "uncommon":

Uncommon Options
 --debug               Print stack traces when exceptions occur
 --endpoint ENDPOINT   EMR web service host to connect to
 --region REGION       The region to use for the endpoint
 --apps-path APPS_PATH Specify s3:// path to the base of the emr public bucket to use. e.g s3://us-east-1.elasticmapreduce

Because of all the implicit "us-east" bias, it becomes self reinforcing. Once you've got a bucket on S3 east, that's where you want to run your webapps otherwise you get billed for the remote bandwidth. Once you've got the webapps, that's where your logs go, hence even more reason to run your MR jobs on the same site: it's where your data lives.

Because it's the default location for stuff, it's also the default location for people serving up data on the site: RPM and Maven repositories, public datasets. This pushes you towards that location so as to avoid the costs of downloading that data from other sites, as well as the speed gain.

Why the bias? Either it's where the the majority of servers lie, or through a combination of cost of electricity, site PUE and bandwidth, it's got the lowest operating costs -hence the most profit per CPU-hour, MB stored or MB downloaded.

That's a shame, because amazon themselves have better options. They're being crucified by Parliament over their tax avoidance strategies -it'd be tactically wise to have something positive to talk about.

[Photo: Crater Lake & Mt Thielsen. Smoke is a forest fire blowing up from CA]

Hadoop in Cloud Infrastructures

People say "should you run Hadoop in the cloud?". I say "it depends".

I think there is value in Hadoop-in-cloud, I talked about doing it in 2010 at Berlin Buzzwords 2010; since then I've had more experience with using Hadoop and implementing cloud infrastructures.

1. If your data is stored in a cloud provider's storage infrastructure, doing the analysis locally is the only rational action. It's that "work near the data" philosophy.
2. If you are only doing some computation -say nightly- then you can rent some cluster time. Even if compute performance is worse, you can just rent some more machines to compensate.
3. You may be able to achieve better security through isolation of clusters (depends on your IaaS vendor's abilities).
4. No upfront capex; fund from ongoing revenue.
5. Easier to expand your cluster; no need to buy more racks, find more rack space.
6. You don't need to care about the problems of networking.
7. Less of a problem of heterogenous clusters if you expand later.

Against that

1. Cost of data storage can only increase at a rate proportional to ingress/retention rates.
2. Cost of cluster time increases at a rate proportional to analysis performed. There is no "spare cluster time" for low priority work.
3. Even if CPU time can scale up, IO rate of persistent data may not.
4. Hadoop contains lots of assumptions about running in a static infrastructure; it's scheduling and recovery algorithms assume this.

Some examples of where Hadoop's assumptions diverge from that of cloud infrastructures:

• HDFS assumes failures are independent, and places data accordingly (Google's Availability in Globally Distributed Storage Systems paper shows this doesn't hold in physical infrastructures, my notion of failure topologies expands on that)
• MR blacklists failing machines, rather than releasing them and requesting new ones.
• Worker nodes handle failure of master nodes by spinning on the hostname, not querying (dynamic) configuration data for new hostnames. Some of the HA HDFS may address that, I'm not tracking it enough.
• Topology scripts are static. I've been slowly tweaking topology logic in 0.23+ but haven't put the dynamicness in there yet (HDFS and MR cache (name->rack) mappings on the assumption that the data is coming from slow to exec scripts, not fast & refreshable in-VM data).
• Schedulers assume #of machines are static, don't allocate and release compute nodes based on demand and with knowledge of cost and quantum of CPU rentals. (I'm not sure quantum is the right term there, I mean the fact that VMs may be rented by the hour, 15 minutes, etc, so your scheduler should retain them for 59 minutes after acquiring them.
• Scheduling doesn't bill different users for their cluster use in a way that is easily mapped to cluster time.

A lot of these are tractable, you just have to put in the effort. The Stratosphere team in Berlin are doing lots of excellent work here, including taking a higher level query language and generating an execution plan that is IaaS aware -you can optimise for fast (many machines) or lower cost (use less machines more efficiently).

In comparison, a physical cluster:

• Offers a lower cost/TB of any corporate filestore to date other than people's desktop computers (which have a high TCO and power cost that is generally ignored), so enables you to store lots of stuff you would otherwise discard.
• Let's you choose the hardware optimised for your current and predicted workloads.
• Has free time for the low priority background work as well as the quicker queries that near-real-time UIs like.
• May be directly accessible from desktops in the organisation (depends on security model of cluster).
• Is easily hooked up to Jenkins infrastructure for execution of work as CI jobs.
• Let's you do fancy tricks like striping of different MR versions across the racks for in-rack locality and different sets of task trackers for foreground vs background work, and different JTs (reduces memory use, cost of failure, etc).
• Is way, way easier to hook up to internal databases, log feeds. To do ETL into your corporate oracle servers, you will need to run something behind the firewall to fetch it off the IaaS storage layer, rather than have your reducers push it to the RDBMS itself.

If you are generating data in house, in house clusters make a lot of sense.
This is why I say "it depends" -it depends on where you collect your data and what you plan to do with it.

As for the way Hadoop doesn't currently work so well in such infrastructures, well the code is there for people to fix. It's also a lot easier to test in-cloud behaviour, including resilience to failure, than it is with physical clusters.

[Photo: Descending Mt Ranier, 2000]

The datacentre is the new laptop

HP has just announced its forthcoming Gen 8 servers. Rather than go on about the usual stuff: CPUs, I/O bandwidth etc., or even the trend to put solid state storage off the PCIx bus, what's interesting to me is this: the servers are explicitly designed to be part of a larger system, a datacentre.

The existing products, they are individual servers you just happen to put into racks, you just happen to hook up to a switch you've stuck at the top of the same rack. The racks may be set up into hot rack/cold rack, but that's mostly a deployment detail the servers don't care about, except in ensuring airflow is good.

This has now changed.

The Datacenter as a computer argued that the software developers need to recognise that a datacentre is the new execution platform, one with mixed availability, limited bandwidth and other concerns that could be ignored before -or at least treated as the special case of "distributed systems", rather than what we have now: "systems". Everything is distributed.

These hardware changes mirror that. Here are some of the new concerns for both the ops team -and the applications themselves.

1. Re-integration of Storage and Computation.
   
2. Availability though replication. Less RAID-style hardware, more
   replication across machines.
3. Inventory tracking -especially for identifying failure points, such as monitoring the history of specific batches of disks. If some appear particularly unreliable, you want to find all of them.
4. Networking: 10 GbE is still a luxury, bonded 2x1 GbE good for availability too. Understanding network failures in data centers shows why ToR switches become the dominant network failure point in a cluster -and from a re-replication perspective, that's not ideal.
   
5. Power management. Beyond just PUE, the metric of datacentre overhead, power consumption in the servers is a big concern.

The new servers then, are designed to live in this world.

Inventory They work out from the rack (don't ask me how, I don't know these things) where they are on it -information that can be propagated to the management tools so that they can be used for inventory tracking.

Networking Lots of ethernet ports. Some slow and inexpensive for management, faster ones for the application.

Power. This work here is something you can point to Chrandrakant Patel in HP Labs for. If you look at his published work, you can see a lot of it is about airflow and cooling in a datacentre. If you can improve that -as the container hosted datacentre pods can do- then your PUE is better. Why instrument the inside of the servers? It ensures that you can keep the hardware within its limits, because you have a better idea of what is going on inside. Every extra degree F, C or K you can take the air up, lets you save a lot of money over time. Yet the risk of overheating -and the cost of doing so- makes this dangerous. Knowing what is happening inside the servers give you more confidence of what's happening.

This is what the new servers enable. Which means that we are going from servers that you stack to servers that are designed to locate themselves in the racks, ideally hosted within a datacentre container that is optimised for airflow and designed to work as close to the limits of temperature as is considered safe based on the information coming out of the servers themselves.

Which is very close to what a laptop does: a box with optimised airflow and fans that come on when they feel it is important, and with a power budget that the system is designed to optimise. The datacentre is the new laptop, at least from a power and cooling perspective.


Now, what about the software? If the datacentre-level application infrastructure can get at the power, topology and network information, it could adapt itself better.

The topology information that the servers can determine could be used to dynamically generate the topology map for the cluster. It is entirely co-incidental that I'm typing this while my new topology patches are being tested in a adjacent console, but those changes (better support for topology sources other than the script runner, ability to dump the current topology) are effectively a precursor. I wouldn't do some fancy integrated java module though -better to have a topology source that just reads a java properties file and by polling for changes, can react to moving topologies. Let the management tooling generate that and it would propagate into HDFS and the RM/MR layer.

Power? If overheating is a problem, that server can be clocked back, which makes it slower. It may be better to actually tell the resource manager that there are less slots on that box, so reducing its actual workload. This could ensure that the work running in the remaining slots doesn't take longer than normal to complete.

Networking? We really need a way to get more information about the network backplane into the application -including the amount of bandwidth currently allocated to applications. Bandwidth can be a precious resource, but right now there is better tooling to manage it in a bittorrent client than there is between applications in a datacentre.

This is a challenge and an opportunity. A challenge: this information needs to be extracted and forwarded to the applications -which then need to act on it. An opportunity -it will make the applications and datacentres work better. Wave goodbye to writing topology scripts that don't work, say hello to being able to move servers around and have them the application infrastructure work out where they are. Worry less about uncontrolled backbone bandwidth use in a shared datacentre; have some policy tooling to manage it across applications. As for power, hope to see the electricity bills decrease.


[Artwork: Sepr on Jamaica Street, Stokes Croft]

My other computer is a datacentre. A very small one.

Once upon a time every computer that filled a whole room, and many people had to look after it. The people who wanted to use it had to use punched cards; submit work and wait for an indeterminate period of time to see the results of that work.

Then minicomputers came, which were only the size of a cabinet in a room. These could be shared between less people, and with terminals, be more interactive. And from these machines came Unix, time_t, and buffer overflows in sprintf() statements.

It was still shared; you still had to compete with other people for finite resources. This is why when the microcomputer came along, and then the interactive workstation, something profound happened: you could do more. You could write interactive programs and be more productive without the long waits. That was before web browser updates, interactive downloading of emergency flash updates and AV Scanners got in the way of productive work, so desktop workstations were actually useful. Of course, all these machines were not that well connected on their own, so people would run round holding floppy disks, until networking products became available. Ethernet itself dates from the era of the workstation, though apparently it's frame length is partly driven by DEC's need to keep memory costs down, hence its pitiful size today. Ethernet, Netware, email and the like evolved, so now people can send me emails inviting me to phone conferences at 01:00 UK time, invitations that will somehow be tentatively stuck in my shared calendar without my knowledge, then synchronised to my phone, so that it can wake me from my sleep to let me know that I am late for a phone conference that I didn't know about. Truly, we are blessed.

And yet the wheel goes round. What is fashionable again? The Datacentre. A room full of boxes turning electrons into waste heat the way thermodynamic entropy requires, routing many of the through the CPUs in the process, so doing useful things. The first big datacentres used many racks of machines, each box with 1-2 cpus, a couple of 512MB HDDs and 1 GbE between them. Now you could build up a single rack with the same storage capacity and compute power.

But you'd still have to share it. Which is why I'm pleased to show off a little addition to our facility: a very small Hadoop cluster.



These are four SL390s servers in 2 U's of rack; the two bits below are just expansion slots in the 4U chassis.

Each one the same basic node used in one of the top 10 supercomputers, though they have many more units Infiniband interconnect and a total power budget of 1.4MW, which is not something I'd want.

The front of the boxes contain all the interconnects; 2x10GbE on the motherboard, and a management ports hooked up to 100 MbE for ILO management;.. Having the ports at the front is something that Allen W has complained about. It does make sense if your switches work that way, if you can set your hold/cold aisles up so that the fronts are accessible. If your switch has its ports at the back, well, it's "suboptimal"

Round the back: power only. Shared PSUs and fans for a bit more resilience.



The twin-socketed Xeon E-series parts have a relatively low power budget for x86-64 servers, though not in ARM terms; the multiple SFF HDDs you can fit into each unit give pretty good bandwidth and 4TB of storage. If you opt for the 3.5" HDDs your bandwidth drops, but you get 6TB/node.

Then there's the RAM: up to 192 GB/node. These ones have a bit less than that in, but there's still more per core than the entire RAM supply in my house.

From a storage perspective, there's not that much capacity: 16-24 TB. The ratio of store:compute and drive:compute is pretty good though, and as you can also sneak in a GPU, if you have compute intensive work, these four machines make for a nice little setup. And given that those TB of storage don't need to be shared with anyone else, it's not so bad.

This then is capable of storing and working through a reasonable amount of data, building up complex in-memory structures and being as responsive mid-afternoon as it is on a weekend, as nobody else is trying to do stuff on it.

At this scale HDFS makes no sense. You don't have the capacity to handle a server failure; 3X replication is too expensive. Better to RAID everything and NFS cross mount the filesystems.

I know the big cluster people will look at these boxes with bemusement, but think about this

1. It's not the only cluster I have access to. This one is free for me to play with new versions and code on without causing problems.
2. I'm sure the mainframe people didn't think much of minicomputers, minicomputer aficionados looked down on desktop computers, and -as we can see- desktop computers are having do accept the growing functionality of phones and other devices.

This then, is my own personal datacentre.

Towards a Topology of Failure

The Apache community is not just the mailing lists and the get togethers: it is the planet apache aggregate blog; this lets other committers share their thoughts -and look at yours. This makes for unexpected connections.

After I posted my comments on Availability in Globally Distributed Storage Systems, Phil Steitz posted wonderful article on the mathematics behind it. This impressed me, not least because of his ability to get TeX-grade equations into HTML. What he did do was look at the real theory behind it, and even attempted to implement Dynamic Programming solution the problem.

I'm not going to be that ambitious, but I will try and link this paper -and the other ones on server failures, into a new concept, "Failure Topology". This is an excessively pretentious phrase, but I like it -if it ever takes off I can claim I was the first person to use it, as I can do with "continuous deployment"

The key concept of Failure Topology is that failures of systems often follow topologies. Rack outages can be caused by rack-level upgrades. Switch-level outages can take out 1* racks and are are driven by the network topology. Power outages can be caused by the failure of power supplies to specific servers, sets of servers, specific racks or even quadrants of a datacentre. There's the also the notion of specific hardware instances, such as server batches with the same run of HDDs or CPU revisions.

Failure topologies, then, are maps of the computing infrastructure that show how these things are related, and where the risk lies. A power topology would be a map of the power input to the datacentre. We have the switch topology for Hadoop, but it is optimised for network efficiency, rather than looking at the risk of switch failure. A failure-aware topology would need to know which racks were protected by duplicate ToR switches and view them as less at risk then single-switch racks. Across multiple sites you'd need to look at the regional power grids, the different telcos. Then is the politics overlay: what government controls the datacentre sites; whether or not that government is part of the EU and hence has data protection rights, or whether there's some DMCA-style takedown rules.

You'd also need to look at physical issues: fault lines, whether the sites were downwind of Mt St Helen's class volcanoes. That goes from abstract topologies to physical maps.

What does all this mean? Well, in Disk-Locality in Datacenter Computing Considered Irrelevant, Ganesha Ananthanarayanan argues that as switch and backplane bandwidth increases you don't have to worry about where your code runs relative to the data. I concur: with 10GbE and emerging backplanes, network bandwidth means that switch-local vs switch-remote will become less important. Which means you can stop worrying about Hadoop topology scripts driving code execution policies. Doing this now opens a new possibility:

Writing a script to model the failure topology of the datacentre.

You want to move from a simple "/switch2/node21" map to one that includes power sources, switches, racks, shared-PSUs in servers, something like "/ups1/switch2/rack3/psu2/node21". This models not the network hierarchy, but the failure topology of the site. Admittedly, it assumes that switches and racks share the same UPS, but if the switch power source goes away, the rack is partitioned and effectively offline anyway -so you may as well do that.

I haven't played with this yet, but as I progress with my patch to allow Hadoop to focus on keeping all blocks on separate racks for availability, this failure topology notion could be the longer term topology that the ops team need to define.

[Photo: sunrise from the snowhole on the (dormant) volcano Mt Hood -not far from Google's Dalles datacentre]