s/macOS Sierra/r/macOS vista/

I've been using macOS sierra for about ten/eleven days now. and I've rebooted my laptop about 6+ times because the system was broken.

Two recurrent problems: failure to wake in the morning, gradual lockup of finder and transitive app failure.

Failure to wake: I go up to the laptop, hit the keyboard and mouse, nothing happens. Only way to fix: hold down the power and wait for a hard restart.

Back in 1999 I worked on a project with HP's laptop group, where we instrumented a set of colleague's laptops with a simple data collection app, then collected a few months of data. At the time this was considered "a lot of data". The result, the paper: The Secret Life of Notebooks. This showed that people tended to have a limited set of contexts, where context was defined as system setup: power, display, IP addr, and application: mail, ppt. And that people were so predictable in their use models, that doing some HMM to predict contexts wouldn't have been hard.

I ended up writing some little app which essentially did that: based on IPAddr and app (PPT, Acroread) full screen, could choose: power policy, network proxy options, sound settings (mute in meetings, etc). It was fairly popular amongst colleagues, because it would turn proxy stuff on and off for you, and know to turn off display timeouts when giving presentations; crank up the savings when on the move. When I look at Windows 8+ adaptation to network settings, or OSX's equivalent of that and the "When on battery ...", I see the same ideas. You don't get any HMM on the laptops though; for that you have to pick up an android phone and look at Google Now, something which really is trying to understand you and your habits. And, because it can read your emails, correlate those habits with emailed plans. If it really wanted to be innovative/scary it would work out who you were associated with (family, friends, colleagues, fellow students...) and use their actions to predict yours. Maybe someday.

User-wise, another interesting feature was how people viewed mail so differently when online vs offline. Offline, you'd see this workflow of outlook-> word-> outlook-> ppt-> outlook-> acroread-> outlook, ... etc, very fast cycles. It seemed uncontrolled window tabbing at first —until you realise it's peple going through attachments. Online, and people's workflow pulled in IE (it was a long time ago), and you'd get a cycle where IE was the most popular app cycled to from outlook. Email was already so full of links that the notion of reading email "offline" was dying. You could do it, but your workflow would be crippled. And that was 15+ years ago. Nowadays things would only be worse.

There was a second paper which was internal, plus circulated/presented to Microsoft. There I looked at system uptime, and especially the common sequence in the log

1998-08-23 18:15 event: hibernate
1998-08-24 09:00 event: boot

or

1998-09-01 11:20 event: suspend
1998-09-01 11:30 event: boot

That is: a lot of the time the laptop through it was going to sleep, it was really crashing.

My theory was that alongside the official ACPI sleep states S1-S5 there was actually a secret state S6, "the sleep you never awake from". Some more research showed that it was generally on startup that the process failed, and it was strongly correlated with external state changes: networks, power, monitors. It wasn't that the laptop made a mess of suspending, it was that when it came back up it couldn't cope with the changed state.

I don't know if macOS sierra has that issue: I do know that it has that problem if left attached to an external display overnight. Looking in the system logs, you can see powernap wakeups regularly (that's all displays off), but come the first user interaction event —where the displays are meant to kick off— they don't come up. This is resulting in system logs not far off from the '99 experiment

2016-09-27 22:20 powerd: suspend
2016-09-27 23:00 powerd: powernap wake
  ....

2016-09-27 23:30 powerd: powernap wake
..

2016-09-28 00:30 powerd: powernap wake
..
2016-09-28 08:30 powerd: powernap wake
..
1998-09-01 09:30 event: boot


That last one: that's me trying to use it.

I've turned off powernap to see if that makes a difference there.

That's the nightly problem. What's happened 3+ times is the lockup of Finder, with a gradual degradation of other applications as they go near its services.

First finder goes, and restarts do nothing

Then the other apps fail, usually when you go near the filesytem dialogs, or the photo collection.

As with finder, restart does nothing.

If it was my own code, I'd assume a lock is being acquired in the kernel on some filesystem resource and never being released. This is why locks should always have leases. Root cause of that lock/release problem? Who knows. I can't help wondering, though, if its related to all the new icloud sync feature, as that's the biggest filesystem change. I've also noticed that I usually have a USB stick plugged in; I'm going to go without that to see if it helps.

When i get this slow failure, i don't rush to reboot. It takes about 10 minutes to get my dev environment back up and running again: the IDEs, the terminal windows, etc, 2FA signing in to webapps, etc. I really don't want to have to do it. Instead I end up with bits of the UI keeling over, while I stick to the IDE, chrome, terminals. I had a bit of problem on Thursday evening when calendar locked up the extent I couldn't get the URLs for some conf calls; I had to use the phone to get the links and type them in.

Anyway, come the evening, after the conf calls and some S3a Scale tests, I kick off a shutdown.

And here a flaw of the OSX UI comes in: it assumes that whatever reason you are trying to do a shutdown for, it is not because finder has crashed. And it gives any application the right to veto the shutdown. You can't just select "shut down..." on the menu, you have to wait for any apps to block it, stop them and then continue. And even after doing all of that, I come in this morning and find the laptop, fans spinning away, me logged out but some dialog box about keychain access required. This is not shutting down, this is half hearted attempt at maybe shutting down sometimes if your OS hasn't got into a mess.

It's notable that Windows has some hard coded assumptions that a shutdown is caused by the failure of something. It also has, from the world of Windows Server, the concept that the user may not be waiting at the console waiting to click OK dialogs popped up by apps. Thus it has a harsher workflow.

1. A WM_QUERYENDSESSION message comes out saying "we'd like to shut down, is that OK? Apps get the opportunity to veto the sesson end, but not if it's tagged as a critical shutdown. And of you don't service that event, you are considered dead and don't get a veto.
2. The WM_ENDSESSION event sent to apps to say "you really are going down —get over it".
3. There is a registry entry WaitToKillAppTimeout you can use to control how long the OS waits for applications and to terminate, WaitToKillServiceTimeout for services, and even HungAppTimeout to control how long an app has to respond to an exit menu request (WM_EXIT?) before being considered dead and so killed.

See? Microsoft know that things hang, that even services can hang, and that if you want to shut down then you want to shut down, not find out 12 hours later that it had stopped with a dialog box.

In contrast macOS Sierra has implicit assumptions that apps rarely hang, the OS services never deadlock, and that shutting down is a rare activity where you are happy to wait for all the applications to interact with you —even the ones that have stopped responding.

This may have held for for OS/X, but for macOS all those assumptions are invalid. And that makes shutdown far more painful and unreliable than it need be.

Now if you go low level and do a "man shutdown", you can see that a similar escalation process is built in there

Upon shutdown, all running processes are sent a SIGTERM followed by a SIGKILL.  The SIGKILL will follow the SIGTERM by an intentionally indeterminate period of time.  Programs are expected to take only enough time to flush all dirty data and exit.  Developers are encouraged to file a bug with the OS vendor, should they encounter an issue with this functionality.

I think from now on, it'll be a shutdown command from the console.

Anyway, because of all these problems, I do currently regret installing macOS sierra. It shipped to meet a deadline, rather than because it was ready.

macOS Sierra is not ready for use unless you are prepared to reboot every day, and are aware that the only way to reboot reliably is from the console.

Enterprise Hadoop: yes, but how are you going to fix it?

EMC's Pivotal HD has started a lot of debate as to whether building on top of Hadoop can be considered being part of a Hadoop ecosystem or whether it's an attempt to co-opt it: to do something and claim that it is part of a bigger system.

Can you say you are "part of the Hadoop stack" when all you are doing a closed source layer on top? I think that's quite nuanced, and depends on what you do -and how it's interpreted.

1. The Apache License grants everyone the freedom to take the source away and do anything they want with it
2. There is no requirement for you to contribute a single line of code back; a single bug report. 

This is a difference between the ASF license and GPL-licensed software which you redistribute: with GPL code the changes must (somehow) be published. 

Other aspects of the ASF license:

1. You can't abuse ASF brand names, which in the Apache Hadoop world means you can't use Apache Hadoop, Apache HBase, Apache Mahout, Giraph, Apache Pig, Apache Hive, etc in your product names. There are some excellent guidelines on this in the wiki page Defining Hadoop -and if you want actual feedback, email the trademarks@ list. It may seem that doing so removes the secrecy/surprise factor of your product announcement, but it's better that than a hurried renaming of all your product and documentation.
2. If you sue other users of the product over patents of yours that you believe apply to the technology -you revoke your own right to the software. I haven't known that to happen with Apache products -though the Oracle/Google lawsuit did cover copyright of APIs and reimplementations thereof. If APIs ever became copyrightable, then decades of progress in the computing industry will grind to halt.

People are also free to look at Apache APIs and clean-room re-implement them; you just can't use the Apache product names at that point. Asserting compatibility becomes indefensible: if you look at the ASF JIRAs, even 100% compatibility across versions is hard to achieve -that's with the same source tree. It's not the binary signature that is (usually) the problem, its what happens afterwards that's trouble. Little things like whether renaming a file is atomic, or what happens when you ask for the block locations of a directory.

Now, what about introducing a closed source product on top of Hadoop and saying you are part of the hadoop ecosystem, that you have x-hundred people working on Hadoop?

This is where it gets tricky.

Some people say "it's like building on Linux" -and there are some very big closed applications that run on Linux. A big one that springs to mind is Oracle RDBMs.

Are the thousands of people who work on Oracle-on-Linux "working on Linux"? Are they working on "Oracle on Linux", or are they working "on Oracle", on Linux?

Whatever way you look at it, those people aren't working in the Linux OS, just on something that runs on top of it . Would you call it part of the Linux "stack", the way MySQL and Apache HTTPD are?

Personally: I have no idea.




What probably doesn't happen from Oracle's work is any direct feedback from their application into the OS. [Correction: it does, thx @tlipcon]. I also doubt that RedHat, Novell and others regression test Oracle RDBMS on their latest builds of Linux. By their very nature, closed-source applications fall out of the normal OSS regression and release test processes, that rely not only on the open source trees, but the open test suites. This is also why Oracle's actions in not releasing all tests for MySQL seems so short sighted: it may hurt MariaDB, but it also hinders Linux regression testing.


Breaking that link between the OS and the application means that Oracle have not been in the position to rapidly adapt to problems in the OS and filesystem, because there's no way to push their issues back upstream, to get changes in, to get new releases out in a hurry to fix a problem with their application or hardware. Instead the onus becomes on the application to deal with the problem themselves.

How have Oracle handled this? Eventually, by getting into the Linux Distribution business itself, with Oracle Unbreakable Linux. By releasing a complete OS build, they can coordinate OS and application releases, they can fix their version of the OS to handle problems that surface in Oracle's applications -on a timetable that works for them. They also get to handle Oracle hardware support in a timely manner, and charge support revenue from users.
 
That works -at a cost. By forking RedHat Linux, Oracle have taken on all the maintenance and testing costs themselves.

The amount that Oracles charge has to cover those costs, or the quality of the Oracle fork of Linux degrades relative to the reference points of RHEL and Debian.

For Oracle, or the combined OS+11g+exadata deal has enough margins in the database that they can come up with a price is was less than ({HP | Dell}-RHEL-Oracle11g), and so presumably those costs can be covered. What's not clear is this: did Oracle get into the business of selling a supported Linux because they saw money in it, or because they concluded that their hardware and database products effectively mandated it?


Other companies getting into the business of redistributing Hadoop-derived products to customers who are paying those companies in the expectation of support are going to have start thinking about this.

If you have just sold something that has some Hadoop JARs in it -code that the customer depends on- and they have a problem, how are you going to fix it?

Here are some strategies:

1. Hope it won't be a problem. Take the Apache artifacts, ship as is. It is, in the opinions of myself and my Hortonworks colleagues, production ready. Push customers with problems to issues.apache.org, forward them yourself. You could do the same with CDH, which in the opinions of my friends at Cloudera, also production ready.Do that, and issues on Apache JIRA will be ignored unless you can replicate them on the ASF artefacts.
2. Build your own expertise: this takes time, and while that happens you aren't in a position to field support calls. If you make your own releases, you end up needing your own test infrastructure, QA'ing it, and tracking the changes in hadoop trunk and branch-1.
3. Partner with the experts: work with people who have in-depth understanding of the code, it's history, why decisions were made and experience in cutting production scale releases suitable for use in web companies and enterprises. That means Hortonworks and Cloudera. Many of the enterprise vendors do this, because they've realised it was the best option.

The web companies, the early adopters went for #1 and ended up with #2: build your own expertise. This is effectively what I did in my HPLabs work on dynamic in-cloud Hadoop. You can see my journeys through the source -while working on big things, little things crop up, especially problems related to networking in a virtual world, configuration in a dynamically configured space, and recovery/sync problems that my service model discovered. I still know my way through a fraction of the code, but every project I work on builds up my understanding, and contributes stuff back to the core, including things like better specifications of the filesystem API's semantics, and the tests to go with it. 

That trail of JIRAs related to my work shows up something else: if you are delving deep into Hadoop, your reading of the code alone should be enough to get you filing bugs against minor issues, niggles, potential synchronization, cleanup or robustness problems. If you are pushing the envelope in what Hadoop can do: bigger issues.

We are starting to see some involvement in hadoop-core from Intel, though apart from the encryption contribs, it still appears to be at an initial state -though Andrew Purtell has long been busy in HBase. We do see a lot activity from Junping Du of VMWare -not just the topology work, but other big virtualisation features, and the day-to-day niggles and test problems you get working with trunk. Conclusion: at least one person in VMWare is full time on Hadoop. Which is great: the more bugs that get reported, the more patches, the better Hadoop becomes. Participating in the core code development project develops your expertise while ensuring that the Apache (hence Hortonworks and Cloudera) artifacts meet your needs.

Are there other contributors from EMC? Intel? I have no idea. You can't tell from gmail & ymail addresses alone; you'd have to deanonymize them by going via LinkedIn. That's not just name match; you can use the LI "find your contacts" scanner to go through those people's email addresses and reverse lookup their names. Same for twitter. I may just do that for a nice little article on "practical deanonymization".

In the meantime, whenever someone comes to you with a product containing the Apache Hadoop stack, say "if there is a problem in the Hadoop JARs - how are you going to fix it?"



[Artwork: See no evil by Inkie, co-organiser of the See No Evil event. Clearly painted with the aid of a cherry picker]

Followup: where can we do interesting things at low-levels

I managed to offend Andrew Purtell of Intel in my last post, so to make up here's some issues why Hadoop benefits from his work




Hardware is changing, CPU arch, memory layouts, cross-core synchronization, instructions built in the CPU. For efficient code we need everything from minimal branch misprediction to things that cut down core operations -especially decompression, checksums, encryption.

That's why the amount of native code in Hadoop is only going to increase -and that's going to take more than just skill to write -it takes the work to identify the problems and bottlenecks in the first place.


Java gets in the way here. You don't know the layout of a java class in memory, so it's near impossible to design cache-efficient designs, pulling in a class on a single cache line, avoiding false sharing within a struct. Even java's volatile opcode forces in a synchronisation barrier -making it much more expensive than C/C++ volatile. It's memory manager didn't go NUMA-aware until Java7 -it's been lagging hardware, but openJDK may speed things up there. Except now there's the problem of testing Hadoop on trunk builds of openJDK.

Looking forward, the end of HDDs is inevitable, but before then, mixed SSD and HDD will be the trend -and HDFS has to use that, somehow deciding where to put data, and knowing where it is.

If you are scheduling an MR job, and you have fast 10GbE networking, then the job placement problem becomes less of "where is there a slot on any of the three server with the data" and more "which of my 3 replicas are in SSD" -then doing work on or near that server, as things will complete that much faster.

And to have some storage of replicas across tiers will need HDFS knowing about the options, and making decisions. Those decisions will be hard, and you'll need something like a rebalancer in each host, moving cold data out of the SSD, so leaving space for warm data. Unless the SSD is actually used for write-through data, in which case the server you wrote two now has two copies, one on SSD, one on HDD. That may be a viable layout for two of the replicae. With P(SSD-fail) << P(HDD fail), you may even consider having more two-copy blocks, though there you would want them on separate boxes.

Other fun areas are OS integration, the underlying filesystem and networking stack. If you look at how today's servers & OSes are tuned, its towards things like databases. In a world where it's many-server designs, parallel operations across a cluster, things could be different

So yes: lots of areas to play in here, all welcome. Especially with tests.

[Photo: Jamaica Street. Centre looks like it is by Paris]

x86 Parts and Server Farms

Intel have ~85% market share in x86 parts; AMD having the rest. That would seem good except for a quote from that report:

Total MPU shipments decreased 6% quarter-over-quarter in the fourth quarter, well below normal seasonality of down 1% quarter-over-quarter due to slowing PC demand and share loss to tablets. We note microprocessor shipments have now declined sequentially in four of the last five quarters.

On the desktop Intel managed to surf the success of Windows & the corresponding demand for x86 parts, so fund the R&D and the Fabs needed to stay ahead of more space efficient RISC CPU architectures. Intel could bring things out like the P5 superscalar CPU with the Pentium, then with the Pentium Pro introduce the first mass market speculative execution CPU architecture, the P6. For these parts and their successors, performance came before power.

The growing performance of those desktop parts not only killed the non-x86 Unix workstation business, they brought the x86 into the server world, pretty much taking a sharp knife to all the Unix on home-rolled CPU business -only IBM's Power and the SPARC lines remain, though how long SPARC will survive remains to be seen. (I'm ignoring Itanium, it being an Intel part too).

That's why when the Hadoop community talks about "commodity" servers, x86 parts are implicit in there. And with 85% market share, that's a lot of CPUs. Or, if you are in networking business, a lot of potentially 10GbE network ports -as nobody else is interested in them.

Intel are supplying the parts for those clusters then, which means they know the number that is selling, and they know something bad about it: most large clusters ship with only one socket filled up.

Problem: Hadoop is usually IO intensive; RAM is often a limiting factor on work. Fill in both sockets and you need to double the RAM too -but even then, may end up idling for data coming off disk or network.

Problem: On the spreadsheets used to cost out a big cluster -you have not only the rows needed to set the parts for each node (ram, storage, CPU), there's one for power. Fill that second socket and not only do you take a CAPEX hit, your OPEX looks worse as your power budget can jump up. And while x86 parts can crank back their wattage when idle, that says "when that second socket you filled is underused, your power bill isn't so bad", as in "If you still overspent on the CAPEX then your OPEX is slightly better".

On a big cluster, not filling that second socket can save enough money you could add another rack of servers -and the storage that comes with it. Storage is a tangible benefit.

Problem: GPUs are where the cycles are coming from, and in HPC clusters, it's worth compiling your code down to it -and working to make sure your code can compile down -and people are exploring GPUs in Hadoop clusters, especially as YARN delivers more cluster flexibility. Intel's attempts to get into the GPU business have failed -apart from the lower power display chipsets used in laptops when they are in battery mode, so GPUs aren't a revenue stream.

Problem on a big datacentre the buyer gets to negotiate prices. This is at its worst (for Intel, AMD, Dell, HP, SuperMicro, SGI) on the big cloud farms, as it will be a take-it-or-leave-it deal for thousands of nodes, and people like Amazon care about both capital cost of parts, and the operational cost of them. The people doing the negotiating know about bill of material curves (see below), and won't be rushing to by the latest part, not if last quarter's part costs significantly less for not much performance difference.

Problem: virtualisation may reduce demand for servers. The more you move to virtual server hosting, the less you need servers for every individual application. The adoption of virtualisation may have driven demand for new blade systems, but now that servers have consolidated, demand may level out -except for the new clusters, HPC racks and cloud service providers.

Problem: The desktop and laptop market are stagnating: the growing end-user products are smart phones and tablets, and with the exception of MS Surface Professional, Intel have barely a presence. Arm owns that market, and AMD have announced their plans to build server-side ARM parts and chipsets. AMD, with only 15% market share, have little to lose after all.

It's not just the CPU design that matters here, it's storage too. As I discussed in my Hadoop Summit EU talk proposal, Hadoop needs to plan ahead here. Tiered storage, many-core WIMPy parts, that's something Hadoop can adopt, especially if we start the groundwork.

And what's part of the nightly Jenkins build of Hadoop? Hadoop on ARM.

There we have it then: Hadoop clusters becoming a big part of the enterprise datacentre, but without demand for CPU parts compared to virtualisation blades and classic HPC systems. GPUs and parts moving from the desktop and ARM cores from the tablet -all have their eyes on the server market, offering massive CAPEX and OPEX savings. Oh, and the desktop market going away while the tablet and phone market remains elusive.

Yes, if you were the premier vendor of today's desktop & server CPUs, you'd be worrying about these -and thinking "even if the desktop business is going down, how can we increase demand for our parts in a growing Hadoop cluster business?"

That can be done in three ways: getting that second socket filled, making sure the CPUs sold have the latest-and greatest parts, and expanding demand for Hadoop clusters.

There is also the SSD business, which is money on the table as far as Intel are concerned: stuff to take from the HDD vendors. There may be competitors -such as Samsung- but if INTC can do mainboards with SSD hooked off the PCIex bus rather than pretending to be disk, there are opportunities that they can exploit.

Finally: 10GbE networking. This is something you need for the latest massive-storage-capacity servers anyway, unless your cluster has so many machines that the loss of single 36TB server isn't significant and the cost of 10GbE switches is. Lower cost of 10GbE switches would be nice.

If you look at the Intel Distribution for Apache Hadoop, then, you can see two features: a justification for multiple CPU sockets filled with the latest parts (encryption using new AES opcodes), and SSD speedup, though not the explicit tiering that the HDFS developers are starting to talk about.

What's not so obvious is why do their own bundling of this, which means they have to take on the entire QA process -which isn't something I've seen in terms of JIRAs filed- and it leaves CPU demand out of mainstream Hadoop. Unless Intel's Hadoop-related product obtains market share rapidly, I'm not sure why they didn't just try and get the CPU-intensive features in early.
[Photo: 3Dom's "The Cycle"; went up on Stokes Croft over the weekend"]

why you should vote for "Hadoop: Embracing Future Hardware"

At some point in the next 10-15 years, the last "rotating iron" hard disk will be made.

That's a profound thought. Admittedly, I may get the date wrong, but the point remains. Just as the CRT, the floppy drive and the CD has gone away, hard disks will become a rarity.

Who cares? Those of us building the future Hadoop platforms do.





GFS& MapReduce, Hadoop HDFS and its MR Engine, are all designed to take advantage of "commodity hardware". That means rather than pay for top of the line Itanium, PowerPC or Sparc servers running a Sysv-derived Unix, they use servers built from x86 parts running Linux. This is not because of any ideological support of the x86 architecture: nobody who has ever written x86 assembler or debugged win32 C++ apps at that level will be fond of the x86. No, x86 parts were chosen as they were the servers with the most cost effective performance, a manageable power budget (compared to Itanium) and because people made servers with them on board.

And why are x86 parts so cost effective -even though they have so many millions of transistors Because Intel have managed to take the revenue from each generation of parts into funding the R&D work and new fabs needed for the next generation of CPU parts and the processes to manufacture them.

It is the mass consumer and corporate demand for PC desktops that has given us affordable high-performance x86 parts,

Even if the Xeon stuff doesn't work in the desktop, the fabs and the core design are shared -the volumes kept the cost down.

With the emergence of phones and tablets as the new consumer internet access point, sales of PC parts are flatlining, and may decrease in future. Our home PC is used as a store for photographs and  a device for a ten year old to play minecraft or -or to watch youtube videos of minecraft. He isn't committed to intel parts, and as for the photgraphs, well, 1TB of cloud storage isn't affordable -yet- but that may change. And when your phone can upload directly to facebook, why faff around downloading things to a local PC?

Even enterprise PCs are changing, they are called "laptops" and SSD storage is moving down from the "ultrabook" class of devices to becoming mainstream -at a guess within 3-5 years they'll be SSD everywhere.

The world of end user devices are changing -which is going to have implications for servers. We need to look at those trends and start planning ahead, not just to handle the "what happens when HDDs go away" problem, but "how can we make best use of these new parts in 18-24 months?

Which brings me round to the whole point of this article: my other talk is Hadoop: Embracing Future Hardware,

Vote for it. If not, you'll be taken by surprise when the future happens around you while you weren't looking.

[Photo: something from the harbourfest , 2008l]

Visiting the France Hadoop Users Group

At the invitation of Cedric Carbone from Talend, I went over to Paris to join in the third France HUG event, giving my talk and listening to the others.

It was really good to meet a group of people all of whom are involved in this stuff. These were Hadoop users, not "curious about all the hype" users -and I enjoyed not just talking about Hortonworks, Hadoop and HDP-1, but listening to what they are up to and the issues they have. I also got some lessons in technical french -all the Hadoop product names are designed for the language.

Now my HDP-1 slides:

HDP-1 introduction for HUG France

I gave a quick demo at the end of the stuff I'm doing on availability -not the bit where vSphere recognises and kills a VM hosting a failed service, restarting it on the same physical host or, if that host itself is in trouble, elsewhere. Instead I showed how the JT can be set up to not only reload its queue of jobs, but go into "safe mode" either at the request of a remote administrator, but also when it detects that the filesystem is offline.

Being able to put the JT into safe mode is beneficial not just for dealing with unplanned availability issues, but planned DFS maintenance. When you flip the switch on the JT it doesn't kill tasks in flight, but it doesn't worry if they fail; it doesn't blacklist tasktrackers or consider the job as failing. You can't schedule new jobs in safe mode either -while that's implicit in that HDFS isn't there to save your JARs, this just makes it more formal. When the FS comes out of safe mode, it reschedules the queued work (the whole job), and new requests can be added.

The UI can show safe mode too -though I've realised that before the code is frozen, the JT setSafeMode() call should take a string explaining why the system has entered this state. It would be set automatically on DFS failure, while a manual request would pick up whatever explanation you asked for, e.g. "Emergency Switch maintenance". Anyone who goes to the JT status page would see this message.

DFS could have the same -in fact maybe the JT ought just to support a message of the day feature. That's feature creep: knowing why things are down is better. Indeed, I could imagine it being part of the payload of exceptions clients get: "W. says no jobs today".

The other talks gave me a chance to revise some French, with HCatalog and an attendee of the Hadoop Summit giving their summary.

Afterwards: baguette, fromage, Kronenbourg 1664 -stuff worth travelling over for.

One interesting discussion I had was on the topic of ECC DRAM in servers. Having done all the reading on availability last year, I not only think that ECC is essential in servers, the time when it should be in desktop systems is drawing close. When machines are shipping with 4+GB of DRAM in even a laptop, P(single-bit-error) is getting high, as Nightingale's "Cells cycles and platters paper showed.

Yet as the attendees pointed out, today you are being given a choice of five ECC'd servers vs 15 non-ECC, and the tangible benefits of more servers is higher than the probabilistic benefits of ECC'd servers.

Why is ECC so expensive? It's not the RAM, which is adds lg(data bits + ecc bits) worth of DRAM: 6 bits for a 32 bit word, 7 bits for a 64 bit line. The percentage of extra ECC bits over DRAM decreases per line width, so as we move to wider memory buses, the incremental $ and W cost of ECC should decrease. Why then the premium? Chipsets and motherboards. ECC is viewed as a premium luxury that isn't even an option in the consumer chipsets, Atom parts in particular. That's just an attempt to gain a price premium on server designs -not in the chipset and mainboard, but on the CPUs itself.

This led into a discussion on what would the ideal Hadoop worker node be. Not an enterprise-priced hot-swap-everything box -that makes sense for the master nodes, but not for the many workers. There are the 'web scale' server boxes that the PC manufacturers will sell you if you have a large enough order. These are the many-disk, nearly no end user maintenance design that you can buy in bulk -but it has to be bulk. They can contain assumptions that they are the sole inhabitants of a rack (things like front-panel networking needing a compatible ToR switch), and they are for organisations whose clusters are so big that disk failures are treated as an ongoing operations task, which can be addressed by decommissioning that node and then replacing the disks at leisure. At scale this is the right tactic -as the decommissioning load is spread across the entire cluster's network bandwidth, it's not an expensive operation LAN-wise.

In smaller clusters you not only lack the spare disk capacity to handle a 24 TB server outage, the impact on your bandwidth is higher. Provided you can power off the server, swap over the disk, have the machine booted, the disk formatted and mounted and Datanode live within 15 minutes, you do a warm swap today. If servers were made for that swap in/out to be easier than with the current web-scale servers, this could/should be possible.

What do we want then?

• Easy to warm swap disks in and out
• Not mandate single model /rack -so network ports, PSUs and racks can't require this.
• ECC memory
• Not require the latest and best CPUs and chipsets. A bit behind the model curve makes a big difference in ASP.
• Low peak power budget. I know the I7 chipsets can power off idle cores; even raise the voltage/clock speed of the active cores in such a situation to finish their work harder in the same power envelope. But in a rack you need to consider the whole rack power budget, and how to stay not only within the rack's power budget, but how to stay in the billing range of the colocation site -some of whom bill by peak possible power budget, not actual use.
• Topology information. Some of the HP racks now do this for the admin tools -be nice to see how to get that into Hadoop.
• Support 2x1 Gb LAN, LAN with throughput that meets the peak loads of re-replicating an entire lost rack.
• Option for adding lot more of RAM, without you having to pull out and throw away the RAM it initially came with.
• Blinking lights on the front that you can control in the user-level software. You know you need it. This may seem facetious but it's how you direct people in the datacentre to the box they need to look at, it's how you qualify the network topology (light up all boxes in row of all racks -boxes that don't light are offline or mis-connected). Plus Thinking Machines shows that Blinky Lights look good at scale.
• Option to buy compatible CPUs to fill in the spare socket in twelve months time. That means the parts should still be on the price list.
• Option to buy compatible servers 12 months down the line, where compatible means "same OS", even if storage and CPU may have incremented.
• CPU parts with excellent Hadoop/Java performance, good native compression and checksumming.
• Linux support for everything on the motherboard.
• Options for: GPU, SSD
• aybe: USB BIOS updates. On a 50 node rack this is just about manageable and means you can skimp on the ILO management board and matching network.
• Ability to get system state (temp, PSU happiness) into OS and then into the management tools of the cluster -even the Hadoop cluster.

No need for:

• RAID controllers.
• Hot swap PSUs, though ability to share peer PSUs is handy. A cold-swap PSU and an on-site spare should suffice, given the rate of PSU failures ought to be much, much less than those of disks.
• Top of the line CPU parts whose cost ramps up much more than performance verses the previous model.
• Dedicated management LAN and ILO cards. Nice and makes managing large clusters much easier, but they add cost that small clusters can't justify.
• Over-expensive interconnect (did anyone say Infiniband?)

I don't know who is doing this yet -if you look at the prepackaged Hadoop stacks they are all existing Hardware. Things will no doubt change -and once that happens, once people starting optimising hardware for Hadoop, we will all get excellent value for money. And Blinking Lights on the front of our boxes that MR jobs can control.