3 years at Hortonworks!

In 2012 I handed in my notice at HP Laboratories and joined Hortonworks : this May is the third anniversary of my joining the team.



I didn't have to leave HP, and in the corporate labs I has reasonable freedom to work on things I found interesting. Yet is was through those interesting things that we'd discovered Hadoop. Paolo Castagna introduced me to it, as he bubbled with enthusiasm for what he felt was he future of server side computing. At the time I was working on the problem of deploying and managing smaller systems -but doing so in the emergent cloud infrastructures. Hadoop was initially another interesting deployment problem: one designed to scale and cope with failures, yet also built on the assumption of a set of physical hosts, hosts with fixed names and addresses, hosts with persistent storage and whose  failures would be independent. some of the work I did at that time with Julio Guijarro included dynamic Hadoop clusters, Mombasa (the long haul route to see elephants). The work behind the scenes to give Hadoop services more dynamic deployments, HADOOP-3628, earned me Hadoop committership. While the branch was never merged in, the YARN service model shows its heritage.

While we were doing this, HP customers were also discovering Hadoop —and building their clusters. I remember the first email coming in from a sales team who had been asked to quote the terasort performance of their servers: the sales team hadn't heard of a terasort and didn't know what to do. We helped. Before long we were the back-end team on many of the big Hadoop deals, helping define and review the proposed hardware specs, reviewing and sometimes co-authoring the bid responses. And what bids they were! At first I thought a petabyte was a lot of storage —but soon some of the deals were for 10+, 20+ PB. Projects where issues like rack weight and HDD resonance were as key to worry about as power and logistics of getting the servers delivered. Production lines which needed to be block booked for a week or two, but in doing so allowing server customisation: USB ports surplus? Skip them. How many CPU sockets to fill-and with what SKU? Want 2.5" laptop HDDs for bandwidth over 3.5" capacity oriented storage? All arrangeable, with even the option of a week-long burn in and benchmark session as an optional extra. This would show that the system worked as requested, including setting benchmarks for sorting 5+ PB of data that would never be published out of fear of scaring people (bear that in mind when you read blogs posts showing how technology X out-terasorts Hadoop —the really big Hadoop sort numbers are of 10+ PB on real clusters, not EC2 XXL SSD instances, and they don't get published).

These were big projects and it was really fun to be involved.

At the same time though, I felt that HP was missing the opportunity, the big picture. The server group was happy to sell the systems for x86 system margins, other groups to set them up. But where was the storage group? Giving us HPL folk grief for denying them the multi-PB storage deals —even though they lacked a Hadoop story and didn't seem to appreciate the technology. Networking? Doing great stuff for HFT systems where buffering was anathema; delivering systems for the enterprise capable of handling intermittent VM migration. But not systems optimised for sustained full link rate bandwidth, decent buffering and backbone scalability through technologies like TRILL or Shortest Path Bridging (you can get these now, BTW).

The whole Big Data revolution was taking place in front of HP: OSS software enabling massive scale storage and compute systems, the underlying commodity hardware making PB storeable, and the explosion in data sources giving the data to work with. And while HP was building a significant portion of the clusters, it hadn't recognised that this was a revolution. It was reminiscent of the mid 1990s, when the idea of Web Servers was seen as "just another use of a unix workstation".

I left to be part of that Big Data revolution, joining the team I'd got to know through the OSS development, Hortonworks, and so defining the future, rather than despair about HP's apparent failure to recognise change. Many of us from that era left: Audrey and I to Hortonworks, Steve and Scott to RedHat, Castagna to Cloudera. Before I get complaints from Julio and Chris, —yes some of the first generation of Hadoop experts are still there, the company is taking Big Data seriously, and there are now many skilled people working on it. Just not me.

What have I done in those three years? Lots of things! Some of the big ones include:

• Hadoop 1 High Availability. One of the people I worked with at VMWare, Jun Ping, is now a valued colleague of mine.
• OpenStack support: Much of the hadoop-openstack code is mine, particularly the tests.
• The Hadoop FS Specification: defining a Python-like syntax for Spivey's Z notation, delving through the HDFS and Hadoop source to really define what  a Hadoop filesystem is expected to do. From the OpenStack Swift work I'd discovered the unwritten assumptions & set out to define them, then build a test suite to help anyone trying to integrate their FS or object store with Hadoop to get started. This was my little Friday afternoon project; nobody asked me to do it -but now that it is there it's proven invaluable in getting the s3a S3 client working, as well as being one of the first checkpoints for anyone who wants to get Hadoop to work on other filesystems. Arguably that helps filesystem competitors —yet what it is really meant to do is give users a stable underpinning of the filesystem, beyond just the method signatures.
• The YARN-117 service model. I didn't start that work, I just did my best to get the experience of the SmartFrog and HADOOP-3628 service models in there. I do still need to document it better, and get the workflow and service launcher into the core code base; Slider is built around them.
• Hoya: proof of concept YARN application to show that HBase was deployable  as a dynamic YARN application, and initial driver for the YARN-896 services-on-YARN work.
• Apache Slider (incubating). A production quality successor to Hoya, combining the lessons from it with the Ambari agent experience, producing an engine to make many applications deplorable on YARN though a minimal amount of Python code. Slider is integrate with Ambari, but it works standalone against ASF Hadoop 2.6 and the latest CDH 5.4 release (apparently). I've really got a good insight into the problems of placement of work where access to data has to be balanced with failure resilience; enough to write a paper if I felt like it —rather than just a blog post.
• The YARN Service Registry. Again, something I need to explain more. An HA registry service for Hadoop clusters, where static and dynamic applications can be registered and used. Slider depends on it for client applications to find Slider and its deployed services; it is critical for internal bonding in the presence of failures. It's also the first bit of core Hadoop with a formal specification in TLA+.
• Spark on YARN enhancements. SPARK-1537 is my first bit of work there, having the spark history server use the YARN timeline service. Spark internals in Scala, collaboration with the YARN team on REST API definitions and reapplying the test experience of Slider to accompany this with quality tests.
• Recently: some spare time work mentoring S3a: into a production ready state.
• Working with colleagues to help shape our vision of the future of Hadoop. Apache Hadoop is a global OSS project, one which colleagues, competitors and users of the technology collaborate to build. I, like the rest of my colleagues get a say there, helping define where we think it can go: then building it.

The latter is a key one to call out. At HP an inordinate amount of my time was spent trying to argue the case for things like Hadoop inside the company itself, mostly by way of PowerPoint-over-email. I don't have to do that any more. When we make decisions it's s done rapidly,  pulling in the relevant people, rather than the inertial hierarchy of indifference and refusal which I sometimes felt I'd encountered in HP.

Which is why working at Hortonworks is so great: I'm working with great people, on grand projects —yet doing this a process where my code is in people's hands within weeks to months, and where an agile team keeps the company nimble. and pretty much all my work has shipped.

If you look at how the work has included applied formal methods, distributed testing, models of system failure and dynamic service deployment, I'm combining production software development with higher level work that is no different than what I was doing in a corporate R&D lab -except with shipping code.

Hortonworks is hiring. If what I've been up to -and how I've been doing it- sounds exciting: then get in touch. That particularly applies to my former HPL colleagues, who have to make their mind up where to go: ink vs enterprise. There is another option: us.

Hadoop 2: shipping!

The week-long vote is in- Hadoop 2 is now officially released by Apache!

Anyone who wants to use this release should download Hadoop via the Apache Mirrors.

Maven and Ivy users: the version you want to refer to is 2.2.0

The artifacts haven't trickled through to the public repos as of 2013-10-16-11:18 GMT -they are on the ASF staging repo and I've been using them happily all week.

This release marks an epic of development, with YARN being a fundamental rethink of what you can run in a Hadoop cluster: anything you can get to run in a distributed cluster where failures will happen, the Hadoop FileSystem the API for filesystem access  -be it in Java or a native client- and data is measured by the Petabyte.

YARN is going to get a lot of press for the way it transforms what you can do in the cluster, but HDFS itself has changed a lot. This is the first ASF release with active/passive HA -which is why Zookeeper is now on the classpath. CDH 4.x shipped with an earlier version of this- and as we haven't heard of any dramatic data loss events, consider it well tested in the field. Admittedly, if you misconfigure things failover may not happen, but that's something you can qualify with a kill -9 of the active namenode service. Do remember to have to >1 zookeeper instance before you try this -testing ZK failure should also be a qualification process. I think 5 is a number considered safer than 3, though I've heard of one cluster running with 9. Nobody has admitted going up to 11.

This release also adds to HDFS

1. NFS support: you can mount the FS as an NFS v3 filesystem. This doesn't give you the ability to write to anywhere other than the tail of a file -HFDS is still not-Posix. But then neither is NFS: its caching means that there is a few seconds worth of eventual consistency across nodes (\cite{Distributed Systems, Colouris, Dollimore & Kindberg, p331}).,
2. Snapshots: you can snapshot some of a filesystem and roll back to it later. Judging by the JIRAs, quotas get quite complex there. What it does mean is that it is harder to lose data by accidental rm -rf operations.
3. HDFS federation: datanodes can store data for different HDFS namenodes, -Block Storage is now a service- while clients can mount different HDFS filesystems to get access to the data. This is something of primarily of relevance to people working at Yahoo! and facebook scale -everyone else can just get more RAM for their NN and tune the GC options to not lock the server too much]

Hadoop 2 also adds is extensive testing all the way up the stack. In particular, there's a new HBase release coming out soon -hopefully HBase 0.96 will be out in days. Lots of other things have been tested against it -which has helped to identify any incompatibilities between the Hadoop 1.x MapReduce API (MRv1) and Hadoop 2's MRv2, while also getting patches into the the rest of the stack where appropriate. As new releases trickle out, everything will end up being built and qualified on Hadoop 2.

Which is why when you look at the features in Hadoop 2.x, as well as headline items "YARN, HDFS Snapshots, ...", you should also consider the testing and QA that went into this -this is the first stable Hadoop 2 release -the first one extensively tested all the way up the stack. Which is why everyone doing that QA -my colleagues, Matt Foley's QA team, the Bigtop developers, and anyone else working with Hadoop that built and tested their code against Hadoop 2.1 beta and later RCs -and reported bugs.

QA teams: your work is appreciated! Take the rest of the week off!

[photo: Deamz and Soker at St Philips, near Old Market]

Joining Hortonworks to evolve #hadoop

I've left HP. I did that on Monday, enjoying a final beer at lunchtime with my soon-to-be-ex colleagues, then heading home for a few weeks of parental responsibilities during the easter break.

Later this month, I will start work at Hortonworks, pushing the Hadoop stack forwards. I am really excited about this -I know a lot of people in the company already, and it's going to be great working with them!

Although the phrase "Big Data" is getting overused, it's obvious to me that there is a real coming together of different trends to make the whole Hadoop-based ecosystem as transformational as web servers were.

1. There are so many devices in the modern world acting as data sources -physical devices such as mobile phones and jet engines, services such as web applications, people making use of devices and services. 
2. In the past less data was generated -and it was normally thrown away. Too expensive to store, no perceived value.
3. The cost per TB of HDD has fallen such that you can now afford to keep that data for later analysis
4. You can't analyse it on single servers as the bandwidth of HDDs hasn't increased at the same rate as the storage capacity.
5. The performance of a single CPU has effectively topped out too. All that is coming is more cores, more operations/joule (hopefully), different forms of parallel computation. The free speedups that the CPU vendors used to dish out are over. It's either single-machine parallelism or multi-machine. Oh, and either way: heterogeneity of some form or other.
6. That means everyone is going to have to embrace parallel computing, on the single machine or in the rack -and with the right algorithms,  that rack can be made to deliver linear and sometimes superlinear speedup.
7. If you want to work with the big datasets that you can collect today, you are going to need a rack of servers and a framework to let you process the data.
8. The Hadoop platform provides the framework to store the data across those hard disks, and to distribute the work across them. It is becoming the single open-source alternative to Google's internal platform.

Where the future gets really interesting is that the Hadoop ecosystem provides those core services of a distributed computing platform: bulk storage (HDFS), scheduling (MRv2),  distributed state (ZooKeeper), integration with existing infrastructure (flume, squoop, Hive). These services can be used to build applications in and above Hadoop -HBase and Giraph are key examples; Cassandra a welcome friend. Big Data is the immediate reason to move into world, but ultimately it's Big Datacentre -not things like Java EE7 that just seem, well, so very last-century.

That's why I'm joining Hortonworks -to go full time on building the future platform for server-side computing.

[photo: preparing to descend into Crickhowell, Wales, 2011]

Just because you can rewrite your codebase doesn't mean you should have to

The strength of automated driven test suites is that you can verify that all the testable/covered parts of your system still work, even after major changes.

The strength of modern IDEs is at a click of mouse you can find wherever a class or method is used, so go to those places and edit them.

Even so: availability should not imply necessity: life is best when you don't have to use these features unless you really, really want to. Everything should work.

And usually it does. But not last week. A small bug surfaced on tuesday: you got an error marshalling strings with square brackets around, "[]". At first I did the obvious tactic: deny that this could be happening, but after repeated evidence to the contrary I sat down and had a look.

It turns out that the library, json-lib, with the same interface as the java Map interfaces, has an extra "feature" in that whenever you add a string attribute, that string value gets parsed as JSON if it starts with "{" or "[". Which means that parentnode.put("request","4,5") would add the string attribute "request":"4,5" to the parent node, the slight variation parentnode.put("request","[4,5]") would generate the attribute "request":[ 4 , 5]. This is fundamentally different, and challenges any assumption the recipient had about types in marshalled data, as the types varies depending on the contents of the strings being marshalled.

Needless to say, I was unhappy. At least by the point that unhappiness was reached, there were now some tests to work out what was going wrong. Which made fixing it possible, the fix being to always single quote strings being added, such as parentnode.put("request","'[4,5]'"). When the put() operation is invoked, the single quotes are stripped, and the unparsed inner values become the attribute. With careful wrapping of the put operations -and ensuring that only one set of double quotes is placed around every string attribute, the tests passed, everything worked, and a new nightly release went out with the issue marked as closed.

Except the next day, the issues came back. Because it wasn't fixed. Not once you took that node, parentnode, and added it under another node: messagenode.put("payload", parentnode).

Doing that appears to trigger a reparse of every string value. Which means all safely planted array strings end up being reparsed, and converted from strings to JSON arrays.

At this point, the unhappiness level changed from "medium" to excessive. With the new tests replicating this behaviour, and no obvious in-source switch to say "be less helpful", the only solution that seemed 100% likely to work with all possible payloads and orderings of payload construction was to rip out the entire library and replace it with one that did not exhibit the same behaviour.

Which is what I spent Thursday and Friday doing: a complete removal of all uses of json-lib and replacement with Jackson. Which, I not only regret in time wasted, but in the way that the json-lib Java-friendly object model is better than the Jackson "look like the DOM" world view, because the DOM, ubiquitous as it is, is pretty painful. Yes, with experience of the XML parsing world, I can certainly use it -but that did not mean I enjoy it. I've had to rip out and place server side and client side code, other things that are visible to other bits of the system that used the same JSON library. I haven't fixed all that code -but add enough downconvert/upconvert that the code compiles and appears to work.

Four hours over two days writing tests to show this problem existed -then
two full days of repair: switching libraries, backtracking on method calls, changing types and seeing won't compile, running tests. Some extra tests this weekend and then only the merge with two days worth of other peoples changes and some reruns and it's ready to commit. That will leave Jenkins to do the final retest and email, then everyone downstream will have to fix their side of the down- and up- converted code (the conversion methods marked as @Deprecated to make them easy to spot), which will take 1+ hour on monday for a couple of people.

When we look back at this, what will we be able to say?

We can now reliably send strings with square brackets around.

This is one of those weeks that I would never use in a motivational talk for anyone interested in taking software engineering.

[Artwork: 3Dom, "Remember the Future", Richmond Road, Montpelier]

Disclaimer: I reserve the right to not know what I'm talking about

There have been some links to this blog with the implications that either because I am a researcher at HP Laboratiories with commit rights to a number of projects -including Apache Hadoop- then my opinions may have some value.

Not so.

Everything I say is obviously my own opinions, do not reflect those of my employer, may be at odds with corporate strategies of which I am blissfully unaware of, if I do mention corporate plans they may change on a whim, etc, etc.

I do work at HP Labs -which is a fun place to work- but think about what that means. It means that I don't do stuff that goes straight into shipping products. The code I write is not considered safe for humanity. I may talk about it, I may write about it, some may get into the field, but only after rigorous review by people who care about such things.

This makes it a bit like a university, except that I don't teach courses. Cleary I am not considered safe near students either, in case I damage thier minds.

"Oh well", people say, "he's a Hadoop committer -that must count for something?"

It does, but I don't get to spend enough time there to keep up with the changes, let alone do interesting stuff of my own ideas.

Whenever I create an issue and say "I will do this", other people in the project -Arun, Tom, Todd and others will stare at their inbox in despair, with the same expression on their faces that my family adopts when I say "I will bake something".

Because then, yes, I may come up with something unusual and possibly even tasty, but they will know that the kitchen will be left with a thin layer of flour over everything, there will be pools of milk, and whatever went in the oven went into the closest bowl-shaped thing I could reach with one hand, rather than anything designed to be used in an oven, let alone watertight. Even if I didn't use flour or milk.

Bear that in mind before believing anything written here.

[Photo:me somewhere in the Belledone mountains; French Alps. For some reason French villages never like it when people on bicycles come to a halt]

301 Moved Permanently

I've moved my blog from 1060.org here. Why? Well, the team at 1060research have pushed out a new release of their NetKernel product, on which the blog was running, and if I wanted to retain the URLs I'd have to upgrade the code myself.  Being lazy and all, I opted not to.

What have I been up to since going offline

1. On twitter @steveloughran. Idle chatter.
2. Finishing up some major project at work that has kept be busy for the past 12-18 months. I am feeling more relaxed now. 
3. Coding in Groovy. It's like Java only better, and trivial to switch between the too. There's great IDE support in IntelliJ IDEA too.
4. Doing some proper Computer Science stuff, as opposed to Software Engineering.
5. Paper Reading. This may seem dull, but there is a lot of interesting stuff out there. If you spend too much time knee-deep in various projects' codebases you get sucked into various issues (log4j configuration etc), and out of touch with higher level problems. 
6. Holiday on the south coast of England. Not far; lazy. I lost a camera, which is a pity, but I've replaced it already.

My most recent set of readings was all the big datacentre papers on DRAM and HDD failures. A good way to revise on concepts like Poisson Distributions, Gauss Distributions, Weibull Distributions (which I never knew of before), and lots of other math-hard problems. It's shocking how much maths I have forgotten; I keep seeing these bits on the paper where they do integration or differentiation and say "clearly then" and the words mean nothing to me. I will have to revise some fundamentals.

With the work I've been doing wrapping up, I'm hoping to get more involved in Hadoop and Hadoop related work. I don't have a schedule for that -I'm just reading now.