Why is so much of my life wasted waiting for test runs to complete?

I've spent the weekend enduring the pain of kerberos-related functional test failures, test runs that take time to finish, especially as its token expiry between deployed services which is the Source of Madness (copyright (c) 1988 MIT).

DSC_0128 - Nibaly

Anyone who follows me on Strava can infer when those runs take place as if its a long one, I've nipped down to the road bike on the turbo trainer and done a bit of exercise while waiting for the results.

Which is all well and good except for one point: why do I have to wait so long?

While a test is running, the different services in the cluster are all generating timestamped events, "log messages" as they usually known,  The code the test runner itself is also generating a stream of events, from any client-side code and wrapping JUnit/xUnit runners, again, tuples of (timestamp, thread, module, level, text) + implicitly (process, host). And of course there's the actual outcome of each test.

Why do I have to wait until the entire test run is completed for those results to appear?

There's no fundamental reason for that to be the case. It's just the way that the functional tests have evolved under the unit test runners, test runners designed to run short lived unit tests of little classes, runs where stdout and stderr were captured without any expectation of structured format. When <junit> completed individual test cases, it'd save the XML DOM build in memory to an XML file under build/tests. After Junit itself completed, the build.xml would have a <junitreport> task to map XML -> HTML in a wondrous piece of XSLT. 

Maven surefire does exactly the same thing, except it's build reporter doesn't make it easy to stream the results to both XML files and to the console at the same time.

The CI tooling: Cruise Control and its successors, of which Jenkins is the near-universal standard took those same XML reports and now generate their own statistics, and again wait for the reports to be generated at the end of the test run.

That means those of us who are waiting for a test to finish have a limited set of choices
  1. Tweak the logging and output to the console, stare at it waiting to see stack traces to go by
  2. Run a single failing test repeatedly until you fix it, again, staring at the output. In doing so you neglect the rest of the code until at the end of the day you are left with the choices of (a) run the hour long test of everything to make sure there are no regressions and (b) commit and push and expect a remote Jenkins to find the problem, at which point you may have broken a test and either need to get those results again & fix them, or rely on the goodwill of a colleage (special callout, Ted Yu, the person who usually ends up fixing SLIDER-1 issues)
Usually I drift into the single-test mode, but first you need to identify the problem. And even then, if the test takes a few minutes, each iteration hurts. And there's the hassle of collecting the logs, correlating events across machines and services to try and understand what's going on. If you want more detail, its over to http:{service host:port}/logLevel and tuning up the logs to capture more events on the next iteration, and so you are off again.

A hard-to-identify problem becomes a "very slow to identify problem", or productivity killer.

Sitting waiting for tests is a waste of time for software engineers.

What to do?

There's parallelisation. Apparently there's some parallelised test runner that the Cloudera team has which we could perhaps pick up and make reusable. That would be great, but you are still waiting for the end of the test runs for the results, unless you are going to ssh into the hosts and play tail -f against log files, or grep for specific event texts.

What would be just as critical is: real time reporting of test results.

I've discussed before how we need to radically improve tests and test runners.

What we also have to recognise is that the test reporting infrastructure equally dates from the era of unit tests taking milliseconds, full test suites and XSL transformations of the results taking 10s of seconds at most.

The world has moved on from unit tests.

What do I want now? As well as the streaming out of those events in structured form directly to the some aggregrator, I want that test runner to be immediately publishing the aggregate event stream and test results to some viewer better than four consoles with tail -f streaming text files (or worse, XML reports). I want HTML pages as they come in, with my test report initially showing all tests enumerated, then filling up as tests run and fail. I'd like the runner to known (history, user input?) which tests were failing, and so run them first. If I checked in a patch to a specific test class, that'll be the one I want to run next, followed by everything else in the same module (assuming locality of test coverage).

Once I've got this, the CI tooling I'm going to run will change. It won't be a central machine or pool of them, it'll be a VM hosted locally or some cloud infrastructure. Probably the latter, so it won't be fighting for RAM and CPU time with the IDE.

Whenever I commit and push a patch to my private branch, the tests should run.

It's my own personal CI instance, it gets to run my tests, and I get to have a browser window open keeping track of the status while I get on with doing other things.

We can do this: its just the test runner reporting being switched from batch to streaming, with the HTML views adapting.

If we're building the largest distributed computing systems on the planet, we can't say that this is beyond us.

(Photo: Nibali descending from the Chartreuse Massif into Grenoble; Richie Porte and others just behind him doomed to failure on the climb to Chamrousse, TdF 2014 stage 13)


The Manchester Dataflow Machine: obscure historical computer architecture of the month

Milind has flagged up some nearby students at Bristol Uni attempting to reimplement the Transputer. I should look at that some time. For now, some little paper I was reading last week while frittering away an unexpected few days in a local hospital.

The Manchester Dataflow Machine

The MDM was some mid-1980s exploration of a post-microprocessor architecture, the same era as RISC, the Transputer and others. It was built from 74F-series logic, TTL rather than CMOS; performance numbers aren't particularly compelling by today's standard. What matters more is its architectural model.

In a classic von-Neumann CPU design, there's a shared memory for data and code; the program counter tells the CPU where to fetch the next instruction from. It's read, translated into an operation and then executed. Ops can work with memory, registers exist as an intermediate stage, essentially an explicit optimisation to deal with memory latency, branching implemented by changing the PC. The order of execution of operations is defined by the order of machine code instructions (for anyone about to disagree with me there: wait; we are talking pure von-Neumann here). It's a nice simple conceptual model, but has some flaws. A key one is that some operations take a long time (memory reads if there a cache misses, some arithmetic operations (example: division). The CPU waits, "stalls" until the operation completes, even if there is a pipeline capable of executing different stages of more than one operation at a time.

What the MDM did was accept the inevitability of those delays -and then try to eliminate them by saying "there is no program counter, there are only data dependencies".

Instead of an explicitly ordered sequence of instructions, your code lists operations a unset or binary operations against the output of previous actions and/or fetches from memory. The CPU then executes those operations in a order which guarantees those dependencies are met, but where the exact order is chosen based on the explicit dependency graph, not the implicit one of the sequence of opcodes produced by the compiler/developer.

Implementation-wise, they had a pool of functional units, capable of doing different operations, wired up by something which would get the set of instructions from (somewhere), stick them in the set of potential operations, and as slots freed up in the functional units, dispatch operations which were ready. Those operations generated results, which would make downstream operations ready for dispatch.

The Manchester Dataflow Machine Architecture

This design offered parallel execution proportional to the number of functional units: add more adders, shifters, dividers and they could be kept busy. Memory IO? Again, a functional unit could handle a read or a write, though supporting multiple units may be trickier. Otherwise, the big limitation on performance comes in conditional branching: you can't schedule any work until you know it's conditions are met. Condition evaluation, then, becomes a function of its own, with all code that comes after dependent on the specific outcomes of the condition.

To make this all usable, a dataflow language was needed; the one the paper talks about is SISAL. This looks like a functional language, one designed to compile down to the opcodes a dataflow machine needs.

Did it work? yes. Did it get adopted? No. Classic procedural CPUs with classic procedural HLLs compiling down to assembly language where the PC formed an implicit state variable is what won out. It's how we code and what we code for.

And yet, what are we coding: dataflow frameworks and applications at the scale of the datacentre. What are MapReduce jobs but a two step dataflow? What is Pig but a dataflow language? Or Cascading? What are the query plans generated by SQL engines but different data flow graphs?

And if you look at Dryad and Tez, you've got a cluster-wide dataflow engine.

At the Petascale, then, we are working in the Dataflow Space.

What we aren't doing is working in that model in the implementation languages. Here we write procedural code that is either converted to JVM bytecodes (for an abstract register machine), or compiled straight down to assembly language? And those JVM bytecodes: down to machine code at runtime. What those compilers can do is reorder the generated opcodes based on the dataflow dependency graph which it has inferred from the source. That is, even though we went and wrote procedurally, the compilers reversed the data dependencies, and generated a sequence of operations which it felt were more optimal, based on its knowledge of and assumptions about the target CPU and the cache/memory architecture within which it resides.

And this is the fun bit, which explains why the MDM paper deserves reading: the Intel P6 CPU of the late 1990s —as are all its successors- right at the heart, built around a dataflow model. They take those x86 opcodes in the order lovingly crafted by the compiler or hard-core x86 assembler coder and go "you meant well, but due to things like memory read delays, let us choose a more optimal ordering for your routines". Admittedly, they don't use the MDM architecture, instead they use Tomasulo's algorithm from the IBM 360 mainframes

A key feature there is "reservation stations", essentially register aliasing, addressing the issue that Intel parts have a limited and inconsistent set of registers. If one series of operations work on registers eax and ebx and a follow-on sequence overwrites those registers, the second set gets a virtual set of registers to play with. Hence, it doesn't matter if operations reuse a register, the execution order is really that of the data availability. The other big trick: speculative execution.

The P6 and successor parts will perform operations past a branch, provided the results of the operations can fit into (aliased) registers, and not anything with externally visible effects (writes, port IO, ...). The CPU tags these operations as speculative, and only realises them when the outcome of the branch is known. This means you could have a number of speculated operations, such as a read and a shift on that data, with the final output being visible once the branch is known to be taken. Predict the branch correctly and all pending operations can be realised,; any effects made visible. To maintain the illusion of sequential non-speculative operation, all operations with destinations that can't be aliased away have to be blocked until the branch result is known. For some extra fun, any failures of those speculated operations can only be raised when the branch outcome is known. Furthermore, it has to be the first failing instruction in the linear, PC-defined sequence that must visibly fail first, even if an operation actually executed ahead of it had failed. That's a bit of complexity that gets glossed over when the phrase "out of order execution" is mentioned. More accurate would be "speculative data-flow driven execution with register aliasing and delayed fault realisation".

Now, for all that to work properly, that data flow has to be valid: dependencies have to be explicit. Which isn't at all obvious once you have more than one thread manipulating shared data, more than one CPU executing operations in orders driven by its local view of the data dependencies.

Initial state
int p=0;
int ready = 0;
int outcome=100;
int val = 0;

Thread 1
p = &outcome;
ready = 1;

Thread 2
if (ready) val = *p;

Neither thread knows of the implicit dependency of p only being guaranteed to be valid after 'ready' is set. if the deference val = *p  is speculatively executed before the condition if (ready)is evaluated, then instead of ready==true implying val == 100, you could now have a stack traces from attempting to read the value at address 0. This will of course be an
obscure and intermittent bug which will only surface in the field in many-core systems, and never under the debugger.

The key point is: even if the compiler or assembly code orders things to meet your expectations, the CPU can make its own decisions. 

The only way to make your expectations clear is by getting the generated machine code to contain flags to indicate the happens-before requirements, which, when you think about it, is simply adding another explicit dependency in the data flow, a must-happen-before operator in the directed graph. There isn't an explicit opcode for that, barrier opcode goes in which tell the CPU to ensure that all operations listed in the machine code before that op will complete before the barrier. Equally importantly, that nothing will be reordered or speculatively executed ahead of it: all successor operations will then happen after. That is, the op code becomes a dependency on all predecessor operations, all that come after have the must-come-after dependency on this barrier. In x86, any operation with the LOCK attribute is a barrier, as are others (like RDTSCP). And in Java, the volatile keyword is mapped to a locked read or write, so is implicitly a barrier. No operations will be promoted ahead of the a volatile R/W, either by Javac, or by the CPU, nor will any be delayed. This means volatile operations can be very expensive, as if you have a series of them, even if there is no explicit data-dependency, they will be executed in-order. It also means that at compile-time, javac will not move operations on volatile fields out of a loop, even if there's no apparent update to them.

Given these details on CPU internals, it should be clear that we now have dataflow at the peta-scale, and at the micro-scale, where what appear to be sequential operations have their data dependencies used to reorder things for faster execution. It's only the bits in the middle that are procedural. Which is kind of ironic really: why can't it be dataflow all the way down? Well the MDM offered that, but nobody took up the offering.

Does it matter? Maybe not. Except that if you look at Intel's recent CPU work, it's adding new modules on the die for specific operations. First CRC, then AES encryption -and the new erasure coding in HDFS work is using some other native operations. By their very nature, these modules are intended to implement in Si algorithms which take many cycles to process per memory access. Which means they are inherently significantly slower than existing functional units in the CPU. Unless the hardware implementations are as fast as operations like floating point division, code that depends on the new operations' results are going to be held up for a while. Because all that OOO dataflow work is hidden, there's no way in the x86 code to send that work off asynchronously.

It'd be interesting to consider whether it would be better to actually have some dataflow view of those slow operations, something like Java's futures, where slow operations are fired off asynchronously, with a follow-up operation to block until the result of the operation is ready -with any failures being raised as this point. Do that and you start to give the coders and compiler writers visibility into where big delays can surface, and the ability to deal with them, or at least optimise around them.

Of course, you do need to make that stuff visible in the language too

Presumably someone has done work like that; I'm just not current enough with my reading.

Further Reading
[1] How Java is having its memory model tightened
[2] How C++ is giving you more options to do advanced memory get/set


3 years at Hortonworks!

In 2012 I handed in my notice at HP Laboratories and joined Hortonworks : this May is the third anniversary of my joining the team.


I didn't have to leave HP, and in the corporate labs I has reasonable freedom to work on things I found interesting. Yet is was through those interesting things that we'd discovered Hadoop. Paolo Castagna introduced me to it, as he bubbled with enthusiasm for what he felt was he future of server side computing. At the time I was working on the problem of deploying and managing smaller systems -but doing so in the emergent cloud infrastructures. Hadoop was initially another interesting deployment problem: one designed to scale and cope with failures, yet also built on the assumption of a set of physical hosts, hosts with fixed names and addresses, hosts with persistent storage and whose  failures would be independent. some of the work I did at that time with Julio Guijarro included dynamic Hadoop clusters, Mombasa (the long haul route to see elephants). The work behind the scenes to give Hadoop services more dynamic deployments, HADOOP-3628, earned me Hadoop committership. While the branch was never merged in, the YARN service model shows its heritage.

While we were doing this, HP customers were also discovering Hadoop —and building their clusters. I remember the first email coming in from a sales team who had been asked to quote the terasort performance of their servers: the sales team hadn't heard of a terasort and didn't know what to do. We helped. Before long we were the back-end team on many of the big Hadoop deals, helping define and review the proposed hardware specs, reviewing and sometimes co-authoring the bid responses. And what bids they were! At first I thought a petabyte was a lot of storage —but soon some of the deals were for 10+, 20+ PB. Projects where issues like rack weight and HDD resonance were as key to worry about as power and logistics of getting the servers delivered. Production lines which needed to be block booked for a week or two, but in doing so allowing server customisation: USB ports surplus? Skip them. How many CPU sockets to fill-and with what SKU? Want 2.5" laptop HDDs for bandwidth over 3.5" capacity oriented storage? All arrangeable, with even the option of a week-long burn in and benchmark session as an optional extra. This would show that the system worked as requested, including setting benchmarks for sorting 5+ PB of data that would never be published out of fear of scaring people (bear that in mind when you read blogs posts showing how technology X out-terasorts Hadoop —the really big Hadoop sort numbers are of 10+ PB on real clusters, not EC2 XXL SSD instances, and they don't get published).

These were big projects and it was really fun to be involved.

At the same time though, I felt that HP was missing the opportunity, the big picture. The server group was happy to sell the systems for x86 system margins, other groups to set them up. But where was the storage group? Giving us HPL folk grief for denying them the multi-PB storage deals —even though they lacked a Hadoop story and didn't seem to appreciate the technology. Networking? Doing great stuff for HFT systems where buffering was anathema; delivering systems for the enterprise capable of handling intermittent VM migration. But not systems optimised for sustained full link rate bandwidth, decent buffering and backbone scalability through technologies like TRILL or Shortest Path Bridging (you can get these now, BTW).

The whole Big Data revolution was taking place in front of HP: OSS software enabling massive scale storage and compute systems, the underlying commodity hardware making PB storeable, and the explosion in data sources giving the data to work with. And while HP was building a significant portion of the clusters, it hadn't recognised that this was a revolution. It was reminiscent of the mid 1990s, when the idea of Web Servers was seen as "just another use of a unix workstation".

I left to be part of that Big Data revolution, joining the team I'd got to know through the OSS development, Hortonworks, and so defining the future, rather than despair about HP's apparent failure to recognise change. Many of us from that era left: Audrey and I to Hortonworks, Steve and Scott to RedHat, Castagna to Cloudera. Before I get complaints from Julio and Chris, —yes some of the first generation of Hadoop experts are still there, the company is taking Big Data seriously, and there are now many skilled people working on it. Just not me.

What have I done in those three years? Lots of things! Some of the big ones include:
  • Hadoop 1 High Availability. One of the people I worked with at VMWare, Jun Ping, is now a valued colleague of mine.
  • OpenStack support: Much of the hadoop-openstack code is mine, particularly the tests.
  • The Hadoop FS Specification: defining a Python-like syntax for Spivey's Z notation, delving through the HDFS and Hadoop source to really define what  a Hadoop filesystem is expected to do. From the OpenStack Swift work I'd discovered the unwritten assumptions & set out to define them, then build a test suite to help anyone trying to integrate their FS or object store with Hadoop to get started. This was my little Friday afternoon project; nobody asked me to do it -but now that it is there it's proven invaluable in getting the s3a S3 client working, as well as being one of the first checkpoints for anyone who wants to get Hadoop to work on other filesystems. Arguably that helps filesystem competitors —yet what it is really meant to do is give users a stable underpinning of the filesystem, beyond just the method signatures.
  • The YARN-117 service model. I didn't start that work, I just did my best to get the experience of the SmartFrog and HADOOP-3628 service models in there. I do still need to document it better, and get the workflow and service launcher into the core code base; Slider is built around them.
  • Hoya: proof of concept YARN application to show that HBase was deployable  as a dynamic YARN application, and initial driver for the YARN-896 services-on-YARN work.
  • Apache Slider (incubating). A production quality successor to Hoya, combining the lessons from it with the Ambari agent experience, producing an engine to make many applications deplorable on YARN though a minimal amount of Python code. Slider is integrate with Ambari, but it works standalone against ASF Hadoop 2.6 and the latest CDH 5.4 release (apparently). I've really got a good insight into the problems of placement of work where access to data has to be balanced with failure resilience; enough to write a paper if I felt like it —rather than just a blog post.
  • The YARN Service Registry. Again, something I need to explain more. An HA registry service for Hadoop clusters, where static and dynamic applications can be registered and used. Slider depends on it for client applications to find Slider and its deployed services; it is critical for internal bonding in the presence of failures. It's also the first bit of core Hadoop with a formal specification in TLA+.
  • Spark on YARN enhancements. SPARK-1537 is my first bit of work there, having the spark history server use the YARN timeline service. Spark internals in Scala, collaboration with the YARN team on REST API definitions and reapplying the test experience of Slider to accompany this with quality tests.
  • Recently: some spare time work mentoring S3a: into a production ready state.
  • Working with colleagues to help shape our vision of the future of Hadoop. Apache Hadoop is a global OSS project, one which colleagues, competitors and users of the technology collaborate to build. I, like the rest of my colleagues get a say there, helping define where we think it can go: then building it.

The latter is a key one to call out. At HP an inordinate amount of my time was spent trying to argue the case for things like Hadoop inside the company itself, mostly by way of PowerPoint-over-email. I don't have to do that any more. When we make decisions it's s done rapidly,  pulling in the relevant people, rather than the inertial hierarchy of indifference and refusal which I sometimes felt I'd encountered in HP.

Which is why working at Hortonworks is so great: I'm working with great people, on grand projects —yet doing this a process where my code is in people's hands within weeks to months, and where an agile team keeps the company nimble. and pretty much all my work has shipped.

If you look at how the work has included applied formal methods, distributed testing, models of system failure and dynamic service deployment, I'm combining production software development with higher level work that is no different than what I was doing in a corporate R&D lab -except with shipping code.

Hortonworks is hiring. If what I've been up to -and how I've been doing it- sounds exciting: then get in touch. That particularly applies to my former HPL colleagues, who have to make their mind up where to go: ink vs enterprise. There is another option: us.


It's OK to submit patches without tests: just show the correctness proofs instead


In a window adjacent to the browser I'm typing this, I'm doing a clean build of trunk prior to adding a two line patch to Hadoop -and the 20+ lines needed to test that patch; Intellij burbles away for 5 minutes as it does on any changes to the POMs on a project which has a significant of the hadoop stack loaded.

The patch I'm going write is to fix a bug introduced by a previous three line patch, one that didn't come with patches because it was "trivial".

It may have been a trivial patch, but it was a broken trivial patch. It's not so much that the test cases would have found this, but they'd have forced the author to think more about the inputs to the two-line method, and what outputs would be expected. Then we get some tests that generate the desired outputs for the different outputs, ones that guarantee that over time the behaviour is constant.

Instead we have a colleague spending a day trying to track down a remote functional test run, one that has been reduced to a multi-hop stack trace problem. The big functional test suites did find that bug (good), but because the cost of debugging and isolating that failure is higher handling that failure is more expensive.; With better distributed test tooling, especially log aggregation and analysis, that cost should be lower —but it's still pretty needless for something that could have been avoided simply by thinking through the inputs.

Complex functional system tests should not be used as a substitute for unit tests on isolated bits of code. 

I'm not going to highlight the issue, or identify who wrote that patch, because it's not fair: it could be any of us, and I am just as guilty of submitting "trivial" patches. If something is 1-2 lines long, it's really hard to justify in your brain the effort of writing the more complex tests to go with it.

If the code actually works as intended, you've saved time and all is well. But if it doesn't, that failure shows up later in full stack tests (cost & time), the field (very expensive), and either way ends up being fixed the way the original could have been done.

And as documented many times before: it's that thinking about inputs & outputs that forces you to write good code.

Anyway: I have tests to write now, before turning on to what is the kind of problem where those functional tests are justified, such as Jersey client not working on Java 8. (I can replicate that in a unit test, but only in my windows server/java 8 VM)

In future, if I see anyone use "trivial patch" as a reason to not write tests, I'll be wheeling out the -1 veto.

I do however, offer an exception: if people can prove their code works, I'll be happy

(photo: wall in Brussels)


Distributed System Testing: where now, where next?

Confluent have announced they are looking for someone to work on an open source framework for distributed system testing.

I am really glad that they are sitting down to do this. Indeed, I've thought about sitting down to do it myself, the main reason I've been inactive there is "too much other stuff to do".


Distributed System Testing is the unspoken problem of Distributed Computing. In single-host applications, all you need to do is show that the application "works" on the target system, with its OS,  enviroment (timezone, locale, ...), installed dependencies and application configuration.

In modern distributed computing you need to show that the distributed application works across a set of machines, in the presence of failures.

Equally importantly: when your tests fail, you need the ability to determine why they failed.

I think there is much scope to improve here, as well as the fundamental problem: defining works in the context of distributed computing.

I should write a post on that in future. For now, my current stance is: we need stricter specification of desired observable behaviour and implementation details. While I have been doing some Formal Specification work within the Hadoop codebase, there's a lot more work to be done there —and I can't do it all myself.

Assuming that there is a good specification of behaviour, you can then go on to defining tests which observe the state of the system, within the configuration space of the system (now including multiple hosts and the network), during a time period in which failures occur. The observable state should continue to match the specification, and if not, you want get the logs to determine why not. Note here that "observed state" can be pretty broad, and includes
  • Correct processing of requests
  • The ability to serialize an incoming stream of requests from multiple clients (or at least, to not demonstrate non-serialized behaviour)
  • Time to execute operations is one (performance),
  • Ability to support the desired request rate (scalability)
  • Persistence of state, where appropriate
  • Reslience to failures of : dependent services, network, hosts, 
  • Reporting of detected failure conditions to users and machines (operations needs)
  • Ideally: ability to continue in the presence of byzantine failures. Or at least detect them and recover.
  • Ability to interact with different versions of software (clients, servers, peers)
  • Maybe: ability to interact with different implementations of the same protocol.
I've written some slides on this topic, way back in 2006, Distributed Testing with SmartFrog. There's even a sub-VGA video to go with it from the 2006 Google Test Automation Conference.

My stance there was
  1. Tests themselves can be viewed as part of a larger distributed system
  2. They can be deployed with your automated deployment tools, bonded to the deployed system via the configuration management infrastructure
  3. You can use the existing unit test runners as a gateway to these tests, but reporting and logging needs to be improved.
  4. Data analysis is a critical area to be worked on.
I didn't look at system failures, I don't think I was worry enough about that, showing we weren't deploying things at scale, and before cloud computing took failures mainstream. Nowadays nobody can avoid thinking about VM loss at the very least.

Given I did those slides nine years ago, have things improved? Not much, no
  • Test runners are still all generating the Ant XML test reports written along with the matching XSLT transforms up by Stephane Balliez in 2000/2001
  • Continuous Integration servers have got a lot better, but even Jenkins, wonderful as it is, presents results as if they were independent builds, rather than a matrix of (app, environment, time). We may get individual build happiness, but we don't get reports  by test, showing that Hadoop/TestATSIntegrationFailures is working intermittently on all debian systems -but has been reliable elsewhere. The data is all there, but the reporting isn't.
  • Part of the problem is that they are still working with that XML format, one that, due to its use of XML attributes to summarise the run, buffers things in memory until the test test case finishes, then writes out the results. stdout and stderr may get reported -but only for the test client, and even then, there's no awareness of the structure of log messages
  • Failure conditions aren't usually being explicitly generated. Sometimes they happen, but then its complaints about the build or the target host being broken.
  • Email reports from the CI tooling is also pretty terse. You may get the "build broken, test XYZ with commits N1-N2", but again, you can get one per build, rather than a summary of overall system health.
  • With a large dependent graph of applications (hello, Hadoop stack!), there's a lot of regression testing that needs to take place —and fault tracking when something downstream fails. 
  • Those big system tests generate many, many logs, but they are often really hard to debug. If you haven't spent time with 3+ windows trying to sync up log events, you've not been doing test runs.
  • In a VM world, those VMs are often gone by the time you get told there's a problem.
  • Then there's the extended life test runs, the ones where we have to run things for a few days with Kerberos tokens set to expire hourly, while a set of clients generate realistic loads and random servers get restarted.
Things have got harder: bigger systems, more failure modes, a whole stack of applications —yet testing hasn't kept up.

In slider I did sit down to do something that would work within the constraints of the current test runner infrastructure yet still let us do functional tests against remote Hadoop clusters of variable size . Our functional test suite, funtests, uses Apache Bigtop's script launcher to start  Slider via its shell/py scripts. This tests those scripts on all test platforms (though it turns out, not enough locales), and forces us to have a meaningful set of exit codes —enough to distinguish the desired failure conditions from unexpected ones. Those tests can deploy slider applications on secure/insecure clusters (I keep my VM configs on github, for the curious), deploy test containers for basic operations, upgrade test, failure handling tests. For failure generation our IPC protocol includes messages to kill a container, and to have the AM kill itself with a chosen exit code.

For testing slider-deployed HBase and accumulo we go one step further. Slider deploys the application, and we run the normal application functional test suites with slider set up to generate failures.

How do we do that? With the Slider Integral Chaos Monkey. That's something which can run in the AM, and, at a configured interval, roll some virtual dice to see if the enabled failure events should be triggered: currently container and AM (we make sure the AM isn't chosen in the container kill monkey action, and have a startup delay to let the test runs settle in before starting to react).

Does it work? Yes. Which is good, because if things don't work, we've got the logs of all the machines in the cluster that ran slider to go through. Ideally, YARN-aggregated logs would suffice, but not if there's something up between YARN and the OS.

So: test runner I'm happy with. Remote deployment, failure injection, both structured and random. Launchable from my deskop and CI tooling; tests can be designed to scale. For testing rolling upgrades (Slider 0.80-incubating feature), we run the same client app while upgrading the system. Again: silence is golden.

Where I think much work needs to be done is what I've mentioned before: the reporting of problems and the tooling to determine why a test has failed.

We have the underlying infrastructure to stream logs out to things like HDFS or other services, there's nothing to stop us writing code to collect and aggregate those -with the recipient using the order of arrival to place an approximate time on events (not a perfect order, obviously, but better than log events with clocks that are wrong). We can collect those entire test run histories, along with as much environment information that we could grab and preserve. Junit: system properties. My ideal: VM snapshots & virtual network configs.

Then we'd go beyond XSLT reports of test runs and go to modern big data analysis tools. I'm going to propose here: Spark Why? so you can do local scripts, things in Jenkins & JUnit, and larger bulk operations. And for that get-your-hands-dirty test-debug festival, I can use a notebook like Apache Zepplin (incubating) can then be a no

we should be using our analysis tools for the automated analysis and reporting of test runs, the data science tooling for the debugging process.

Like I said, I'm not going to do all this. I will point to a lovely bit of code by Andrew Or @ databricks, spark-test-failures. which gets Jenkins's JSON-formatted test run history, determines flaky tests and posts the results on google docs. That's just a hint of what is possible —yet it shows the path forwards.

(Photo: "crayola", St Pauls:work commissioned by the residents)


Dynamic Datacentre Applications: The placement problem

We're just in the wrap-up for Slider-0.80-incubating; A windows VM nearby is busy downloading the source .zip file and verifying that it builds & tests on windows.

Before people feel sorry for me consider this: given a choice between windows C++ code and debugging Kerberos,  I think I'd rather bring up MSDN in a copy of IE6 while playing edit-and-continue games in Visual Studio than try and debug the obscure error messages you get with kerberos on Java. Not only are they obscure, my latest Java 7 update event changed the text of the error which means "now we've upgraded the JVM you don't have the encryption settings to use kerberos".

Which is a shame, because I have to make sure everything works with kerberos.

Anyway: build works and the tests are running —which keeps me happy. If all goes to plan, Slider 0.80-incubating will be there for download within a week.

Some of the new features include
  • SLIDER-780: Ability to deploy docker packages
  • SLIDER-663 zero-package cluster definition. (It has a different name, but it essentially means "no need to build a redistributable zip file for each application). While the zip-based distribution is essential for things you want to share, for things you are developing or using yourself, this is lighter weight.
  • SLIDER-733 Ability to install a package on top of another one. This addresses "the coprocessor problem": how to add a new HBase coprocessor JAR without rebuilding everything. And, with SLIDER-633, you can define that new package easily,
Notably, all these features are by people other than me —specifically, by colleagues.

 slider team

As anyone who knows me will realise: that's because my Hortonworks colleagues are really great people who know more about computing than I ever have or will and are better at applying that knowledge than myself —someone who uses "test driven development" as a way of hiding his inability to get anything to work right first time.

And yet these people still allow me to work with them —showing there still things that they consider something I'm still up to handling.

What have I been up to? Placement, specifically SLIDER-611: Über-JIRA - placement phase 2

Right from the outset, Hadoop HDFS and MR have been built on some assumptions about failure, availability and bandwidth
  1. Disks will fail: deal with it by using replication over 2h replacement part support contracts and RAID-array rebuilding
  2. Servers will fail: app developers have to plan for that.
  3. Some servers are just unreliable; apps may work this out without even needing to be told. Example: stragglers during a map identifies servers whose disks may be in trouble.
  4. If an application fails a few times it's unreliable and should be killed.
  5. The placement/scheduling of work is primarily an optimisation to conserve bandwidth. That is: you place work for proximity to the desired data.
  6. If the desired placement cannot be obtained, placement elsewhere is usually acceptable.
  7. It's better to have work come up fast somewhere near the data than wait for minutes for it to potentially come up on the actual node requested.
  8. If work is placed on a different machine, the cost to other apps (i.e. the opportunity cost) is acceptable.
  9. It's OK if two containers get allocated to the same machines ("affinity is not considered harmful")
  10. The actions of one app running on a node are isolated from the others (e.g. CPU throttling and vmem limiting is sufficient to limit resource conflict).
  11. Work has a finite duration, and containers know when they are finished. Nothing else needs to make that decision for them.
For analytics works where the data is known, the placement strategy is ideal. It ensures that work comes up fast at the expense of locality: bandwidth is precious, but time even more so. a nearby-placement will run the work at a cost of network capacity and the implicit risk of locality placement for other work trying to run on the (now in use) node.

But what about long lived services, such as HBase and Kafka?

HBase uses short-circuit reads for maximum performance working with local data. Although region servers don't have to be co-located with the data, until there's a full Hbase compaction, most of the data for an RS is likely to be remote. (for each Block b, P(data-local) = f(nodes/3), (roughly; the 3-replica-2-rack policy complicates the equation as blocks are not spread completely randomly).

Therefore: restarting on the wrong node can slow down that region server's performance for an extended period of time.

HBase is often used low-latency apps; if other things are running there then it can impact performance. That is, you'd rather not have all the CPU + storage capacity taken up by lower priority work if that work tangibly impacted network and disk.

If you've configured the HBase rest and thrift servers with hard coded ports, they are at risk of conflicting for port numbers with other services.

Kafka? If you bring up a kafka instance on another node, all its local data is lost and it has to rebuild it from the last snapshot+stream. This is expensive; cost O(elapsed-time-since-snapshot * event arrival rate). Once rebuilt, performance recovers.

Kafka loves anti-affinity in placement; reduces the no. of instances that die on a node failure, and impact on the system until the rebuild is complete.

Both these apps then are things you want to deploy under YARN, but they have very different scheduling and placement requirements.

Long-lived services in general
  • May be willing to wait for an extended period to come up on the same server as before. That is, even if a server has crashed and is rebooting, or is down for a quick hardware fix —its better to wait before giving up.
  • But they can recover, and ultimately giving up is desireable.
  • Unreliability is not a simple metric of failures, it's failures in a recent time period that matters. That holds for the entire application, as well as distributed components.
  • Can fail in interesting ways.

With slider's goal "support long-lived services in YARN without making you rewrite them", we see that difference and get to address it.

YARN Labels
A key feature is in Hadoop 2.6: labels.  (YARN-796). Admins can give nodes labels; give queues shared/exclusive access to sets of labelled nodes, give users those rights.

Slider picks this up by allowing you to assign different components to different labels. The region servers in a production HBase cluster could all be given the property yarn.label=production;

We're using labels to isolate bits of the cluster for performance. They all share HDFS, so there is some cross-contamination, but IO-heavy analytics work can be kept off the nodes. We'd really like HDFS prioritisation for even better isolation, such as giving shortcut-reads priority over TCP traffic. Future work.

You can also split up a heterogenous cluster, with GPU or SSD labels, more RAM nodes etc. Even if an app isn't coded for label awareness, you can (in the Capacity Scheduler) get different queues to manage labels, so grant different users access to the nodes.

One thing that's interesting to consider is, in an EC2 cluster, labelling nodes as full vs spot-priced. You could place some work on spot-priced nodes, others on full. Not only does this give better guarantees of existence, if HDFS is only running on the full nodes, different performance characteristics. I'd be interested to know of any experiences here.

Placement Escalation.
SLIDER-799 AM to decide when to relax placement policy from specific host to rack/cluster

This  kept me busy in March; a fun piece of code. 

As mentioned earlier, YARN schedulers like to schedule work fast, even if non-local. An am can ask for "do-not-relax" placement, but then there's no relaxation even if a node never comes back.

What we've done is taken the choice about when to relax out of YARN's hands and into the AMs. By doing so, you can specify a time delay in minutes to hours, rather than relying on YARN to find a space and having it back off in a few tens of seconds at most.

This is easier to summarise than go through the details. For the curious, the logic to pick a location is in RoleHistory; escalation in OutstandingRequestTracker. Note that code is all part of our Model; we don't let that directly interact with YARN, which is something for what is controller's task. The model builds up a list of Operations which are then processed afterwards. This really helps testing: we can test the entire model through a mock YARN cluster, taking the actions and simulating their outcome, then add failure events, restarts, etc. Fast tests for fast dev cycles.

Node reliability tracking
We've had this for a while, not with explicit blacklisting but basic greylisting, building up a list of nodes we don't trust and never asking for them explicitly. What's changed is the sophistication of listing and how we react to it.
  1. We differentiate failure types; Node failure counters discard those which are node-independent (example: container memory limits exceeded), and those which are simply pre-emption events. (SLIDER-856).
  2. Similarly, component role failure counters don't count node failures or pre-emption in the reliability statistics of a role.
  3. The counters of role & node failures used for deciding, respectively if an app is failing or a node is unreliable , are reset on a regular, schedule basis (a few hours, tunable).
  4. If we don't trust a node, we don't ask for containers on it, even if is the last place where it ran. (Exception: if you declare that a component placement policy is "strict". It's always asked for again, and there is no escalation).
Reliability tracking should make a difference if a node is playing up. There's a lot more we could do here —we just have to be pragmatic and build things up as we go along.

Where next?

Absolutely key is anti-affinity. I don't see YARN-1042  coming soon —but that's OK. Now we do our own escalation, we can integrate that with anti affinity.

How? ask for a container at a time, blacklisting all those nodes where we've already got an instance. Ramp-up time will be slower, especially taking in to account that escalation may result in container allocations taking minutes before a dead/overloaded node is given up on.

Maybe it could be something like
  1. inital request: blacklist all but those we last ran on.
  2. escalation: relax to all but: nodes those with outstanding requests or allocated containers -or considered too unreliable.
  3. do this in parallel, discarding allocations which assign >1 instance to the same node.
  4. If, after a certain time, nodes are still unallocated, maybe consider relaxing restriction (as usual: configurable policy and timeouts per role) 
(for "nodes", read "nodes within the label set allowed for that role")

I need to think about this a bit more to see if it would work, estimate ramp-up times, etc.

Anything else?

Look at Über-JIRA : placement phase 3 for some ideas.

Otherwise, SLIDER-109, Detect and report application liveness. Agents could report in URLs they've built for liveness probes, either they check themselves or the AM hits them all on a schedule (with monitor threads designed to detect the probes themselves hanging). All the code for this is from the Hadoop 1 HA monitor work I did in 2012; it's checked in waiting to be wired up. All we need it someone to do the wiring. Which is where the fact that slider is an OSS project comes in to play.

  1. All of the implemented features listed here are available to anyone who wants to download and run Slider.
  2. All the new ideas are there for someone to implement. Join the team! Check out the source, enhance it, write those simulation tests, submit patches!
This is not just me being too busy; we need a broader developer community to keep slider going and get it out of incubation. The needs of the many, the code of the many, improves the product for all.

And it's really interesting. I could get distracted putting in time on this. Indeed, SLIDER-856 kept me busy two weekends ago to the extent that I got told off for irresponsible role modelling (parental, not slider codebase). Apparently spending a weekend in front of a monitor is a bad example for a teenage boy. But the placement problem is not just something I find interesting. Read the Borg paper and notice how they call out placement across failure domains and availability zones. They've hit the same problems. Add it to slider and collect real world data and you've got insight into scheduling and placing cluster workloads that even Google would be curious about.

So: come and play.


Build tools as Proof Engines


Someone has put up a thoughtful post on whether you can view make as a proof system. I thought about it and have come up with a conclusion: maybe in the past —but not any more.

As background, it's worth remembering
  1. I did "write the book on Ant", which is a build system in which you declare the transformational functions required to get your application into a state which you consider shippable/deployable, a set of functions which you define in a DAG for the Ant execution engine to generate and order from and then apply to your environment
  2. Early last year, in my spare time, I formally specified the Hadoop FS APIs in Spivey's Z notation, albeit in a Python syntax.
  3. In YARN-913 I went on to specify the desired behaviour of the YARN registry as a TLA+ specification.
And while I don't discuss it much, during my undergraduate work on Formal Specification of and implementation Microprocessors, I was using Gordon's HOL theorem prover. The latter based on Standard ML, just to fend off anyone who believed me when I was claiming not to understand functional programming earlier this week. I didn't meant it, it just entertained the audience. Oh, and did I mention I'm happy to write Prolog too?

This means that (a) I believe I understand about specifications, (b) I vaguely remember what a proof engine is, and (b) I understand how Prolog resolves things, and specifically, why "!" is the most useful operator when you are trying to write applications.

Now I note that the author of that first post, Bob Atkey, does not only has a background of formal logic, SML and possibly even worked with the same people as me, his knowledge is both greater and more up to date than mine. I just have more experience of breaking builds and getting emails from jenkins telling me this.

Now, consider a software project's build
  1. A set of source artifacts, S, containing artifacts s1..sn
  2. A declaration of the build process, B
  3. A set of external dependencies, libraries, L.
  4. A build environment, E., comprising the tools needed for the build, and the computer environment around them, including the filesystem, OS, etc.
The goal is to reach a desired state of a set of redistributables, R, such that you are prepared to ship or deploy them.

The purpose of the build system, then, is to generate R from S through a series of functions applied to (S, L) with tools T within the environment E. The build process description, B, defines or declares that process.

There's many ways to do that; a single line bash file cc *.c && cc *.o could be enough to compile the lexical analyser example from Ch02 of the dragon book.

Builds are more complex than that, which is where tools like make come in.

Make essentially declares that final list of redistributables, R, and a set of transformations from inputs artifacts to output artifacts, including the functions (actions by the tools) to generate the output artifacts from the input artifacts.

The make runtime looks at what artifacts exist, what are missing, and what are out of date somehow builds a chain of operations that are hypothesised to produce a complete set of output artifact whose timestamp in the filesystem is later than that of the source files.

It is interesting to look at it formally, with a rule saying that to derive .o from .c, a function "cc -c" is applied to the source. Make looks at the filesystem for the existence of that .o file, its absence or "out-of-dateness" and, if needed applies the function. If multiple rules are used to derive a sequence of transformations then make will build that chain then execute them.

One interesting question is "how does make build that series of functions, f1..fn, such that:

R = fn(fn-1(fn-2(fn-3..f1(S, L)

I believe it backchains from the redistributes  to build a series of rules which can be applied, then runs those rules forwards to create the final output.

If you view the final redistributables as a set of predicates whose existence is taken as truth, absence as false, and all rules are implies operators the define a path to truth, not falsehood (i.e. we are reasoning over Horn Clauses, then yes, I think you could say "make is a backward chaining system to build a transformation resulting in "truth".

The problem is this: I'm not trying to build something for the sake of building it. I'm trying to build something to ship. I'm running builds and tests repeatedly, from my laptop on my desk, from my laptop in a conference auditorium, a hotel room, and a train doing 190 mph between Brussels and London. A that's all just this week.

Most of the time, those tests have been failing. There are three possible (and non-disjoint) causes of this
  • (a) the null hypothesis: I can't write code that works.
  • (b) a secondary hypothesis: I am better at writing tests to demonstrate the production code is broken than I am at fixing the production code itself.
  • (c) as the external environment changes, so does the outcome of the build process.

Let's pretend that (a) and (b) are false; that I can actually write code that works first time, with the tests intended to show that this condition is not met being well written and correct themselves. Even if such a case held, my build would have been broken for a significant fraction of the time it was this week.

Here's some real examples of fun problems, for "type 3 fun" on the Ordnance Survey Fun Scale.
  1. Build halting as the particular sequence of operations it had chosen depended on maven artifacts which were not only absent, but non-retrievable from a train somewhere under the English Channel.
  2. Connection Reset exceptions talking to an in-VM webapp from within a test case. A test case that worked last week. I never did find the cause of this. Though I eventually concluded that it last worked before I installed a critical OS.X patch (last weeks, not this week's pending one). The obvious action was "reboot the mac' —and lo, it did fix it. I just spent a lot of time on hypotheses (a) and (b) before settling on cause (c)
  3. Tests hanging in the keynote sessions because while my laptop had an internet connection, GET requests against java.sun.com were failing. It turns out that when Jersey starts some servlets up, it tries to do a GET for some XSD file in order to validate some web.xml XML document. If the system is offline, it skips that. But if DNS resolves java.sun.com, then it does a GET and blocks until the document is returned or the GET fails. As as the internet in keynote was a bit overloaded, tests just hung. Fix: edit /etc/hosts to put java.sun.com == or turn off the wifi.
  4. A build at my hotel failing as the run crossed the midnight marker and maven decided to pull down some -SNAPSHOT binaries from mvn central, which were not the binaries I'd just built locally, during the ongoing build and test run.
What do all these have in common? Differences in the environment of the build, primarily networking, except case (4), which was due to the build taking place at a different time from previous builds.

Which brings me to a salient point
The environment of a build includes not only the source files and build rules, it includes the rest of the Internet and the connections to it. furthermore, as the rule engine uses not just the presence/absence of intermediate and final artifacts as triggers for actions, time is an implicit part of the reasoning.

You made could make explict that temporal logic, and have a build tool which look at the timestamp of newly created files in /tmp and flagged up when your filesystem was in a different timezone (Oh, look ant -diagnostics does that! I wonder who wrote that probe?) But it wouldn't be enough, because we've reached a point in which builds are dependent upon state external to even the local machine.

Our builds are therefore, irretrievably nondeterministic.