3 years at Hortonworks!

In 2012 I handed in my notice at HP Laboratories and joined Hortonworks : this May is the third anniversary of my joining the team.



I didn't have to leave HP, and in the corporate labs I has reasonable freedom to work on things I found interesting. Yet is was through those interesting things that we'd discovered Hadoop. Paolo Castagna introduced me to it, as he bubbled with enthusiasm for what he felt was he future of server side computing. At the time I was working on the problem of deploying and managing smaller systems -but doing so in the emergent cloud infrastructures. Hadoop was initially another interesting deployment problem: one designed to scale and cope with failures, yet also built on the assumption of a set of physical hosts, hosts with fixed names and addresses, hosts with persistent storage and whose  failures would be independent. some of the work I did at that time with Julio Guijarro included dynamic Hadoop clusters, Mombasa (the long haul route to see elephants). The work behind the scenes to give Hadoop services more dynamic deployments, HADOOP-3628, earned me Hadoop committership. While the branch was never merged in, the YARN service model shows its heritage.

While we were doing this, HP customers were also discovering Hadoop —and building their clusters. I remember the first email coming in from a sales team who had been asked to quote the terasort performance of their servers: the sales team hadn't heard of a terasort and didn't know what to do. We helped. Before long we were the back-end team on many of the big Hadoop deals, helping define and review the proposed hardware specs, reviewing and sometimes co-authoring the bid responses. And what bids they were! At first I thought a petabyte was a lot of storage —but soon some of the deals were for 10+, 20+ PB. Projects where issues like rack weight and HDD resonance were as key to worry about as power and logistics of getting the servers delivered. Production lines which needed to be block booked for a week or two, but in doing so allowing server customisation: USB ports surplus? Skip them. How many CPU sockets to fill-and with what SKU? Want 2.5" laptop HDDs for bandwidth over 3.5" capacity oriented storage? All arrangeable, with even the option of a week-long burn in and benchmark session as an optional extra. This would show that the system worked as requested, including setting benchmarks for sorting 5+ PB of data that would never be published out of fear of scaring people (bear that in mind when you read blogs posts showing how technology X out-terasorts Hadoop —the really big Hadoop sort numbers are of 10+ PB on real clusters, not EC2 XXL SSD instances, and they don't get published).

These were big projects and it was really fun to be involved.

At the same time though, I felt that HP was missing the opportunity, the big picture. The server group was happy to sell the systems for x86 system margins, other groups to set them up. But where was the storage group? Giving us HPL folk grief for denying them the multi-PB storage deals —even though they lacked a Hadoop story and didn't seem to appreciate the technology. Networking? Doing great stuff for HFT systems where buffering was anathema; delivering systems for the enterprise capable of handling intermittent VM migration. But not systems optimised for sustained full link rate bandwidth, decent buffering and backbone scalability through technologies like TRILL or Shortest Path Bridging (you can get these now, BTW).

The whole Big Data revolution was taking place in front of HP: OSS software enabling massive scale storage and compute systems, the underlying commodity hardware making PB storeable, and the explosion in data sources giving the data to work with. And while HP was building a significant portion of the clusters, it hadn't recognised that this was a revolution. It was reminiscent of the mid 1990s, when the idea of Web Servers was seen as "just another use of a unix workstation".

I left to be part of that Big Data revolution, joining the team I'd got to know through the OSS development, Hortonworks, and so defining the future, rather than despair about HP's apparent failure to recognise change. Many of us from that era left: Audrey and I to Hortonworks, Steve and Scott to RedHat, Castagna to Cloudera. Before I get complaints from Julio and Chris, —yes some of the first generation of Hadoop experts are still there, the company is taking Big Data seriously, and there are now many skilled people working on it. Just not me.

What have I done in those three years? Lots of things! Some of the big ones include:

• Hadoop 1 High Availability. One of the people I worked with at VMWare, Jun Ping, is now a valued colleague of mine.
• OpenStack support: Much of the hadoop-openstack code is mine, particularly the tests.
• The Hadoop FS Specification: defining a Python-like syntax for Spivey's Z notation, delving through the HDFS and Hadoop source to really define what  a Hadoop filesystem is expected to do. From the OpenStack Swift work I'd discovered the unwritten assumptions & set out to define them, then build a test suite to help anyone trying to integrate their FS or object store with Hadoop to get started. This was my little Friday afternoon project; nobody asked me to do it -but now that it is there it's proven invaluable in getting the s3a S3 client working, as well as being one of the first checkpoints for anyone who wants to get Hadoop to work on other filesystems. Arguably that helps filesystem competitors —yet what it is really meant to do is give users a stable underpinning of the filesystem, beyond just the method signatures.
• The YARN-117 service model. I didn't start that work, I just did my best to get the experience of the SmartFrog and HADOOP-3628 service models in there. I do still need to document it better, and get the workflow and service launcher into the core code base; Slider is built around them.
• Hoya: proof of concept YARN application to show that HBase was deployable  as a dynamic YARN application, and initial driver for the YARN-896 services-on-YARN work.
• Apache Slider (incubating). A production quality successor to Hoya, combining the lessons from it with the Ambari agent experience, producing an engine to make many applications deplorable on YARN though a minimal amount of Python code. Slider is integrate with Ambari, but it works standalone against ASF Hadoop 2.6 and the latest CDH 5.4 release (apparently). I've really got a good insight into the problems of placement of work where access to data has to be balanced with failure resilience; enough to write a paper if I felt like it —rather than just a blog post.
• The YARN Service Registry. Again, something I need to explain more. An HA registry service for Hadoop clusters, where static and dynamic applications can be registered and used. Slider depends on it for client applications to find Slider and its deployed services; it is critical for internal bonding in the presence of failures. It's also the first bit of core Hadoop with a formal specification in TLA+.
• Spark on YARN enhancements. SPARK-1537 is my first bit of work there, having the spark history server use the YARN timeline service. Spark internals in Scala, collaboration with the YARN team on REST API definitions and reapplying the test experience of Slider to accompany this with quality tests.
• Recently: some spare time work mentoring S3a: into a production ready state.
• Working with colleagues to help shape our vision of the future of Hadoop. Apache Hadoop is a global OSS project, one which colleagues, competitors and users of the technology collaborate to build. I, like the rest of my colleagues get a say there, helping define where we think it can go: then building it.

The latter is a key one to call out. At HP an inordinate amount of my time was spent trying to argue the case for things like Hadoop inside the company itself, mostly by way of PowerPoint-over-email. I don't have to do that any more. When we make decisions it's s done rapidly,  pulling in the relevant people, rather than the inertial hierarchy of indifference and refusal which I sometimes felt I'd encountered in HP.

Which is why working at Hortonworks is so great: I'm working with great people, on grand projects —yet doing this a process where my code is in people's hands within weeks to months, and where an agile team keeps the company nimble. and pretty much all my work has shipped.

If you look at how the work has included applied formal methods, distributed testing, models of system failure and dynamic service deployment, I'm combining production software development with higher level work that is no different than what I was doing in a corporate R&D lab -except with shipping code.

Hortonworks is hiring. If what I've been up to -and how I've been doing it- sounds exciting: then get in touch. That particularly applies to my former HPL colleagues, who have to make their mind up where to go: ink vs enterprise. There is another option: us.

It's OK to submit patches without tests: just show the correctness proofs instead

In a window adjacent to the browser I'm typing this, I'm doing a clean build of trunk prior to adding a two line patch to Hadoop -and the 20+ lines needed to test that patch; Intellij burbles away for 5 minutes as it does on any changes to the POMs on a project which has a significant of the hadoop stack loaded.

The patch I'm going write is to fix a bug introduced by a previous three line patch, one that didn't come with patches because it was "trivial".

It may have been a trivial patch, but it was a broken trivial patch. It's not so much that the test cases would have found this, but they'd have forced the author to think more about the inputs to the two-line method, and what outputs would be expected. Then we get some tests that generate the desired outputs for the different outputs, ones that guarantee that over time the behaviour is constant.

Instead we have a colleague spending a day trying to track down a remote functional test run, one that has been reduced to a multi-hop stack trace problem. The big functional test suites did find that bug (good), but because the cost of debugging and isolating that failure is higher handling that failure is more expensive.; With better distributed test tooling, especially log aggregation and analysis, that cost should be lower —but it's still pretty needless for something that could have been avoided simply by thinking through the inputs.

Complex functional system tests should not be used as a substitute for unit tests on isolated bits of code. 

I'm not going to highlight the issue, or identify who wrote that patch, because it's not fair: it could be any of us, and I am just as guilty of submitting "trivial" patches. If something is 1-2 lines long, it's really hard to justify in your brain the effort of writing the more complex tests to go with it.


If the code actually works as intended, you've saved time and all is well. But if it doesn't, that failure shows up later in full stack tests (cost & time), the field (very expensive), and either way ends up being fixed the way the original could have been done.

And as documented many times before: it's that thinking about inputs & outputs that forces you to write good code.

Anyway: I have tests to write now, before turning on to what is the kind of problem where those functional tests are justified, such as Jersey client not working on Java 8. (I can replicate that in a unit test, but only in my windows server/java 8 VM)

In future, if I see anyone use "trivial patch" as a reason to not write tests, I'll be wheeling out the -1 veto.

I do however, offer an exception: if people can prove their code works, I'll be happy

(photo: wall in Brussels)

Distributed System Testing: where now, where next?

Confluent have announced they are looking for someone to work on an open source framework for distributed system testing.

I am really glad that they are sitting down to do this. Indeed, I've thought about sitting down to do it myself, the main reason I've been inactive there is "too much other stuff to do".



Distributed System Testing is the unspoken problem of Distributed Computing. In single-host applications, all you need to do is show that the application "works" on the target system, with its OS,  enviroment (timezone, locale, ...), installed dependencies and application configuration.

In modern distributed computing you need to show that the distributed application works across a set of machines, in the presence of failures.

Equally importantly: when your tests fail, you need the ability to determine why they failed.

I think there is much scope to improve here, as well as the fundamental problem: defining works in the context of distributed computing.

I should write a post on that in future. For now, my current stance is: we need stricter specification of desired observable behaviour and implementation details. While I have been doing some Formal Specification work within the Hadoop codebase, there's a lot more work to be done there —and I can't do it all myself.

Assuming that there is a good specification of behaviour, you can then go on to defining tests which observe the state of the system, within the configuration space of the system (now including multiple hosts and the network), during a time period in which failures occur. The observable state should continue to match the specification, and if not, you want get the logs to determine why not. Note here that "observed state" can be pretty broad, and includes

• Correct processing of requests
• The ability to serialize an incoming stream of requests from multiple clients (or at least, to not demonstrate non-serialized behaviour)
• Time to execute operations is one (performance),
• Ability to support the desired request rate (scalability)
• Persistence of state, where appropriate
• Reslience to failures of : dependent services, network, hosts, 
• Reporting of detected failure conditions to users and machines (operations needs)
• Ideally: ability to continue in the presence of byzantine failures. Or at least detect them and recover.
• Ability to interact with different versions of software (clients, servers, peers)
• Maybe: ability to interact with different implementations of the same protocol.

I've written some slides on this topic, way back in 2006, Distributed Testing with SmartFrog. There's even a sub-VGA video to go with it from the 2006 Google Test Automation Conference.

My stance there was

1. Tests themselves can be viewed as part of a larger distributed system
2. They can be deployed with your automated deployment tools, bonded to the deployed system via the configuration management infrastructure
3. You can use the existing unit test runners as a gateway to these tests, but reporting and logging needs to be improved.
4. Data analysis is a critical area to be worked on.

I didn't look at system failures, I don't think I was worry enough about that, showing we weren't deploying things at scale, and before cloud computing took failures mainstream. Nowadays nobody can avoid thinking about VM loss at the very least.

Given I did those slides nine years ago, have things improved? Not much, no

• Test runners are still all generating the Ant XML test reports written along with the matching XSLT transforms up by Stephane Balliez in 2000/2001
• Continuous Integration servers have got a lot better, but even Jenkins, wonderful as it is, presents results as if they were independent builds, rather than a matrix of (app, environment, time). We may get individual build happiness, but we don't get reports  by test, showing that Hadoop/TestATSIntegrationFailures is working intermittently on all debian systems -but has been reliable elsewhere. The data is all there, but the reporting isn't.
• Part of the problem is that they are still working with that XML format, one that, due to its use of XML attributes to summarise the run, buffers things in memory until the test test case finishes, then writes out the results. stdout and stderr may get reported -but only for the test client, and even then, there's no awareness of the structure of log messages
• Failure conditions aren't usually being explicitly generated. Sometimes they happen, but then its complaints about the build or the target host being broken.
• Email reports from the CI tooling is also pretty terse. You may get the "build broken, test XYZ with commits N1-N2", but again, you can get one per build, rather than a summary of overall system health.
• With a large dependent graph of applications (hello, Hadoop stack!), there's a lot of regression testing that needs to take place —and fault tracking when something downstream fails. 
• Those big system tests generate many, many logs, but they are often really hard to debug. If you haven't spent time with 3+ windows trying to sync up log events, you've not been doing test runs.
• In a VM world, those VMs are often gone by the time you get told there's a problem.
• Then there's the extended life test runs, the ones where we have to run things for a few days with Kerberos tokens set to expire hourly, while a set of clients generate realistic loads and random servers get restarted.

Things have got harder: bigger systems, more failure modes, a whole stack of applications —yet testing hasn't kept up.

In slider I did sit down to do something that would work within the constraints of the current test runner infrastructure yet still let us do functional tests against remote Hadoop clusters of variable size . Our functional test suite, funtests, uses Apache Bigtop's script launcher to start  Slider via its shell/py scripts. This tests those scripts on all test platforms (though it turns out, not enough locales), and forces us to have a meaningful set of exit codes —enough to distinguish the desired failure conditions from unexpected ones. Those tests can deploy slider applications on secure/insecure clusters (I keep my VM configs on github, for the curious), deploy test containers for basic operations, upgrade test, failure handling tests. For failure generation our IPC protocol includes messages to kill a container, and to have the AM kill itself with a chosen exit code.

For testing slider-deployed HBase and accumulo we go one step further. Slider deploys the application, and we run the normal application functional test suites with slider set up to generate failures.

How do we do that? With the Slider Integral Chaos Monkey. That's something which can run in the AM, and, at a configured interval, roll some virtual dice to see if the enabled failure events should be triggered: currently container and AM (we make sure the AM isn't chosen in the container kill monkey action, and have a startup delay to let the test runs settle in before starting to react).

Does it work? Yes. Which is good, because if things don't work, we've got the logs of all the machines in the cluster that ran slider to go through. Ideally, YARN-aggregated logs would suffice, but not if there's something up between YARN and the OS.

So: test runner I'm happy with. Remote deployment, failure injection, both structured and random. Launchable from my deskop and CI tooling; tests can be designed to scale. For testing rolling upgrades (Slider 0.80-incubating feature), we run the same client app while upgrading the system. Again: silence is golden.

Where I think much work needs to be done is what I've mentioned before: the reporting of problems and the tooling to determine why a test has failed.

We have the underlying infrastructure to stream logs out to things like HDFS or other services, there's nothing to stop us writing code to collect and aggregate those -with the recipient using the order of arrival to place an approximate time on events (not a perfect order, obviously, but better than log events with clocks that are wrong). We can collect those entire test run histories, along with as much environment information that we could grab and preserve. Junit: system properties. My ideal: VM snapshots & virtual network configs.

Then we'd go beyond XSLT reports of test runs and go to modern big data analysis tools. I'm going to propose here: Spark Why? so you can do local scripts, things in Jenkins & JUnit, and larger bulk operations. And for that get-your-hands-dirty test-debug festival, I can use a notebook like Apache Zepplin (incubating) can then be a no

we should be using our analysis tools for the automated analysis and reporting of test runs, the data science tooling for the debugging process.

Like I said, I'm not going to do all this. I will point to a lovely bit of code by Andrew Or @ databricks, spark-test-failures. which gets Jenkins's JSON-formatted test run history, determines flaky tests and posts the results on google docs. That's just a hint of what is possible —yet it shows the path forwards.

(Photo: "crayola", St Pauls:work commissioned by the residents)

Dynamic Datacentre Applications: The placement problem

We're just in the wrap-up for Slider-0.80-incubating; A windows VM nearby is busy downloading the source .zip file and verifying that it builds & tests on windows.

Before people feel sorry for me consider this: given a choice between windows C++ code and debugging Kerberos,  I think I'd rather bring up MSDN in a copy of IE6 while playing edit-and-continue games in Visual Studio than try and debug the obscure error messages you get with kerberos on Java. Not only are they obscure, my latest Java 7 update event changed the text of the error which means "now we've upgraded the JVM you don't have the encryption settings to use kerberos".

Which is a shame, because I have to make sure everything works with kerberos.

Anyway: build works and the tests are running —which keeps me happy. If all goes to plan, Slider 0.80-incubating will be there for download within a week.

Some of the new features include

• SLIDER-780: Ability to deploy docker packages
• SLIDER-663 zero-package cluster definition. (It has a different name, but it essentially means "no need to build a redistributable zip file for each application). While the zip-based distribution is essential for things you want to share, for things you are developing or using yourself, this is lighter weight.
• SLIDER-733 Ability to install a package on top of another one. This addresses "the coprocessor problem": how to add a new HBase coprocessor JAR without rebuilding everything. And, with SLIDER-633, you can define that new package easily,

Notably, all these features are by people other than me —specifically, by colleagues.

 

As anyone who knows me will realise: that's because my Hortonworks colleagues are really great people who know more about computing than I ever have or will and are better at applying that knowledge than myself —someone who uses "test driven development" as a way of hiding his inability to get anything to work right first time.

And yet these people still allow me to work with them —showing there still things that they consider something I'm still up to handling.

What have I been up to? Placement, specifically SLIDER-611: Über-JIRA - placement phase 2

Right from the outset, Hadoop HDFS and MR have been built on some assumptions about failure, availability and bandwidth

1. Disks will fail: deal with it by using replication over 2h replacement part support contracts and RAID-array rebuilding
2. Servers will fail: app developers have to plan for that.
3. Some servers are just unreliable; apps may work this out without even needing to be told. Example: stragglers during a map identifies servers whose disks may be in trouble.
4. If an application fails a few times it's unreliable and should be killed.
5. The placement/scheduling of work is primarily an optimisation to conserve bandwidth. That is: you place work for proximity to the desired data.
6. If the desired placement cannot be obtained, placement elsewhere is usually acceptable.
7. It's better to have work come up fast somewhere near the data than wait for minutes for it to potentially come up on the actual node requested.
8. If work is placed on a different machine, the cost to other apps (i.e. the opportunity cost) is acceptable.
9. It's OK if two containers get allocated to the same machines ("affinity is not considered harmful")
10. The actions of one app running on a node are isolated from the others (e.g. CPU throttling and vmem limiting is sufficient to limit resource conflict).
11. Work has a finite duration, and containers know when they are finished. Nothing else needs to make that decision for them.

For analytics works where the data is known, the placement strategy is ideal. It ensures that work comes up fast at the expense of locality: bandwidth is precious, but time even more so. a nearby-placement will run the work at a cost of network capacity and the implicit risk of locality placement for other work trying to run on the (now in use) node.

But what about long lived services, such as HBase and Kafka?

HBase uses short-circuit reads for maximum performance working with local data. Although region servers don't have to be co-located with the data, until there's a full Hbase compaction, most of the data for an RS is likely to be remote. (for each Block b, P(data-local) = f(nodes/3), (roughly; the 3-replica-2-rack policy complicates the equation as blocks are not spread completely randomly).

Therefore: restarting on the wrong node can slow down that region server's performance for an extended period of time.

HBase is often used low-latency apps; if other things are running there then it can impact performance. That is, you'd rather not have all the CPU + storage capacity taken up by lower priority work if that work tangibly impacted network and disk.

If you've configured the HBase rest and thrift servers with hard coded ports, they are at risk of conflicting for port numbers with other services.


Kafka? If you bring up a kafka instance on another node, all its local data is lost and it has to rebuild it from the last snapshot+stream. This is expensive; cost O(elapsed-time-since-snapshot * event arrival rate). Once rebuilt, performance recovers.

Kafka loves anti-affinity in placement; reduces the no. of instances that die on a node failure, and impact on the system until the rebuild is complete.

Both these apps then are things you want to deploy under YARN, but they have very different scheduling and placement requirements.

Long-lived services in general

• May be willing to wait for an extended period to come up on the same server as before. That is, even if a server has crashed and is rebooting, or is down for a quick hardware fix —its better to wait before giving up.
• But they can recover, and ultimately giving up is desireable.
• Unreliability is not a simple metric of failures, it's failures in a recent time period that matters. That holds for the entire application, as well as distributed components.
• Can fail in interesting ways.

With slider's goal "support long-lived services in YARN without making you rewrite them", we see that difference and get to address it.

YARN Labels
A key feature is in Hadoop 2.6: labels.  (YARN-796). Admins can give nodes labels; give queues shared/exclusive access to sets of labelled nodes, give users those rights.

Slider picks this up by allowing you to assign different components to different labels. The region servers in a production HBase cluster could all be given the property yarn.label=production;

We're using labels to isolate bits of the cluster for performance. They all share HDFS, so there is some cross-contamination, but IO-heavy analytics work can be kept off the nodes. We'd really like HDFS prioritisation for even better isolation, such as giving shortcut-reads priority over TCP traffic. Future work.

You can also split up a heterogenous cluster, with GPU or SSD labels, more RAM nodes etc. Even if an app isn't coded for label awareness, you can (in the Capacity Scheduler) get different queues to manage labels, so grant different users access to the nodes.

One thing that's interesting to consider is, in an EC2 cluster, labelling nodes as full vs spot-priced. You could place some work on spot-priced nodes, others on full. Not only does this give better guarantees of existence, if HDFS is only running on the full nodes, different performance characteristics. I'd be interested to know of any experiences here.

Placement Escalation.
SLIDER-799 AM to decide when to relax placement policy from specific host to rack/cluster

This  kept me busy in March; a fun piece of code. 

As mentioned earlier, YARN schedulers like to schedule work fast, even if non-local. An am can ask for "do-not-relax" placement, but then there's no relaxation even if a node never comes back.

What we've done is taken the choice about when to relax out of YARN's hands and into the AMs. By doing so, you can specify a time delay in minutes to hours, rather than relying on YARN to find a space and having it back off in a few tens of seconds at most.

This is easier to summarise than go through the details. For the curious, the logic to pick a location is in RoleHistory; escalation in OutstandingRequestTracker. Note that code is all part of our Model; we don't let that directly interact with YARN, which is something for what is controller's task. The model builds up a list of Operations which are then processed afterwards. This really helps testing: we can test the entire model through a mock YARN cluster, taking the actions and simulating their outcome, then add failure events, restarts, etc. Fast tests for fast dev cycles.

Node reliability tracking
We've had this for a while, not with explicit blacklisting but basic greylisting, building up a list of nodes we don't trust and never asking for them explicitly. What's changed is the sophistication of listing and how we react to it.

1. We differentiate failure types; Node failure counters discard those which are node-independent (example: container memory limits exceeded), and those which are simply pre-emption events. (SLIDER-856).
2. Similarly, component role failure counters don't count node failures or pre-emption in the reliability statistics of a role.
3. The counters of role & node failures used for deciding, respectively if an app is failing or a node is unreliable , are reset on a regular, schedule basis (a few hours, tunable).
4. If we don't trust a node, we don't ask for containers on it, even if is the last place where it ran. (Exception: if you declare that a component placement policy is "strict". It's always asked for again, and there is no escalation).

Reliability tracking should make a difference if a node is playing up. There's a lot more we could do here —we just have to be pragmatic and build things up as we go along.

Where next?

Absolutely key is anti-affinity. I don't see YARN-1042  coming soon —but that's OK. Now we do our own escalation, we can integrate that with anti affinity.

How? ask for a container at a time, blacklisting all those nodes where we've already got an instance. Ramp-up time will be slower, especially taking in to account that escalation may result in container allocations taking minutes before a dead/overloaded node is given up on.

Maybe it could be something like

1. inital request: blacklist all but those we last ran on.
2. escalation: relax to all but: nodes those with outstanding requests or allocated containers -or considered too unreliable.
3. do this in parallel, discarding allocations which assign >1 instance to the same node.
4. If, after a certain time, nodes are still unallocated, maybe consider relaxing restriction (as usual: configurable policy and timeouts per role) 

(for "nodes", read "nodes within the label set allowed for that role")

I need to think about this a bit more to see if it would work, estimate ramp-up times, etc.

Anything else?

Look at Über-JIRA : placement phase 3 for some ideas.

Otherwise, SLIDER-109, Detect and report application liveness. Agents could report in URLs they've built for liveness probes, either they check themselves or the AM hits them all on a schedule (with monitor threads designed to detect the probes themselves hanging). All the code for this is from the Hadoop 1 HA monitor work I did in 2012; it's checked in waiting to be wired up. All we need it someone to do the wiring. Which is where the fact that slider is an OSS project comes in to play.

1. All of the implemented features listed here are available to anyone who wants to download and run Slider.
2. All the new ideas are there for someone to implement. Join the team! Check out the source, enhance it, write those simulation tests, submit patches!

This is not just me being too busy; we need a broader developer community to keep slider going and get it out of incubation. The needs of the many, the code of the many, improves the product for all.

And it's really interesting. I could get distracted putting in time on this. Indeed, SLIDER-856 kept me busy two weekends ago to the extent that I got told off for irresponsible role modelling (parental, not slider codebase). Apparently spending a weekend in front of a monitor is a bad example for a teenage boy. But the placement problem is not just something I find interesting. Read the Borg paper and notice how they call out placement across failure domains and availability zones. They've hit the same problems. Add it to slider and collect real world data and you've got insight into scheduling and placing cluster workloads that even Google would be curious about.

So: come and play.