I for one will not defend our nation's critical infrastructure home wifi base stations

Apparently the UK government is planning to spend lots of money building a "Cyber National Guard", which means that a "reserve guard" of civilians will be available on call when the military needs them to organise "cyber strikes" against enemies or defend the national infrastructure.

Or as the Daily Mail (remember, it's not a real paper and has a history of supporting fascism) says:

A new ‘Cyber National Guard’ of part-time reservists will be open to computer whizzkids who cannot pass the current Territorial Army fitness tests, on the basis that press-ups do not aid computer skills. ‘A TA for computer geniuses’, as Mr Hammond called it.

He poured scorn on ‘crude and bonkers attacks by armchair generals’ who have criticised him for cutting the number of soldiers – and made it clear conventional forces faced more cuts in the switch.

In theory I meet some of the criteria, even if I am fitter than the sneering prejudices of the Daily Mail would think.

• I know a lot about computers
• I was the first person ever to publicly demonstrate a signed ActiveX-in-IE security hole.I know how to use WireShark to read HTTP and TCP conversations.
• I used to be able to read x86 ASM (there was some in the film Elysium, entertainingly enough)
• 1. I'm good at breaking things, including large Distributed Systems.
• I'm down as co-author on a patent on how to use a subverted battery to talk to a subverted BIOS to get all your keystrokes without the OS getting a lookin

However, this does not make me suitable for some national-cyber-guard-thingy because if you look at the one documented instance of a nation state committing a (peacetime) over-the-net attack on another nation state, Olympic Games, it's clear that this project took person-years of effort to come up with a virus so subtle it could make use of multiple 0-day exploits to get into Windows, then trickle over to the SCADA-managed industrial machinery by way of USB sticks that were neither near empty or near full (to make the loss in capacity less obvious). Once there, recognise the characteristics of an iranian enrichment centrifuge, change their spin rates to destroy them -all the while reporting valid parameters to the ops team. That's not a activity of some weekend developers: That is the R&D spend of a goverment, the integration of gained knowledge of the Iranian enrichment process, the ability to write code to destroy it -the testing of that on real hardware, and the transport mechanism using 0-day exploits and a forged signing certificate. That is what the future of inter-nation-state conflict over the net looks like, and it doesn't depend on script-kiddies running metasploit.

The classic reserve forces model trains people, part time, to be soldiers, a process which roughly consists of

1. learning how to avoid getting killed.
2. learning how to shoot at things and people
3. learning how to follow orders even when the outcome involves a non-0 zero probability of becoming a KSI statistic.
4. learning how to use skills 1 & 2 to achieve goals the person shouting the orders wants to achieve.

That training and learning scales well, as shown in the twentieth century by two global conflicts and the Korean war. Teaching people how to code malware to infiltrate and damage other government's national and commercial infrastructure does not. What does that leave? Botnets are designed to be O(1) scaling, so you don't need regiments of engineers there. Unless it is just script-kiddie work rummaging around opposing computing facilities -but that's something best done in peacetime, to a relaxed schedule (which is presumably why the Chinese govt. do appear to have associates doing that).

As for myself, unless I am needed to write JUnit tests to break the North Korean missile program, well, I'm not useful.

Maybe, therefore, it's not military attack the army wants, it's defending the nation's critical infrastructure.

Which is what exactly?

Because if its the network, then what you need is not on skills in things like configuring telco-scale switches, it's having that network set up with as much intrusion and anomaly detection as possible, with the people on call to handle the support calls. You aren't going to be able keep part time computer people around to field calls on that if all they know about network security is that turning off UPNP on the home router is a good idea.

No, network management at the government scale is a skill for the few, not the many. That doesn't mean that those people who do have to look after corporate and national intranets shouldn't be up to date with current tools and thinking w.r.t. defending critical network security from nation states, of which a key one is don't buy routers from a company owned by the Chinese Army.

Beyond the network, well there's the many desktops out there. I can just about defend my home set of machines by Romanian Botnet gangs, primarily by disabling most browser plugins, updating flash weekly and not running code I don't trust. I also have to defend my home passwords from an 11 year old. Neither skill will give me a chance to defend my machines against a nation state, not unless the attack by the state in question involves a small boy looking over your shoulder as you type in the password to give him extra time on the machine. In that case -and only in that case- would my advice -use some uppercase characters and learn to touch type- would be of use.

But going round locking down PCs by deleting security risks like Acroread? Enforcing password policies? These are IT dept things, not something for a team of reservist-cyber-warriors. Even there though, the threat posed by foreign governments sending spear-phished PPT documents containing ActiveX controls with 0-day exploits in them (ActiveX is derived from Ole Control Extensions which is built on top of Object Linking and Embedding, all atop the COM common object model). Once the unsuspecting slideware gets opened, whatever payload it goes on to download.


Where does that leave? If the outer network is the netops, the deskopt the PC IT dept, what's left? The applications.

It means designing and building applications that don't let you steal millons if you attach a KVM switch to an employee's desktop.

It means designing databases apps so that SQL injection attacks never work, irrespective of how the input comes in, and validation of that data at entry time, in flight and when it is stored in the database -so that corruption is detected.

It means having some way of backing up application state securely, so that if damage is done to it, then you can recover.

It means thinking about security in a more complex way than generic "user" -with different levels of access to different things, and it means having a defensible audit trail so that if someone were to download large quantities of an organisations files and stick them up
on wikileaks or hand them to a paper -at least you know what's been taken.

From that perspective, people like myself are more generally useful if we do actually make the things we code as securely as possible, and put in audit trails on the off chance that we can't. And it implies that the government would be better of spending money teaching us to understand Kerberos as well as other aspects of secure computing, rather than having some pool of script-kiddies whose skill stops at metaspoit and wireshark, and whose networking understanding stops at home routers. Of course there the problem lies that some of the course matter needed to keep those foreign nation states at bay, "don't trust HTTPS or SSH with 1024 bit keys" may start the audience asking questions you don't want asked.

Overall, these proposals -as presented in the newspapers- appear to be naively unrealistic and based on a complete misunderstanding of how you can both attack and defend core computing infrastructures -as well as who is capable and responsible for doing so.

Why then do they appear?

I can see a number of possibilities

1. Null Hypothesis: Politican is clueless, and said things they don't understand;
2. Politician reported someting realistic, but reporter clueless and reached a conclusion that makes no sense
   
3. Both Politican and Reporter clueless leading to a complete misunderstanding, where the reporter didn't realise
4. the politician was talking bollocks and then exaggerated even that
5. Politician knew that what he was talking about was utterly unrealistic, but did it in an attempt to distract the audience and justify other actions

Given the structure of the talk -to offset cuts in the core "shoot things, achieve goals ordered to achieve, come back alive" bit of the army and reserve forces -he's probably doing #5: divert and justify those actions. But the scary thing is: he may actually believe what he's been saying.

How to update an ubuntu 12.x box to protoc 2.5.0 and so build Apache hadoop 2.x+

Here are some working notes on how to push up an Ubuntu 12.x machine to a version of protoc that builds Hadoop. It is a shame that protobuf.jar doesn't include a protoc compiler
Actually it's also a shame that protobuf.jar isn't backwards compatible across versions. I'm
now ooking at Apache Thrift and thinking "maybe I should adopt that in my own code".


Run these as root:

cd ~
apt-get -y remove protobuf-compiler
curl -# -O https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
gunzip protobuf-2.5.0.tar.gz 
tar -xvf protobuf-2.5.0.tar 
cd protobuf-2.5.0
./configure --prefix=/usr
make
make install
protoc --version

Then log in as you and again run

protoc --version

The output of the --version commands must be "libprotoc 2.5.0". Any error; any different version: your build of Hadoop won't.

Hoya: coding and testing

I've been back in the UK this month, and am busy working on Hoya. The key activity right now is adding support for Apache Accumulo. This is because it forces me to make the changes sooner rather than later to go multi-application; to make Hoya more generic.

This means I've had to go from a simple model of (master, worker+) to a multi-role world one of (master, tablets+, [monitor], [gc]).

A two-role model is straightforward: the AM forks off the master, while all the other containers are automatically workers. Compare the no. of containers you have to the desired number, and request/release containers. When you get given containers, or one fails, you know its always a worker, so update the numbers. Flexing is achieved by having the client specify a new count of workers. Simple.

Multi-role means that not only do you have to have more complex data structures, you need to map containers and container requests to specific roles. That means when the RM notifies the AM of some new container assignments, the AM has to decide what they are for. The trick there -as suggested by one of my colleagues who I shall not name- is to use the priority:int field in the request, mapping that to role numbers. The AM then needs to spin up the relevant launcher for that role, and record details about the container so that on any failure notification it knows which role's node count to decrease. The AM then needs to scan all the role set to decide whether the current state is good, and how to react if not. There's also the detail of configuration; you have to move from simple options for, say, --workers 4 --workerheap 512M, to per role numbers and  options: --role worker 4  --roleopt worker jvm.heapsize 512M -role monitor 1 --roleopt monitor jvm.heapsize 1G

This gets turned into a set of role option maps, which have to be used to configure the specific roles.

I've also had to change how the client flexed the cluster size -from a simple int value to a whole new JSON cluster specification. Which leads to an interesting trick: you can not only tune the size of a cluster by role -you can change parameters like heapsize as you go.

Multi-role then: lots more complexity in managing things.

The other bit of complexity is of course deploying different application types. Here I've introduced the notion of a Provider; something that provides client-side setup services: build initial json spec, preflight checking of specifications, helping to set up the AM with extra resources, and patching configuration directories.

Server side, these providers now need to help manage the roles, with in-VM startup of the master process, and launcher thread startup of all the remote processes. This is still ongoing work. I did have accumulo execing, though its startup script needs to take the env variables of HADOOP_PREFIX and ZOOKEEPER_PREFIX as a preamble to its own classloading, which is something I may have to bypass before long on the basis it is trying to be too clever. But bring up the Accumulo master and the first thing it says is "init me". I may patch accumulo to add a special "--init" option here, but to keep the #of changes down, and support similar use cases, I've instead gone from a single executable to the ability to run a series of programs in the AMs container. And how have I ended up implementing that? With YARN services.

I've added a new YARN service, ForkedProcessService, which executes a native process when started, stopping itself when that process finishes -and converting a non-zero exit code into service failure with an ExitException with the relevant exit code. Any YARN service that can host children -of which the Hoya AM is one- can create one of these services, add it as a child, configure it and start it running. Registering as a service lifecycle listener lets this parent service get told when the child has finished, and can react to it failing and succeeding.

That works for HBase, with one process "bin/hbase master", but not for Accumulo's potential workflow of [accumulo init, accumulo master]. For that I've had to do another service, SequenceService, which runs its children in sequential order, only starting one when the previous one finishes successfully. Then I have my Provider Services extend that, to get the ability to exec a sequence of commands.

List<String> commands =
  buildProcessCommand(cd, confDir, env, masterCommand);

ForkedProcessService masterProcess = buildProcess(getName(), env, commands);
CompoundService compound = new CompoundService(getName());
compound.addService(masterProcess);
compound.addService(new EventNotifyingService(execInProgress,
                         cd.getOptionInt(
                           OPTION_CONTAINER_STARTUP_DELAY,
                           CONTAINER_STARTUP_DELAY)));
addService(compound);

   
The AM can now deploy a provider's in-AM role by telling the provider service to start() and let it sort the details out for itself. Which is pretty slick: it's services through Hoya, out-of-VM processes for the real work. To achieve this I've had to add two new container services, SequenceService: runs a list of services in order, failing when any of the child services file, stopping when they are all done. CompoundService runs a set of services in parallel, again propagating failures. Once all children have stopped, this parent service does too.

Comparing it to SmartFrog's workflow components, there are obvious similarities, though without the issues that component always had -because child services were potentially remote and only accessible via RMI, you couldn't interact with a stopped service, while the deployment model mostly mandated that everything was done declaratively at deploy time. In Hoya all the wiring up of services is done in-code; there's no attempt to be more declarative, and with every service running in-process, keeping an eye on them is much easier. That said, I'm having to do things like pass in callback interfaces -the execInProgress interface is implemented by the AM to let it know that the master process is going live. in SF I'd have stuck a reference to an exported RMI interface into the distributed component description graph, and let this workflow service pull it down. Still, here we avoid RMI -which is not a bad thing.

The other difference is that I am not trying to run any of these service components on any node in the cluster other than the single HoyaApplication Master. The goal is to only run on each allocated container the final program Hoya is managing, not any Hoya code at all. YARN does the work of getting the binaries over, starting the process and reporting failure. That assumption of a failure-aware execution layer atop a shared filesystem, with Zookeeper to provide consistent distributed state, means that I can delegate all those details to Hadoop.

Furthermore, I'm only trying to deploy applications that live in a Hadoop cluster -use HDFS for persistent state over a local filesystem, dynamically locate other parts of the application via ZK, and don't treat unannounced termination of a process or loss of a node as a disaster, more a mild inconvenience. This is precisely what today's classic Enterprise Java & Spring Server apps don't do: they use databases, message queues and often build-time URLs -if not build time, then certainly static for the lifespan of the program.

Handling those applications is a pain, because your deployment framework has to choreograph startup, wiring together distributed components of the system from dynamically built up hostnames and URLs of other component locations, try to execute the (expected) shutdown routines rather than just kill the processes, and, hardest of all, deal with the persistent data problem. Usually that's addressed by having a shared database, with the MQ to route stuff to services

Testing YARN applications

One thing I should discuss is the challenge of testing YARN applications, which I've now documented a bit on the wiki

1. MiniYARNCluster is your friend
2. MiniHDFS cluster loses logs when torn down, but is handy to keep you honest -make sure some tests use it for YARN deployment
3. Linux VMs are more relalistic than an OS/X desktop, though there is the overhead of getting protoc updated. Once you've done it -snapshot it.

I now have a VM running in Rackspace Cloud that exists to do the Linux side D/L and test of Hoya and other Hadoop things -I just push stuff to github when I need to, and (currently manually) trigger a git pull and rebuild. The thought of automating all of that and going to jenkins has occurred to me -its the hassle of setting up the network proxies that puts me off -and the challenge of getting all that log data that doesn't get merged in with the the Log4J output.

I have thought of how to merge in the log output of the Hoya JVMs running in YARN containers. These JVMs generate data using the commons-logging API (Hadoop-internal) and SLF4J (Hoya), both of which are backed by Log4J. All I need to do is get that Log4J data merged with the output of the Junit JVM.

Merge post-test-run

Output all log4j records in the JSON format our little JSON log emitter then merge the VM results back in with the JUnit JVM. The clocks on all the logs will be consistent (it's the same host/VM), so they can be strictly ordered (I often end up using timestamps to compare what's going on in the RM & Test runner with the AM). The output would then need to be reworked so that the log entries are presented in a human readable format, instead of one JSON record per (long) line, with a process identifier included on each line.

Stream live to the test runner
Use a Log4J back end that forwards log events to a remote process, perhaps the test runner JVM itself. The collector would then output the Log events intermixed with the local events. Again, we need to differentiate the processes.

Back when I wrote the SmartFrog distributed xUnit test runner (JUnit3, SFUnit, ...), we did forward test results over the wire to a collector service, which could then aggregate test results from processes - similar to the work in [Duarte06]  . In this design we were running tests against multiple remote implementations, showing the results per test, rather than per implementation, so highlighting where things behaved differently. We also streamed out the results as XHTML, emitting the summary record at the end, rather than trying to buffer everything in the JVM until the end of the run. That in-JVM buffering is needed so as the classic Ant XML format sticks the summary in the attributes of the XML document. As a result, if the JVM dies: no output. If the test run generates too much log data: OOM and no output. Streaming XHTML means that you don't OOM from lots of log data, and if the process dies, your browser can still make a lot of the details. Oh, and with the right style sheet the results were directly viewable in the browser without the overhead of the XSL transform (though that always helps with summary data).

What, then, would I like? But which I don't have time to implement.

1. Test runner to collect log events from multiple processes
2. Test runner to stream these events out to a file, preserving log level and adding host information
3. Test runner to print to stdout some feedback on what is going on so that someone sitting in front of a screen can see what is happening. This should be in a format for humans
4. Something to take the streamed output and generate the traditional XML format for the benefit of Jenkins and other CI tools
5. Something to take that streamed output and generate reports that can intermix the log events from different processes, and, by virtue of the retained log level data, let me show/hide different log levels, so that I can say "show the warnings", leaving info and debug hidden.

This would be quite an undertaking: I actually wonder if you could have some fun with the Kafka back end for Log4J and doing the analysis in Samza. That would be a bit of overhead on a single desktop though -but it could scale well to production tests. The alternative is have the Test Runner open some network port (RESTy/Post over Hadoop RPC), with log4J posting them in direct, runner to output to file as the JSON records; push them to screen using human-log4j, abusing the thread-ID field to identify remote processes. Then add some post-test run code for the generation of classic XML output, as well as something designed to display multi-process, multi-host test results better. Ideally, have Jenkins display that too.

Like I said: no time. But maybe next year I'll mentor a Google Summer of Code project to do this -it's a great cross-Apache bit of work, with Jenkins and possibly Junit & TestNG in the mix. Target Hadoop, Bigtop and other bits of the Hadoop stack. Yes, a nice project: volunteers with free time invited.

[Photo: somewhere in SF's Mission District: like Stokes Croft except with high-end furniture shops and the graffiti is mostly on the side streets, rather than integrated with the shops]

HDP-2.0 Beta

Hot on the heels of the Apache Hadoop 2.1 beta comes the beta of Hortonworks Data Platform 2.0, which is uses those Hadoop 2.1 binaries as the foundation of the platform.




HBase 0.96 is in there, which is the protobuf 2.5-compatible version, as is Hadoop 2.1-beta. It won't be appreciated to the outside world how traumatic the upgrade from protocol buffers to from 2.4 to 2.5 was, but it was, believe me. The wire format may be designed to be forward- and backward- compatible, but the protobuf.jar files, and java code generated by protoc, is neither. Everything had to be moved forwards to protobuf 2.5 in one nearly-simultaneous move, which, for us coding against hadoop branch-2.1 and HBase 0.96 meant a few days of suffering. Or more to the point: a few days holding our versions locked down. On slide 7 of my HBase HUG Hoya talk I give a single line to build HBase for Hoya "without any changes"
n

As the audience may recall, I diverged into the fact that this only worked on specific versions of Hadoop 2.1, and, if you had Maven 3.1 installed, by explicitly skipping any site targets when building the tarballs. Even then, I didn't go near the how to upgrade protoc on ubuntu 12.x either.

Anyway, it's done, we can now all use protbuf 2.5 and pretend it wasn't an exercise in shared suffering that went all the way up the stack, all a week before the 2.1 beta came out.

As I was in the US at the time all this took place, I really got to appreciate how hard it is to get a completely tested Hadoop stack out the door -identifying integration problems between versions -and addressing them, testing for scale and performance on our test clusters. Everyone: QA, development and project management were working really hard to get this all out the door, and they deserve recognition as well as a couple of days rest.  It also gives me a hint of how hard it must be for a Linux vendor to get that Linux stack out the door -as things are even more diverse there. It shows why RHEL own the enterprise Linux business -and why they don't rush to push out leading-edg

Returning to the HDP-2 beta, there's a lot of stuff in the product: now that YARN is transforming a Hadoop cluster in a data-centric platform for executing a whole set of applications, things are getting even more exciting:  download HDP-2.0 beta and see.

As for me, I note a mention of Hoya in the blog posting, as well as the way HBase appears above and not alongside YARN, which is a bit of a premature layout -either way it means I am going to be busy.

Hadoop 2.1-beta: the elephants have come out to play

After many, many months, Hadoop 2.1 Beta is ready to play with



From Arun's announcement:  Users are encouraged to immediately move to hadoop-2.1.0-beta

Some other aspects of the announcement (with my comments in italic)

• API & protocol stabilization for both HDFS & YARN:
  Protobuf-format payloads with an API designed to be forward compatible; SASL auth.
• Binary Compatibility for MapReduce applications built on hadoop-1.x
  This is considered critical -test now and complain if there are problems.
• Support for running Hadoop on Microsoft Windows
  Irrespective of your view of what server OS/cloud platform to run on, this means all Windows desktops can now talk HDFS & be used for Hadoop-dev.
• HDFS Snapshots
  Lots of work by my colleagues here, especially Nicholas Tsz Wo: you can take a snapshot of directories and recover from your mistakes later. (JIRA, details).
• Substantial amount of integration testing with rest of projects in the ecosystem
  That includes the trauma of switching from protobuf 2.4 to 2.5 at the behest of the HBase team, something that led to seven days of trauma earlier this month as we had to do a lock-step migration of the entire cross-project codebase. Credit to all here.

What big things of mine are in there?

Primarily  YARN-117: hardening the YARN service model for better resilience to failure and reliable subclassing. There's certainly more than a hint of my old HADOOP-3628 work in there, which is itself somewhat related to SmartFrog -but the YARN service model itself has some interesting aspects that I can't take credit for. I should write it all up. Now that we have a common service API, we could now take away the many service entry points and write a single entry point that takes the name of a class, walks it through its lifecycle and runs it. This is precisely what YARN-679 proposes. Which in turn exactly what the Hoya entry point is. I'm using it there in both entry point and testing, so that I can evolve it based on my experience of using it in real apps.


The HADOOP-8545 openstack module isn't in there -I'd have liked it but at the same time didn't want to cause extra trouble by getting it in. FWIW the patch applies as is, it works properly -anyone can add it to their own build.

Minor tweaks whose implications are profound but nobody has noticed yet

• HADOOP-9432 Add support for markdown .md files in site documentation. This gives you an enhanced text format for docs that has good editor support, and renders directly in github.
• HADOOP-9833 move slf4j to version 1.7.5. This is initially for downstream apps that share the classpath, but it adds an option to Hadoop itself: the ability of Hadoop modules to switch from the commons-logging API to the SLF4J one: varargs with level-specific execution of an efficient unformatted printf output. This makes debug statements that much cleaner -and with print statements throughout the codebase, helps it overall.
• More FS tests and the patches to S3 and S3n to fix some aspects of their behaviour which could lead to loss of data if you got your rename destination wrong.
• All those patches I've done to trunk since 0.21. Because this release incorporates them: it is the big ASF first release of all the stuff that hasn't been backported. I'd recommend upgrading for my network diagnostics alone.

Even so, these are noise compared to the big pieces of work, of which the key ones are HDFS enhancements and the YARN execution engine, YARN being the most profound.

I was one of the people who +1'd this release, here is what I did to validate the build. Notice that my process involved grabbing the binaries by way of the ASF M2 staging repo: I need to validate the downstream build more than just the tarball.

# symlink /usr/local/bin/protoc to the homebrew installed 2.5.0 version

# delete all 2.1.0-beta artifacts in the mvn repo:

  find ~/.m2 -name 2.1.0-beta -print | xargs rm -rf

# checkout hbase source: from Apache: branch-0.95 (commit # b58d596 )

# switch to ASF repo (arun's private repo is in the POM, with JARs with the same sha1 sum, I'm just being rigorous)
<repository>
 <id>ASF Staging</id>
 <url>https://repository.apache.org/content/groups/staging/</url>
</repository>

# clean build of hbase tar against the beta artifacts

mvn clean install assembly:single -DskipTests -Dmaven.javadoc.skip=true -Dhadoop.profile=2.0 -Dhadoop-two.version=2.1.0-beta

# Observe DL taking place

[INFO] --- maven-assembly-plugin:2.4:single (default-cli) @ hbase ---
[INFO] Assemblies have been skipped per configuration of the skipAssembly parameter.
[INFO]                                                                        
[INFO] ------------------------------------------------------------------------
[INFO] Building HBase - Common 0.95.3-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.pom
Downloaded: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.pom (2 KB at 3.3 KB/sec)
Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/2.1.0-beta/hadoop-project-2.1.0-beta.pom
...

# get md5 sum of hadoop-common-2.1.0-beta artifact in https://repository.apache.org/content/groups/staging/:
0166f5c94d3699b3a37efc16ebb1ceea3acb3b53
# verify version of artifact in local m2 repo
    $ sha1sum ~/.m2/repository/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar
    0166f5c94d3699b3a37efc16ebb1ceea3acb3b53

# in hbase/hbase-assembly/target , gunzip then untar the hbase-0.95.3-SNAPSHOT-bin.tar file

# Patch the Hoya POM to use 2.1.0-beta instead of a local 2.1.1-SNAPSHOT

# run some of the hbase cluster deploy & flexing tests

mvn clean test  -Pstaging

 (all tests pass after 20 min)

Functional tests

# build and the Hoya JAR with classpath pulled in

mvn package -Pstaging

# D/L the binary .tar.gz file, and scp to an ubuntu VM with the hadoop conf properties for net-accessible HDFS & YARN services & no memory limits on containers

https://github.com/hortonworks/hoya/tree/master/src/test/configs/ubuntu
https://github.com/hortonworks/hoya/blob/master/src/test/configs/ubuntu/core-site.xml
https://github.com/hortonworks/hoya/blob/master/src/test/configs/ubuntu/yarn-site.xml

# stop the running hadoop-2.1.1-snapshot cluster

# start the new cluster services

hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
   
yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager
   
# Verify in dfshealth.jsp that NN is up, version is built ( 2013-08-15T20:48Z by hortonmu from branch-2.1.0-beta ). (the NN came up in safe mode as a couple of in-flight blocks were missing; NN recovered from this happily after 20s)

# copy hbase-0.93 tarball to HDFS

hdfs dfs -copyFromLocal hbase-0.95.3-SNAPSHOT-bin.tar hdfs://ubuntu:9000/hbase.tar

# unfreeze an hbase cluster that was running on hbase-0.95.2 & Hadoop-2.1.1 (protobuf 2.4) versions & set up to use hdfs://ubuntu:9000/hbase.tar as the hbase image to install

java -jar target/hoya-0.3-SNAPSHOT.jar \
 org.apache.hadoop.hoya.Hoya thaw cl1  --manager ubuntu:8032 \
 --filesystem hdfs://ubuntu:9000

(notice how I have to spec the classname -this is the service launcher set up as the main class for the Hoya JAR; it can run any YARN service)

# verify cluster is running according to Hoya CLI and YARN web GUI

# point browser at HDFS master: http://ubuntu:8080/master-status , verify it is happy

# fault injection: ssh-in and kill -9 all HRegionServers

# verify that within 60s the # no. of region servers is as desired; YARN notified Hoya of container loss; replacement containers requested and region servers deployed.

# Freeze the cluster:
java -jar target/hoya-0.3-SNAPSHOT.jar org.apache.hadoop.hoya.Hoya\
 freeze cl1 \
 --manager ubuntu:8032 --filesystem hdfs://ubuntu:9000

# verify that hoya list command and YARN GUI show app as finished; HBase cluster is no longer present

This validation shows what YARN can bring to the table: you can not only run stuff in the same cluster, near the data, YARN works with the App Master to notify it of things failing, leaving it to the AM to handle it as it chooses. For Hoya, the action is: except when shutting down, ask for the number of containers needed to keep the cluster at its desired state. This is how we can run HBase clusters inside a YARN cluster. And, if you look at the source code, more is on the way...

Returning to Arun's annoucement: download and play with this, especially if you've been using Hadoop 2.0.5 or other pre-2.1 YARN platform. For anyone using 0.2.3.x in production, it's time to start regression testing during this beta phase, and coming up with an update strategy. For anyone on branch-1 based products, the upgrade is probably more significant -the HDFS improvements justify it irrespective of the features of YARN that matter most in large clusters and heterogeneous workloads. Again: download, start making sure that your code works in it -because these are the last few weeks to find critical bugs in that code before everything gets locked down until the successor release.


[photo: elephants in the Tarangire National Park, Tanzania. Probably the wildest night camping of the trip. Children can't run in the campsite in case they get mistaken for prey, and the driver couldn't get at the truck all night due to the 6 lions sleeping by it].

Publishers: I'm not going to write or review a book on Hadoop for you

This is another of my stock responses, to go with those on LI connection requests and recruiters. This one is for publishers, with a special callout to Packt Publishing:

1. I agree, I have the knowledge and skills to write an excellent book on Hadoop.
2. I have no motivation to do so.
3. I wouldn't publish it through your tier-2 publishing house anyway.
4. I will not review any of your books as it is not worth my time

Amazon's Print on Demand (HP WebPress!) services appear to have brought some new players to the publishing market, ones with no upfront investment in paying for an initial print run. This lets them get authors to write books and, after a bit of reviewing, stick it up for sale -with no risk.  As a result, the have clearly commission-based "author executives" who waste their life scanning LI for people with Hadoop in their resume then effectively spamming them.

Every so often , an email hits one of my inboxes noting that I work on Hadoop and wondering if I am interested in writing a book for their -invariably unknown- publishing house.

From: Parita Khedekar <paritak@packtpub.com>
Date: 22 January 2013 12:02
Subject: [Steve], Author a "Big Data Analytics with R and Hadoop" book for Packt.
To: stevel

Hi Steve,

My name is Parita Khedekar and I am an Author Relationship Executive for Packt Publishing. We specialize in publishing IT related books, e-books, and articles that have been written by experts in the field.

We are currently looking out for prospective authors to write our book related to Big Data:

Big Data Analytics with R and Hadoop  aimed at Data Analysts and Scientists using Hadoop who need to take advantage of the R integration in their projects.
Scaling Big Data with Hadoop and Solr aimed at Solr developers who want to know how to leverage the flexible search functionality of Apache Solr and the Big Data processing of Apache Hadoop, to create the indexes for both general search and augmented data analytics.

Given your experience with this technology, I was wondering if you would be interested in authoring either of this titles.

Looking forward to hear from you and do let me know if you have any queries or doubts.

Thanks,
--

Parita Khedekar
Author Relationship Executive
PACKT Publishing
paritak@packtpub.com

The answer to this is, "no", for the following reason: I have already written a successful book on an open source project.

I know precisely how much effort it takes to update a good book: a lot. A book on software is effectively a software artefact, code that is required to work, along with consistent documentation. The more ambitious the first edition, the higher the maintenance costs- unless it has somehow been designed for maintenance up front.

Writing a paper of 10 pages can be done in under a week, revisions included. A chapter of 20 pages doesn't take two weeks, it takes 3-4. A book of 10 chapters doesn't take 10 * 3 weeks, it takes months more. Books, like software, are not O(n) products.

I know precisely how rapidly a book on a poplar open source project goes out of date: at the release rate of the software. Whereas a book on a closed source product, say MS Office 2010, is entirely in sync with the product for its entire lifespan, a book on an OSS app ages visibly With every release. This increases pressure for timely updates.

I know the ROI issues with writing a book, namely how many hours it takes, and how much you get in return: no much per hour of work. I also know that publishers don't have to care about how many hours the authors put in, as long as they meet their deadlines. Their ROI equations are based in cost of post-authoring actions: review management, typesetting and printing.  With print on demand, printing risk is eliminated, leaving only typesetting and reviewing. Skip quality typesetting and get reviewers to work  for free and you have no costs other than some minion to track the authoring process.

I also know -and this is relevant to anyone going "hey, how would you like to write a book"- multiple publishers. As well as Manning Press, people in O'Reilly. Both are brands with reputations for quality books.

So, for anyone inviting me to take up the opportunity to write a book on Hadoop for them:
 go away.

Regarding reviewing, it adds more work to my life for no benefit. Usually the reward is a PDF or hardcopy of the book. But consider this, a PDF costs the publishers $0, so you are being paid $0/hour. Even if you get the hardcopy, a $30 book would cost $15 to print -for a 2h review you are being paid $7.50. I consider my time more valuable than this.

There is one exception, If the book is by someone I know or work with, I may be able to put aside the time to do the reviewing -such people can contact me direct



Finally, the author talent acquisitions team should take the same advice as recruiters: do your research.

Take this approach

From: Anish Sukumaran <anishs@packtpub.com>
Date: 15 July 2013 12:32
Subject: [Steve], Author a 110 page book 'Apache ZooKeeper Administrator's Guide ' for Packt Publishing
To: stevel@hortonworks.com

Hello Steve,

My name is Anish and I am an Author Acquisition Executive at Packt Publishing. Packt is a rapidly growing, dedicated IT book Publishing firm and has rolled out more than thousand books on various titles till date.

Packt is now planning to publish a book titled as 'Apache ZooKeeper Administrator's Guide ' which would be a 110 page micro book and in the process of seeking potential authors to work on it , I also came across your Presentations on Slideshare. it is evident that you have a commendable experience and knowledge in this area.

It would be my pleasure to invite you to write this book for us.

Do let me know your decision and also if you have any queries, I will be happy to answer them.

Looking forward to hear from you.

Kind regards,
Anish Sukumaran
Author Acquisition Executive
PACKT Publishing
www.packtpub.com
MSN: anishs@packtpub.com

This "Author Acquisition Executive"  has been delving though slide share, looking at the presentations to see if they could identify possible authors, before approaching them.

Yet clearly, they have failed to do two things

1. Look up some internal spreadsheet of people who have already turned down a publishing opportunity with the publisher.

2. Go to Amazon and enter my name -because if they had done so they'd know I was already an author, perfectly capable of getting a book published by a quality publisher if I so chose to sit down and write one. Therefore I was already doing it for some other publishing house, or I was not in the mood to destroy all my free time for the next 12 months to write one.

Henceforth anyone spamming me with an publishing opportunity from a near-unknown Print on Demand shop will not only get sent this link, their approach will be added and ridiculed.

(photos: Therapy  by Inkie, commissioned piece for the hairdressers by the Highbury Vaults, captured at sunset)

Hoya: HBase on YARN

I didn't go to the Hadoop Summit, though it sounds really fun. I am having lots of of fun at a Big Data in Science workshop at Imperial college instead, where problems like "will the code to process my data still work in 50y", as well as the problems that the Square Kilometre Array will have (10x the physics dataset, sources across a desert, datacentre in the desert too)

What did make it over to the Summit is some of my code, the latest of which is Hoya, HBase on YARN. I have been busy coding this for the last four weeks:


Having the weather nice enough to work outside is lovely. Sadly, the wifi signal there is awful, which doesn't matter until I need to do maven things, where I have to run inside and hold the laptop vertically beneath the base station two floors above.



It's not that readable, but up on my display is the flexing code in Hoya: the bit in the AM that handles a request from the client to add or remove nodes. It's wonderfully minimal code, all it does is compare the (possibly changed) value of worker nodes wanted with the current value, and decides whether to ask the RM for some more nodes (using the predefined memory requirements of a Region Server), or to release nodes -in which case the RM will kill the RS, leaving the HBase master to notice this and handle the lost.

Asking for more nodes leaves the YARN RM to satisfy it, when then calls back to the RM saying "here they are". At which point Hoya sets up a launch request containing references to all the config files and binaries that need to go to the target machine, and a command line that is the hbase command line. There is no need for a Hoya-specific piece of code running on every worker node; YARN does all the work there.

Some other aspects of Hoya for the curious

• Hoya can take a reference to a pre-installed HBase instance, one installed by management tools such as Ambari, or kickstart installed into all the hosts. Hoya will ignore any template configuration file there, pushing out its own conf/ dir under the transient YARN-managed directories, pointing HBase at it.
• Although hbase supports multiple masters, Hoya just creates a single master exec'd off the Hoya AM. All but the live HBase master are simply waiting for ZK to give them a chance to go live -they're there for failure recovery. It's not clear we need that, not if YARN restarts the AM for us.
• Hoya remembers its cluster details in a ~/.hoya/clusters/${clustername} directory, including the HBase data, a snapshot of the configuration, and the JSON file used to specify the cluster.  You can machine-generate the cluster spec if you want.
• The getClusterStatus() AM API call returns a JSON description of the live cluster, in the same JSON format. It just adds details about every live node in the cluster. It turns out that classic Hadoop RPC has a max string size of <32K, so I'll need to rework that for larger clusters, or switch to protobuf, but the idea is simple: the same JSON structure is used for both the abstract specification of the cluster, and the description of the instantiated cluster. Some former colleagues will be noting that's been done before, to which the answer is "yes, but this is simpler and with a more structured format, as well as no cross-references".
• I've been evolving the YARN-679 "generic service entry point" for starting both the client and services. This instantiates the service named on the command line, hooking up signal handling to stop it. It then invokes -if present- an interface method, int runService() to run the service, exiting with the given error code. Oh, and it passes down the command line args (after extracting and applying conf file references and in-line definitions from it), before Service.init(Config) is called. This entry point is designed to eliminate all the service-specific entry points, but also provide some in-code access points to -letting you use it to create and run a service from your own code, passing in the command line args as a list/varags. I used that a lot in my tests, but I'm not yet sure the design is right. Evolution and peer-review will fix that.

Developing against YARN in its last few weeks of pre-beta stabilisation was entertaining -there was a lot of change. A big piece of it -YARN-117, was my work; getting it in meant that I could switch from the fork I was using to that branch, after which I was updating hadoop/branch-2; patching my code to fix any compile issues, retesting every morning. Usually: seamless; one day it took me until mid-afternoon for all to work, an auth-related patch on Saturday stopped test clusters working until Monday. Vinod was wonderfully help here, as was Devaraj with testing 50+ node clusters. Finally, on the Tuesday the groovyc support in Maven stopped working for all of us in the EU who caught an incompatible dependency upgrade first. To their credit the groovy dev team responded fast there, not only with a full fix out by the end of the day, but with some rapid suggestions on how to get back to a working build. It's just as you are trying to get something out for a public event, these things always hit your schedule: plan for them.

Also: Hoya is written in a mix of Java (some of the foundational stuff), and Groovy -all tests and the AM & client themselves. This was my second attempt at a Groovy YARN app, "Grumpy" being my first pass back in Spring 2012, during my break between HPLabs and Hortonworks. That code was never finished and too out of date to bother with; I started with the current DistributedShell example and used that -while tracking changes made to there during the pre-beta phase and pulling it over. The good news: a big goal of Hadoop 2.1 is stable protobuf-based YARN protocols, stable classes to help.

Anyway, Hoya works as a PoC, we should be letting out for people to play with soon. As Devaraj has noted: we aren't committed to sticking with Groovy. While some features were useful:  lists, maps, closures, and @CompileStatic finds problems fast as well as speeding up code, it was a bit quirky and I'm not sure it was worth the hassle. For other people about to YARN apps, have a look at Continuuity Weave and see if that simplifies things .


P.S: we are hiring.