Testing against S3 and object stores —keep your keys out of SCM

Nobody wants to commit their AWS credentials to an SCM repository, not when there are robots looking for your keys so they can spin up bitcoin mining VMs.


That's why git-secrets is something you should preinstall on all repos where you go near repositories. Unfortunately, it's pattern matching picks up a lot of invalid patterns in the Hadoop codebase, and a few in spark. In the absence of a decent pattern to only scan text files, after installing I edit out the regexps from .git/config and rely on it scanning purely for the strings in ~/.aws/credentials.

That keeps the keys out of SCM, if everyone is set up that way.

Which leaves the next problem: if you save your hadoop configuration files to SCM, how do you get AWS keys into system configurations?


The answer: XInclude


This is one of my configuration files, specifically hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml

This is a special file, tagged as .gitignore to keep it out of the repos. Yet still the keys are in that source tree, still at risk of sneaking out.

To deal with this, stick an absolute XInclude reference into the file, pointing to the configuration file where the keys really live.

<configuration>
  <include xmlns="http://www.w3.org/2001/XInclude"
    href="file:///home/stevel/.aws/hadoop-keys.xml" />
</configuration>

This tells the Hadoop config loader to grab the keys from a file in ~/.aws; one which lives carefully out of the SCM-managed space.

Provided the contents of that directory are kept private, my keys will not get checked in.

They can, however leak, in various ways, including

• In the /config URL of a service
• Code which accidentally logs it.
• If it gets saved to files/datasets used in bug reports.
• Malicious code running in your system which grabs the keys and exports/logs them. This is why no OSS jenkins servers are set up with the keys needed to test against object store.

Clearly, more should be done to lock it down.

There are some patches under the S3A phase 2 JIRA to support credentials better, [HADOOP-12723, HADOOP-12537]. This is somewhere where anyone who can test these patches is invaluable. I'm currently doing the review-then-commit in S3, but I'm only doing it at weekends, when I have some spare time, and even then, as a full test run takes 2+ hours, not reviewing very much.

Anyone can review Hadoop patches, confirm whether they worked or not, show how they didn't. It's the way to verify that forthcoming code from other people works for you, and of contributing time and effort into the community. Other people have done the coding —time to help with the testing.

Especially now I've just documented how to keep the keys safe when you set up S3 for a test run. Once the auth-keys.xml file is in the src/test/resources directory, Maven tests s3, depending on the specific properties, s3a, s3n and s3a are all tested.

Update: 2016-06-20: I've discovered that if you use the id:secret in a URL for an S3x filesystem, e.g s3a://AWS01:sec/ret@mybucket then the secrets get logged everywhere, because the Hadoop code assumes there aren't secrets in the Path or Filesystem URIs. We've cranked back a bit on the leakage (by stripping it from the FS URIs), but it still gets out everywhere. Fix: don't do that. Hadoop 2.8 will explicitly tell you off whenever you do so.

Update: 2018-02-07: Hadoop 2.9+ uses a more efficient Stax XML parser over a DOM based one. That broke this secret-sequestering somewhat dramatically. For Hadoop 2.9+, do not use the file: prefix

  <include xmlns="http://www.w3.org/2001/XInclude"
    href="///users/stevel/.aws/auth-keys.xml" >
  </include>

FWIW, it's possible to have base auth-key.xml with XIncludes which supports both parsers by having both references in the file, and add a <fallback/> element in both so an unresolvable ref is a no-op. Worth knowing if you need to test across all branches.

[photo: Mina Road Park, St Werburghs]

Voxxdev Bristol 2016: theory meets practitioners

I've been hanging out with other bits of the Bristol tech/dev world recently, rather than the usual big data suspects.

Two weeks ago, I actually infiltrated the local Test meetup, which was entertaining, not just for the Bath Gem Ale which JustEat has on tap in their meeting area, but because I sit there, beer in hand, while speakers covered Applied Exploratory Testing, what its like turning up as a QA team to a new company and the patterns-of-untestability that you encounter (and how to start to get things under control).

This week, I took a break from worrying about the semantics of flush() and its impact of durability of timeline 1.5 event histories. (i.e. why my incomplete apps aren't showing up in an ATS-backed  Spark History server if file:// is the intermediate FS of the test run). Wandered down to the Watershed Cinema with Tom White and a light hangover related to Tom White's overnight stay including an evening visit the Bravas Tapas Bar —and into a one-day dev conference, Voxxdev Bristol 2016.

It was a good day. Oracle have been putting a lot of effort into the conf as a way raising visibility of what's going on in tech in the area to make more people aware the west of england a more interesting place to be than London, and with other companies and one of the local universities had put together a day-long conference.

I was one of the speakers; I'd put in my Household Infosec talk, but the organisers wanted something more code-related, and opted for Hadoop and Kerberos, the Madness Beyond The Gate. I don't think that was the right talk for the audience. It's really for people writing code to run inside a Hadoop cluster, and to explain to the QE and support people that the reason they suffer so much is those developers aren't testing on secure clusters. (that's the version I gave at Cloudera Palo Alto last month). When you are part way into a talk and you realise that you can't assume the audience knows how HDFS works then you shouldn't really be talking to them about retrieving block-tokens from the NN from a YARN app handed off a delegation token in the launch context by a client authed against the KDC. Normally I like a large fraction of the audience to come out of a talk feeling they'd benefited; this time I'm not sure.

I felt a bit let down the oracle big data talk, though impressed that people are still writing Swing apps. I was even more disappointed by the IoT talk, where he not only accused Hadoop of being insecure (speaker missed my talk, then), most of his slides seemed lifted from elsewhere: one a cisco IoT arch, one dell hadoop cluster design. Julio pointed out later that the brontobyte slide was one HP Labs have been using. Tip: if you use other's slides, either credit them or make sure the authors aren't attendees.

I really liked some of the other talks. There was a great keynote by a former colleague, Dave Cliff, now at Bristol Uni, talking about what they are up to. This is the lecture series on their cloud computing course. 



That's a big change given that in 2010, all they had was a talk Julio and I gave in the HPC course

I might volunteer to give one of the new course's talks in exchange for being able to sit in on the other lectures (and exemption from exams, tutorials and homework, obviously)

My favourite talk turned out to be Out of the Fire Swamp, by Adrian Colyer.

Adrian writes "The morning paper" blog, which is something I'm aware of, in awe of and in fear of. Why fear? There's too many papers to read; I can't get through 1/day and to track the blog would only make it clear how behind I was. I already have a directory tree full of papers I think I need to understand. Of course, if you do read a related paper/day, it probably gets easier, except I'm still trying to complete [Ulrich99], as my and relate it to modern problems.

Adrian introduced the audience to data serialization, causality and happens-before, then into linearalizability [HW90]




This was a really good talk.



All the code we write is full of assumptions. We assume that n + 1 > n, though we know in our head that if n = 2^31 and its stored in a signed int32, that doesn't hold. (more formally, in two's complement binary arithmetic, for all registers of width w, n+1 > n only holds for all n where n < 2^(w-1).

Sometimes even that fundational n+1 > n assumption catches us out. We assume that two assignments in source code happen in order, though in fact not only does the JVM reserve the right to re-order things, even, in the past, wrongly —and anyway, the CPU can reorder stuff as well.

What people aren't aware of in modern datacentre-scale computing is what assumptions the systems underneath have made in order to give their performance/consistency/availability/liveness/persistence features or whatever it is they offer. To put differently: we think we know what the systems we depend on do, but every so often we find out our assumptions were utterly wrong. What Adrian covered is the foundational CS-complete assumptions, that you had really be asking hard questions about when you review technologies.

He also closed with a section on future system trends, especially storage, with things like  non-volatile DIMMS (some capacitor + SSD to do a snapshot backup on power loss), faster SSD and those persistent technologies with performance between DRAM and SSD: looking at a future where tomorrow's options about durability vs. retrieval time are going to be (at least from some price points), significantly different from today's. Which means that we'd better stop hard coding those assumptions into our systems.


Overall, a nice event, especially given it in the second closest place to my house where you could hold a conference (Bristol University would have been 7 minutes rather than 15 minutes walk). I had the pleasure of meeting some new people in the area working on stuff, including the illustrious James Strachan who'd come up from Somerset for the day.

I liked the blend of some CS "Lamport Layer" work with practical stuff; gives you both education and things you can make immediate use of. I know Martin Kleppman has been going round evangelising classic distributed computing problems to a broader audience, and Berlin Buzzwords 2016 has a talk Towards consensus on distributed consensus  —it's clearly something the conferences need.

If there was a fault, as well as some of the talks not being ideal for the audience (mine), I'd say it's got the usual lack of diversity of a tech conference. You could say "well. that's the industry", but it doesn't have to be, and even if that is the case it doesn't have to be so unbalanced in the speakers. In the BBuzz even not only are there two women keynoting, we submission reviewers were renewing the talks anonymously: we didn't know who was submitting, instead going on the text alone.

For the next Bristol conference, I'd advocate going closer in Uni/industry collaboration by offering some of the students tickets. Maybe even some of the maths & physics students rather than just CS. I also think maybe there should be a full strand dedicated to CS theory. Seriously: we can do causality, formality, set theory & relational algebra, paxos+ZK+REEF, reliability theory, graphs etc. Things that apply to the systems we use, but stuff people kind of ignore. I like this idea ... I can even think of a couple of speakers. Me? I'd sit in the audience trying to keep up.

[HW90] Herlihy and Wing, Linearizability: A Correctness Condition for Concurrent Objects, 1990.

[Ulrich99] Ulrich, A.W., Zimmerer, P. and Chrobok-Diening, Test architectures for testing distributed system, 1999. Proceedings of the 12th International Software Quality Week.

trouble ahead

It's now over a month since I outlined my 2016 plans: get fitter, do more on testability.

What's the progress?

I've got a place on the 2016 Fred Whitton Challenge, which is widely regarded as the hardest one-day "fun" ride in the UK.

Having done it in 2014, I am minded to concur


That time 4 people got helicoptered off; I have no intention of qualifying for the same transport this year.

Instead: I now have to get fit enough to do 110 miles up 30% gradients, when I'm currently fit enough to do 25-30 miles.

In 10 weeks.

2016 plannage

Goals for 2016?

1. Reduce weight and percentage body fat to where it was of May 2014, where I'd just got in shape for the annual Fred Whitton ride. I'm putting my name into the lottery for that again, which would provide extra motivation. Strategy: eat less, do more work on the bike.
2. Try and avoid visits to the Bristol Royal Infirmary for any reason. This essentially means "don't overdo things, be it on-road climbs or off-road descents". I believe the family will be happy here. Note the implied the conflict with goal #1. Ideally both goals could be achieved —but failure of both is also possible.
3. Make some progress on distributed testing. There's surprisingly little literature here.

I'll start on #3 by catching up with those few bits of published literature, as well as playing with things like Spock and JUnit Lambda, then delving into the test runners and reporters. I'm starting to explore some of this with Slider and some corners of Spark that I've been involved in.

Links to work in the area welcome...

[photo: Torres del Paine, Xmas, 1994. Windy, as I recall]

Developer INFOSEC in a Post-Sony era

There's going to be a dev-conference in Bristol in February, voxxed days bristol;
where I'll be presenting my Hadoop and Kerberos tak,, maybe with a demo of the Slider kdiag command, which after a couple of iterations I intend to move into Hadoop core as part of HADOOP-12649: Improve UGI diagnostics and failure handling. Any version which gets into hadoop core will be able to use some extra diagnostics I hope to add to UGI itself, such as the ability to query the renew time of a ticket, get the address of the KDC and probe for it, maybe more. Because Kerberos doesn't like me this week, at least not with zookeeper.

It should be a fun day and local developers should think about turning up. I was at the last little developer bash in November, where there was a great mix of talks ranging from the history of the Transputer to the challenge of implementing a mobile-app only bank.



What apparently hasn't make the cut is my other proposal, Household INFOSEC in a Post-Sony era. Which is a shame, as I have the outline of a talk there which would be seminal. This wouldn't a be a talk about platitudes like "keep flash up to date", it'd be looking about my rollout of a two-tier network, with the IoT zone hooked directly up to the internet by the router provided by the ISP, and a DD-WRT router offering a separate trusted-machine subnet and high-entropy-passworded, no-SSID-published wifi network restricted to devices I can control.

What I was really going to talk about, however, was how we are leaking personal information from all the devices we use, the apps we host in them, the cars we drive, the cameras we use (guess whose camera app considers camera GPS and timestamp information non-personally identifiable and hence uploadable? That's right: Sony). We leak that information, it gets stored elsewhere, and we now depend on the INFOSEC capabilities of those entities to keep that data private. And that's hard to pull off —even with Kerberos everywhere.

One aspect of my homework here is working out what data I have on my computers which I consider sensitive.

There's photos, which are more irreplaceable than anything else: my policy there is Disaster Recovery: using Google Photos as the off site backup; a local NAS server in the trusted subnet for on-site resilience to HDD failures. That server is powered off except for the weekly backups, reducing its transitive vulnerability to any ransomware that gets onto into the trust zone.

There's passwords to web sites, especially those with purchasing rights (amazon, etc), and financial institutions (paypal, banks) —in the UK my bank uses its chipped debit cards as the physical credential for login and cash transfer, so it's fairly secure. The others: less so, especially if my browser has been intentionally/unintentionally saving form information with things like CVV numbers.

And what else matters? I've come to the conclusion it is the credentials needed to gain write access to the ASF source code repositories. Not just Hadoop —it's the Ant source, it's slider, it's anything else where I am creating code where any of the following criteria are met

1. The build is executed by a developer or a CI tool on a machine holding information (including their own credentials) to which someone malicious wants access.
2. The build is executed by a developer or a CI tool on a machine running within a network to which someone malicious wants access.
3. Generated/published artifacts are executed during a build process on a machine/network to which someone malicious wants access.
   
4. The production code is executed somewhere where there is data which someone malicious wants to get at or destroy, or where adverse behaviour can the system is advantageous to someone malicious.

You can point to the Hadoop stack and say: it's going that way —but so is the rest of the OSS codebase. The LAMP stack, tomcat web servers, Xerces XML parsers, open office, linux device drivers, clipboard history savers like glipper, Python statistics libraries, ... etc. We live in a world where open source is everywhere from the datacentre to the television. If anyone malicious has the opportunity to deliberately inset vulnerabilities into that code —then they get to spread them across the planet. That source code then, is both a juicy target for anyone looking for 0-day exploits, but also for inserting new ones.

We've seen attacks on Kernel.org, and the ASF. With the dominance of git as the SCM tool, and it's use of SHA-1 checksums, the value of breaking into the servers is diminishing —what you need to go is get the malicious code checked in, that is: committed using the credentials of an authorised developer.

That'll be us then.

More succinctly: if the Internet becomes the location of an arms race, we're now targets en route to strategic goals by entities and organisations against whom we don't stand a chance.

How do you defend against nation states happy to give away USB-chargeable bicycle lights at an OSS conference? Who have the ability to break through your tier-3 ISP firewall and then the second level DD-WRT router that you never locked down properly and haven't updated for three weeks. We're don't stand a chance, not really

No doubt I'll come over as excessively paranoid, but its not as if I view my personal systems a direct target. It's just the source code repos do which I do have access are potentially of interest. And with other people in the Hadoop space building those same projects, something injected into the build using my credentials then has transitive access to everyone else who checks out and builds the same codebase. That's what worries me.

WTF do we do?

Short-term I'm switching my release process to a VM that's only ever used for that, so at least the artifacts I generate aren't indirectly contaminated by malware; I also need to automate a final SHA1 audit of every dependent artifact.

Medium term: I need to come up with a plan for locking down my git ssh/svn credentials and passwords so they aren't trivially accessible to anything malicious running on any laptop of mine. I know github is moving to 2FA and U2F auth, but that's for web and API auth: not git repo access. What the Linux Kernel team have is a much better story: 2FA for granting 24h of write access from a specific IP address.

Long term: I have no idea whatsoever

[photo: two Knog lights you charge up via USB ports. We should all know to be cautious about plugging in untrusted USB sticks —but who would turn down a freebie bike light given away at an OSS developer conference?]

Remembering the glaciers: Greenland

In July 2012, while wandering around on a flight to the US, I looked out the window and got to see what Greenland looks like from above.



When this ice melts, it raises the ocean.



And the mountains will be bare rock, as they have not been since before the ice ages began.



There's something deeply tragic about a planeload of people, sitting in their seats, windows shuttered, watching the videos streamed to keep the populace happy —while if they chose to look up they could see the great icefields melting. Arguably, that's a description for society as a whole —and all of us in it.

Remembering the glaciers: Fox Glacier, New Zealand

The southern hemisphere has its own ice. I can't find any of my 1994 Peru/Chile expedition photos, so nothing of the Southern Patagonian Ice Cap, the Torres del Paine or snow-capped volcanoes.

Instead, here's one from New Zealand, the Fox Glacier.



This was on South Island, somewhere down the west coast, December 1998.

What's impressive here is how it comes down almost to sea level: we were staying nearby and this was a short and mostly flat walk from where we were staying. It's in the temperate rain forests of the region. The fact that the entire valley looks fairly glacial implies that its been bigger in the past -like many of the other Fjords on the island, it shows the glaciers once made their way out to a lower sea level. (the same goes for Scottish sea glens, presumably)

The whole glacier was making the sound of gently running water, right down by the terminal moraines. Apparently this glacier actually advanced between 2005-2009, but it's retreating again.

[Team: Steve, Bina]
[Photo: Fujichrome ASA100 slide (presumably) on an olympus zoom-lens compact camera I'd bought in NZ]