Gardening the commons

It's been a warm-but-damp summer in Bristol, and the vegetation in the local woods has been growing well. This means the bramble and hawthorn branches have been making their way out of the undergrowth and into the light —more specifically the light available in the mountain bike trails.

Being as both these plant's branches have spiky bits on them, the fact that they are growing onto the trails hurts, especially if you are trying to get round corners fast. And, if anyone is going round the trail without sunglasses they run a risk of getting hurt in/near the eye.

I do always wear sunglasses, but the limitations on taking the fast line though the trails hurts, and as there a lot of families out right now, I don't want the kids to get too scraped.

So on Saturday morning, much to the amazement of my wife, I picked up the gardening shears. Not to do anything in our garden though —to take to the woods with me.



Findings

1. Those Kevlar backed gloves that the Clevedon police like are OK on the outside for working with spiky vegetation, but the fingertips are vulnerable.
2. Gardening gets boring fast.
3. When gardening an MTB trail, look towards the direction oncoming riders will take.
4. A lot of people walking dogs get lost and ask for directions back to the car park.
5. Someone had already gardened about 1/3 of the Yertiz trail. (pronounciation based on: "yeah-it-is"
6. Nobody appreciates your work.

I appreciate the outcome my own work, I can now go round at speed, only picking up scrapes on the forearms on the third of the trail that nobody has trimmed yet. I actually cut back on the inside of the corners there for less damage on the racing line, while cutting the outside and face height bits for the families. Now they can have more fun on the weekends, I can do my fast work weekday lunchtimes.


They can live their lives with fewer wailing children, and I've partially achieved my goal of less blood emitted per iteration. I'll probably finish it off with the final 1/3 at some point, maybe mid-august, as I can't rely on anyone else.

There are no trail pixies but but what we make



Which brings me to OSS projects, especially patches and bug reports in Hadoop.

I really hate it when my patches get completely ignored, big or small.

Take YARN-679 for example. Wrap up piece of YARN-117, code has celebrated its third birthday. Number of reviewers. Zero. I know its a big patch, but it's designed to produce a generic entry point for YARN services with reflection based loading of config files (you can ask for HiveConfig and HBaseConfig too, see), interrupt handling which even remembers if its been called once, so on the second control-C bypasses the shutdown hooks (assumption: they've blocked on some IPC-retry chatter with a now-absent NN), and bail out fast. Everything designed to support exit codes, subclass points for testability. This should be the entry point for every single YARN service, and it hasn't had a single comment by anyone. How does that make me feel. Utterly Ignored —even by colleagues. I do, at least, understand what else they are working on...it's not like there is a room full of patch reviewers saying "there are no patches to review —let's just go home early". All the people with the ability to review the patches have so many commitments of their own, that the time to review they can allocate is called "weekends".

And as a result, I have a list of patches awaiting review and commit, patches which are not only big diffs, they are trivial ones which fix things like NPEs in reporting errors returned over RPC. That's a 3KB patch, reaching the age where, at least with my own child, we were looking at nursery schools. Nothing strategic, something useful when things go wrong. Ignored.

That's what really frustrates me: small problems, small patches, nobody looks at it.

And I'm as guilty as the rest. We provide feedback on some patch-in-progress, then get distracted and never do the final housekeeping. I feel bad here, precisely because I understand the frustration.

Alongside "old, neglected patches", there are "old, neglected bugs". Take, for an example HADOOP-3733 "s3:" URLs break when Secret Key contains a slash, even if encoded. Stuart Sierra gave a good view of the history from his perspective.

The bug was filed in 2008

1. it was utterly ignored
2. Lots of people said they had the same problem
3. Various hadoop developers said "Cannot reproduce"
4. It was eventually fixed on 2016-06-16 with a patch by one stevel@apache.
5. On 2016-06-16 cnauroth@apache filed HADOOP-13287 saying "TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains '+'.".

Conclusions

• The Hadoop developers neglect things
• if we'd fixed things earlier, similar problems won't arise.

I mostly concur. Especially in the S3 support, where historically the only FTEs working on it were Amazon, and they have their own codebase. In ASF Hadoop, Tom White started the code, and it's slowly evolved, but it's generally been left to various end users to work on.

Patch submission is also complicated by the fact that for security reasons, Jenkins doesn't test the stuff. We've had enough problems of people under-testing their patches here that there is a strictly enforced policy of "tell us which infrastructure you tested against". The calling out of "name the endpoint" turns out to be far better at triggering honest responses than "say that you tested this". And yes, we are just as strict with our colleagues. A full test run of the hadoop-aws module takes 10-15 minutes, much better than the 30 minutes it used to take, but still means that any review of a patch is time consuming.

I would normally estimate the time to review an S3 patch to take 1-2 hours. And, until a few of us sat down to work on S3A functionality and performance against Hive and Spark, those 1-2 hours were going weekend time only. Which is why I didn't do enough reviewing.

Returning to the S3 "/", problem

1. This whole thing was related to AWS-generated secrets. Those of us whose AWS secrets didn't have a "/" in this couldn't replicate the problem. Thus it was a configuration-space issue rather than something visible to all.
2. There was a straightforward workaround, "generate new credentials", so it wasn't a blocker.
3. That related issue, HADOOP-13287, is actually highlighting a regression caused by the fix for HADOOP-3733. In the process for allowing URLs to contain "/" symbols, we managed to break the ability to use "+" in them. 
4. The regression was caught because the HADOOP-3733 patch included some tests which played with the tester's real credentials. Fun problem: writing tests to do insecure things which don't leak secrets in logs and assert statements.
5. HADOOP-13287 is not an example of "there are nearby problems" so much as "every bug fix moves the bug", something noted in Brook's "the mythical man month" in his coverage of IBM OS patches.
6. And again, this is a c-space problem, it was caught because Chris had + in his secret.

Finally, and this is the reason why it didn't surface with many of us, even though we had "/" in the secret is because the problem only arises if you put your AWS secrets in the URL itself, as s3a://AWSID:secret-aws-key@bucket

That is: if your filesystem URI contains secrets, which, if leaked —threaten the privacy and integrity of your data and is at risk of running up large bills, then, if the secret has a "/", the URL doesn't authenticate.

This is not actually an action I would recommend. Why? Because throughout the Hadoop codebase we assume that filesystem URIs do not contain secrets. They get logged, they get included in error messages, they make their way into stack traces that can go into bug reports. AWS credentials are too important to be sticking in URLs.

Once I realised people were doing this, I did put aside a morning to fix things. Not so much fixing the encoding of  "/" in the secrets (and accidentally breaking the encoding of "+" in the process), but:

1. Pulling out the auth logic for s3, s3n and s3a into a new class, S3xLoginHelper.
2. Having that code strip out user:pass from the FS URL before the S3 filesystems pass it up to their superclass.
3. Doing test runs and seeing if that is sufficient to keep those secret out the logs (it isn't).
4. Having S3xLoginHelper print a warning whenever somebody tries to use secrets in URLs.
5. Edit the S3 documentation file to tell people not to do this —and warning the feature may be removed.
6. Edit the Hadoop S3 wiki page telling people not to do this.
7. Finally: fix the encoding for /, adding tests
8. Later, fix the test for +

That's not just an idle "may be removed" threat. In HADOOP-13252, you can declare which AWS credential providers to support in S3A, be it your own, conf-file, env var, IAM, and others. If you start doing this, your ability to embed secrets in s3a URLs goes away. Assumption: if people know what they are doing, they shouldn't be doing things so insecure.

Anyway, I'm glad my effort fixing the issue is appreciated. I also share everyone's frustration with neglected patches, as it wastes my effort and leaves the bugs unfixed, features ignored.

We need another bug bash. And I need to give that final third of the Yertiz trail a trim.

Travels abroad

Early in July I did a bike ride with many other parents from Alexander's primary school; bikes loaded into a truck and driven down to Bergerac, us flying out to the airport, then riding south at 60-100 miles/day, ending up in San Sebastian, Spain, skirting the pyrenees carefully.


Here's me, at the Atlantic coast, June 2006.





This was the first time I'd been at the French/Spanish border of Hendaye/Irun for 26 years. Then we loaded the bikes onto a train in London, then three days later a train+ferry+overnight train to get to Hendaye station, again getting on our bikes at the Atlantic seaside —this time, August 1990.



As usual, not the combined weight of rider+bike+luggage is constant, even though now the bike is a Ti+Carbon CX machine, and then it was a steel MTB with panniers full of camping equipment and a change of clothes. (*)

Two weeks later, after zig-zagging up the highest roads in the Pyrenees, wobbling in and out of France and Span, we got to the Mediterranean Sea, somewhat the worse for wear.


Return journey: similar, preload the bikes, overnight train to paris, onboard to London, pickup the bikes and then further trains west. 24+ hours, at least.


In 1990, that was a major expedition. I got a credit card for the first time, changed money into FFr and Spanish Pesetas, took some travellers checks to change money as went down. Maps: paper things hooked onto the handlebars. Camera: my first 35mm autofocus Canon compact camera. It had a little self timer, hence the various staged team-selfie pics. The bikes were generations behind what I have now, as were: The lights, the camping kit, much of the clothing. Communications: postcards.

This time, it's not a expedition, it's just a week on the mainland. Money? Euros left over from my June trip to Berlin, topping up at ATMs. Camera? on the phone plus a compact digital camera with proper zoom lens. Maps: Garmin GPS and offline google maps. Communications: phone with free phone and data roaming round Europe. Luggage: cards, a bag full of chargers and miscellaneous electronics. Riders: much, much worse. If I'd had Strava then, today's numbers would be so bad I'd give up in despair.

And now: not a multi-train expedition, a quick holiday from the local airport, followed by a few days in San Sebastian and Bilbao, back home from another direct flight, home in two hours, then driving over to Portsmouth to collect a child: a low effort, low stress day. It took as long to drive to Portsmouth as it did from our Bilbao hotel to our Bristol house.


This was not a trip to foreign lands where money is different, where a passport is needed to be handy as you cross the borders, where things are exotic. We are in Europe now, and have moved on from the neighbouring countries being far, far, away.

Except now: things have changed. The rate of decay of the UKP:EUR exchange rate meant that we had to run to the ATMs, fearing that the rate would be worse if waited 24h. Where today, we could roam freely as part of a continent, the decisions of fellow UK voters means that we're taking a step backwards as a nation.

While that exoticness 26 years ago made for more of a wilder expedition, we have gained so much. Yet now Britain is turning its back on it in exchange for a social and economic fiasco. It's going to be a disaster, and when that happens, the people to blame: the paper proprietors, the politicians who lied —they'll get away with it. The country will suffer, worse of all those people in the forgotten towns who believed the lies. Well: they voted for it —their fault. If need be, I can walk away from the country, over to the continent which I consider myself a full citizen of.

(*) We had a regime of wash the previous day's clothes in the morning, hang to dry on the bike, wear the fresh stuff in the evening and again on the following day's ride. It worked, though on dusty days things got a bit gritty.

trouble ahead

It's now over a month since I outlined my 2016 plans: get fitter, do more on testability.

What's the progress?

I've got a place on the 2016 Fred Whitton Challenge, which is widely regarded as the hardest one-day "fun" ride in the UK.

Having done it in 2014, I am minded to concur


That time 4 people got helicoptered off; I have no intention of qualifying for the same transport this year.

Instead: I now have to get fit enough to do 110 miles up 30% gradients, when I'm currently fit enough to do 25-30 miles.

In 10 weeks.

Modern Alpinism

I am back from a 2+ week vacance in the French Alps —the first since 2009. More specifically, the first since the DVLA took away my driving license on the basis that I was medically unfit to drive. They now consider me no more dangerous than anyone else, so we could drive down there. My main issue with driving now is that I've forgotten things: motorway awareness, the width of my car (very important in bristol), what it feels like to go round corners fast enough for car to indicate through the steering wheel that it doesn't want to —and worst of all, how to parallel park into small gaps.

Two weeks in the French Alps, along with 2-day approaches/retreats is a good revision there, giving me the learning matrix of

	Day	Night	Bad Weather
Autoroute	√	√	√
French village	√	√	√
French urban (excluding Paris)	√	√
Windy country roads	√		√
Alpine passes	√		√

There's a few left, and I didn't try any overtakes, but otherwise that's reasonable coverage of the configuration space.

While there: MTB work, some on-road work including a 100 mile epic, a bit of basic Alpine hut hiking/scrambling —as well as staying with friends, eating too many cheese-based products (tartiflettes &c), consuming french beer and cidre, catching the Tour de France Chamrousse Stage and generally having fun.

I got to review some of the places last visited in 2009:


Where I'm pleased to see that I don't seem significantly rounder





The main difference this time was that someone was rinsing their clothes in this village fountain half-way up Col du Barrioz, so we had to sprint off before we got harassed, crossing the pass and sheltering from some rain in a bar/tabac where I opted to not drink that day



a policy which the camera's EXIF data implies lasted exactly nine minutes





Anyway: an excellent holiday —it was great to be back in the Alps.

Strava bringeth bad news

I'm in the bay area right now, and the new owner of a Google Nexus phone, which is very good at integrating with Google apps, including calendar, contacts and mail. It also runs Strava, the defacto standard app for logging your cycling, then uploading the results to compare with others. I'm assuming its running Hadoop at the back end, given their Platform Product Manager is one of the ex-Yahoo! Hadoop team.

If this is the case, Hadoop is indirectly bringing me bad news.

Yesterday I went out on the folding bike and climbed the Santa Cruz mountains, west of the Bay Area flatlands.

It's a great steep climb from behind Palo Alto up to the skyline ridge, narrow and free from almost all traffic bar the many other locals who felt that Saturday morning was to nice to waste. For me: 26 minutes climbing, 40s of rest -fast enough to come in the top 50% of the day of everyone else running the app, and hence 2233 of 5810 of everyone who has ever done it. Not bad work.


If there's a warning sign, it is that people faster than me have a quoted average power output less then my 257W -and as they all took less time, that means that their total exertion is less than my 412kJ. Why do those ahead of me come in lower? It means they are carrying less excess weight.

If I'd stopped there and descended (carefully, that bikes 20" rims overheat), I could get back and feel relatively smug.

Only this time I descended the far side and climbed back up "Alpine West" -first time ever. And it destroyed me

It was long. I stopped for lunch partway up, but needed that rest, and continued, expecting the overall climb to be on a par with the earlier one. It wasn't. In fact the total climb was double, 600m. Which I was not ready for. Unlike the morning, where I'd got to pass lots of people, there the road was near empty -and those people I did meet were going past me. The rate of ascent, 562m, is less than the 600m/hour rate we used to plan for when crossing the alps -a rate sustained over a week or more, carrying panniers. Not today.

The message from Strava then, which means something Hadoop worked out, is that I am overweight and completely lacking in endurance. It' doesn't quite spell it out, but the graphs bring the message

This is datamining at work.

The Cheddar Expedition

Last weekend I took advantage of a confluence of events: sunny weather and no parental obligations, to go for a quick ride down to Cheddar and back.

Plan: southwest via Long Ashton/Festival Way to Backwell; meander on to Yatton. From there, the Strawberry Line; a traffic free cycle route along a disused railwayline to Axbridge Cheddar. A curved approach, but a flat one -my left knee is still recovering from something I did to it in January. Cheddar is the other side of the Mendip hills; a minor 300 metre summit, but with most of the roads having sustained 15-25% gradients, it's not easy.

The old railwaypath would get me over on a flat and quiet road, then a single climb up the spectacular Cheddar Gorge, down the other side and then a fairly undulating route home along country lanes.

Departure time: 12:30. Expected return time: 17:00-17:30, "before it gets dark"

Actual return time: 18:30, "an hour after sunset"

To be fair, I did get to see a very dramatic sunset.


It's just that I had same feeling you get if you are up some cliff or mountain when the sun goes down and you hadn't intended to be there at that time.

Sunset can mean you are in trouble. It's why sunrise is somewhat a nicer thing to see in the mountains, though as it means that the avalanche, serac collapse and other ice-melt-triggered objective hazards are becoming more likely, even that can be unwelcome. That's in the Alps though. Here sunset is the one to worry about, as it means you have a lot of work in darkness to get home.

Why did I come in an hour late? Route closures.

The first problem was that the Strawberry Line's tunnel through the crest of Mendips was closed. There was no advance signage of this until I got to the path closure, followed a sign through some "farm mud" track, and was the dumped on the A38 road. Of all the routes over the Mendips that you can cycle, this has to be the one marked "least pleasant". At the summit I turned off and got onto a bridlepath, so could get off the main road. Of course, being just a track in the woods, it was fairly muddy, so not having tyres with tread on them meant the descent was fairly slow.

Eventually I did get the bike down, and then continued on to Cheddar, coming in about half an hour late.

When I get to Cheddar though, there are lots of signs "Cheddar Gorge shops still open". That's a warning sign. You only get signs staying "still open" if it was considered likely that you thought they'd be closed -which left me to wondering "why would they be closed".

The reason people may have thought they were closed, as it turned out, were that the gorge road had been shut since November due to flooding. The whole of Mendips is a limestome massif, with all drainage in the southern watershed being underground. There'd been enough rain that the caves flooded and blocked up; the water surfaced in the gorge, and caused lots of problems. Which were now my problems.

At this point it's 3pm, 2-2.5 hours to go, and I'm on the wrong side of the hills with the route I came out of play, and the route I'd planned to take back also closed. This is not ideal. I have some on-road options, but either it's back on the main roads or carry on down to Wells and up, which is pretty time consuming.

Immediate action: cafe for hot chocolate, chocolate of some other form and wifi for the phone to download some high-res OpenStreetMap datasets.

It'd have been straightforward if I'd been a bike set up for the conditions, meaning tyres that had traction or control in mud. I didn't


For the uphill I could just get off and push -at least my shoes had good traction- but the undulations and descents were hard, not least because you don't want to lose control a short distance away from a 200 metre cliff edge:




I didn't get off this bridleway until about 16:30, which then fed forward to the schedule for the rest of the ride. Sunset before I'd got past the airport, darkness from there to home.

Anyway: two route closures and a bicycle not set up for off-road alternative options resulted in me getting home well after dark. That's when you are glad that modern bike lights are so good.

And to be fair, I did get some great views, not just the sunset, but the late sun over the Somerset Levels, here with Glastonbury Tor, sometimes known by its more fanciful name "Isle of Avalon"


The gorge itself is good from up here, though you only see the top half. It cuts right down to sea level, a geological feature from the last ice age.



In 1903 Britain's oldest complete human skeleton was found here: a 9000 year old stone-age era inhabitant of the area.

In 1997 mitochondrial DNA sampling of the town found someone who was a direct descendant on the maternal side.

That's fascinating on it's own: some of the people living in the town are descendants of the same people that lived their nine thousand years earlier. Whatever changes happened in the UK: romans, saxons, vikings, whatever, they just lived through it all. And they came up with what can be -if done properly- a most excellent cheese.

Footnote: cavers cleared out the blocked sinkholes earlier this week and the road is now intermittently open.

Crater Lake: T+11

Following on from my "Page Mill T+20" trip, in late August we ended up Crater Lake for the Corvallis "Mid Valley Bicycle Club" annual circuit of the lake.

2001:



and 2012

The colours are different as in 2012 the Lassen fires 80 miles to the south are adding a light smoke to the air.

The original picture was taken with a Sony camera, 2048*1536; 3 Megapixels. The resolution is less than my desktop monitor, which makes it appear grainy as a background.

in 2012:, the original size of 4000*3000 means four times as many dots; the panasonic compact has a leica lens and makes up for the loss of a viewfinder by the ability to display a grid + diagonals over the image, to increase P(horizontal(horizon)).

In August 2001, Bina was 5 months pregnant; now our son is 10 and did the loop on a tandem, working with Mike Wilson, who races in the PNW CX circuit in the category below his age to make it more challenging. That may seem to have given Alexander some help -but it also meant that he was made to do it at a fairly aggressive pace, with none of this resting business.


I did it on a borrowed MTB, with knobbly tires, and took a couple of detours to add 12+ miles to my route. Even so, compared to Alexander, I look suspiciously tired.



I got back to the campground (because of those detours, honest) about a hour after him, tired, needing my rest and refreshments.

What is the ten year old doing? Running around chasing chipmunks. Then he comes over and tries to steal my beer.



That's it then -isn't it? I may as well retire now.

Were it not for the fact that university education is becoming so expensive that my son will need a large amount of cash to get through it then I have no further contribution to make towards my DNA's survival.

Disclaimer: I reserve the right to not know what I'm talking about

There have been some links to this blog with the implications that either because I am a researcher at HP Laboratiories with commit rights to a number of projects -including Apache Hadoop- then my opinions may have some value.

Not so.

Everything I say is obviously my own opinions, do not reflect those of my employer, may be at odds with corporate strategies of which I am blissfully unaware of, if I do mention corporate plans they may change on a whim, etc, etc.

I do work at HP Labs -which is a fun place to work- but think about what that means. It means that I don't do stuff that goes straight into shipping products. The code I write is not considered safe for humanity. I may talk about it, I may write about it, some may get into the field, but only after rigorous review by people who care about such things.

This makes it a bit like a university, except that I don't teach courses. Cleary I am not considered safe near students either, in case I damage thier minds.

"Oh well", people say, "he's a Hadoop committer -that must count for something?"

It does, but I don't get to spend enough time there to keep up with the changes, let alone do interesting stuff of my own ideas.

Whenever I create an issue and say "I will do this", other people in the project -Arun, Tom, Todd and others will stare at their inbox in despair, with the same expression on their faces that my family adopts when I say "I will bake something".

Because then, yes, I may come up with something unusual and possibly even tasty, but they will know that the kitchen will be left with a thin layer of flour over everything, there will be pools of milk, and whatever went in the oven went into the closest bowl-shaped thing I could reach with one hand, rather than anything designed to be used in an oven, let alone watertight. Even if I didn't use flour or milk.

Bear that in mind before believing anything written here.

[Photo:me somewhere in the Belledone mountains; French Alps. For some reason French villages never like it when people on bicycles come to a halt]