Television Viewing & the Deanonymization of Large Sparse Datasets.

[preamble: this is not me writing against collecting data analysing user behaviour, including Tv viewing actions. I cherish the fact that Netflix recommends different things to different family members, and I'm happy for the iPlayer team to get some generic use data and recognise that nobody actually wants to watch Graham Norton purely from the way that all viewers stop watching before the introductory credits are over. What is important here is that I get things in exchange: suggestions, content. What appears to be going on here is that a device I bought is sending details on TV watching activity so as to better place adverts on a a bit of the screen I paid for, possibly in future even interstitially during the startup of a service like Netflix or iPlayer. I don't appear to have got anything in exchange, and nobody asked me if I wanted the adverts let alone the collection of the details of myself and my family, including an 11 year old child.]




Just after Christmas I wandered down to Richer Sounds and bought a new TV, first one in a decade, probably second TV we've owned since the late 1980s. My goal was a large monitor with support for free to air DTV and HD DTV, along with the HDMI and RGB ports to plug in useful things, including a (new) PS3 which would run iPlayer and Netflix. I ended up getting a deeply discounted LG Smart TV as the "smart" bits came with the monitor that I wanted.

I covered the experience back in March, where I stated that I felt that smart bit was AOL-like in its collection of icons of things I didn't want and couldn't delete, it's dumbed down versions of Netflix and iPlayer, and its unwanted adverts in the corner. But that's it, the netflix tablet/TV integration compensates for the weak TV interface, and avoids the problem of PS3 access time limits on school nights, as the PS3 can stay hidden until weekends.



Last week I finally acceded to the TV's "new update available" popups, after which came the "reboot your TV" message. Which I did, to then get told that I had to accept an updated privacy policy. I started to look at this, but after screen 4 of 20+ gave up, mentioning it briefly on that social networking stuff (who give me things like Elephant-Bird in exchange for their logging my volunteered access -access where I turn off location notification in all devices).

I did later regret not capturing that entire privacy policy by camera, and tried to see if I could find it on line, but at the time, the search term "LG SmartTV privacy policy" returned next to nothing apart from a really good policy for the LG UK web site, which even goes into the detail of identifying each cookie and its role. I couldn't see the policy after a quick perusal of the TV menus, so that was it.

Only a few days later, Libby Miller pointed me at an article by DoctorBeet, who'd spun wireshark up to listen to what the TV was saying, and so showing how his LG TV is doing an HTTP forms  POST to a remote site of every channel change, as well as details on filenames in USB sticks.

This is a pretty serious change on what a normal television does. DoctorBeet went further and looked at why. Primarily it appears to be for advert placement, including in that corner of the "smart" portal, or a start time after you select "premium" content like iPlayer or netflix. I haven't seen that which is good -an extra 1.5MB download for an advert I'd have to stare through is not something I'd have been happy with.

Anyway, go look at his article, or even a captured request.

I'm thinking of setting up wireshark to do the same for an evening. I made an attempt yesterday but as the TV is CAT-5 to a 1Gbs hub, then an ether over power bridge to get into the base station, it's harder than I'd thought. My entire wired network is on switched ports so I can't packet sniff, and the 100 MB/s hub I dredged up from the loft turned out to be switched too. That means I'd have to do something innovative like use the WEP-only 802.11b ether to wifi bridge I also found in that box, hooked up to an open wifi base station plugged into the real router. Maybe at the weekend. A couple of days logs would actually be an interesting dataset even if it just logs PS3 activity hours as time-on-HDMI-port-1

What I did do is go to the "opt out of adverts" settings page DoctorBeet had found, scrolled down and eventually followed some legal info link to get back to the privacy settings. Which I did photo this time, and which are now up on Flickr.

Some key points of this policy

Information considered to be non personally identifiable include MAC addresses and "information about the live content you are watching"






That's an interesting concept, which I will get back to. for now. note that that specific phrase is not indexed anywhere into BigTable, implying it is not published anywhere that google can index it.


Or "until you sit through every page with a camera this policy doesn't get out much"

If you have issues, don't use the television



That's at least consistent with customer support.

Anyway. there's a lot more slides. One of them gives a contact, who when  you tap in to LinkedIn not only shows that he's the head of legal at LGE UK,  that he's one hop away from me: datamining in action.

Now, returning to a key point: Is TV channel data Non-personal information?

Alternatively: If I had the TV viewing data of a large proportion of a country, how would I deanonymize it?

The answer there is straightforward, I'd use the work of [2004 Arvind Narayanan and Vitaly Shmatikov], Robust De-anonymization of Large Sparse Datasets.

In that seminal paper, Narayanan and Shmatikov took the anonymized Netflix dataset of (viewers->(movies, rankings)+), and deanonymized it by comparing film reviews on Netflix with IMDb reviews, looking for reviews that appeared on IMDb shortly after a Netflix review with ratings matching/close to that a Netflix review. They then took the sequence of a viewers' watched movies and looked to see if a large set of their Netflix review met that match critera. At the end of which they managed to deanonymize some Netflix viewers -correlating them with an IMDb reviewer may standard deviations out from from any other candidate. They could then use this  match to identify those movies which the viewer had seen and yet not reviewed on IMDb.

The authors had some advantages, both netflix and IMDb had reviews, albeit on a different scale. the TV details don't so the process would be more ad-hoc

1. Discard all events that aren't movies
2. Assume that anything where the user comes in late to some threshold isn't a significant "watch event" and discard.
3. Assume that anything where the user watches all the way to the end is a significant "watch event" and may be reviewed later.
4. Assume that watching events where the viewer changes channel some distance into a movie -say 20 min- as a significant watch failure event, which may be reviewed negatively.
5. Consider watch events where the user was on the same channel for some time before the movie began as less significant than when they tuned in early.
6. If information is collected when a user explicitly records a movie, a "recording event", that is treated even more significantly.
7. Go through the IMDb data looking for any reviews appearing a short time after a significant set of watch events, expecting higher ratings from significant watch events and recording events, and potentially low ratings from a significant watch failure.

I don't know how many matches you'd get here -as the paper shows, it's the real outliers you find, especially the watchers of obscure content.

Even so, the fact that it is would to possible to identify at least one viewer this way shows that TV watching data is personal information. And I'm confident that it can be done, based on the maths and the specific example in the Robust De-anonymization of Large Sparse Datasets paper.

Conclusion: irrespective of the cookie debate, TV watching data may be personal -so the entire dataset of individual users must be treated this way, with all the restrictions on EU use of personal data, and the rights of those of us with a television.

Foreign News

The cracks all the way to the top of the small feudal island-state of Great Britain became visible this week, as a show trial and exposure of police and state security activities exposed the means the regime retains power.



For centuries Britain has endured a caste system, where those at the bottom had little education or career prospects, while those in the ruling "upper class" lived an entirely separate life -a life that began with a segregated education from their school, "eton", to their universities, oxford and cambridge and then employment in "the city" or political power in "parliament". Similar to the French Polytechniques system, while it guarantees uniformity and consistency amongst the hereditary rulers, the lack of diversity reduces adaptability. Thus the elite of this island have had trouble leading it out of the crises that have befallen it since 2008 -when it became clear that it offshore tax-haven financial system had outgrown the rest of the country. The emergency measures taken after the near-collapse of the countries economy have worsened the lives for all outside a small elite -exacerbating the risks of instability.

This month some of the curtains on the inner dealings of that ruling oligarchy were lifted, giving the rest of the country a glimpse into the corrupt life of the few. A show trial of the editors of a newspaper showed how the media channels -owned by a few offshore corporations- were granted free reign by the rulers, in exchange for providing the politicians with their support and the repetition of a message that placed the blame for the economic woes on the previous administration and outgroups such as asylum seekers and "welfare scroungers".

A disclosure of how the media were creating stories based on intercepting the voicemail messages of anyone of interest forced the government to hand a few of the guilty to the legal system -while hoping that the intiminate relationship between these newspaper editors and those in government do not get emphasised. Even so, this scandal has already forced the government to postpone approving a transaction that would give a single foreign oligarch, Murdoch, near absolute control of television and the press. Public clamour for some form of regulation of the press has also forced the regime to -reluctantly- add some statuatory limitations to their actions. It remains to be see what effect this has -and whether the press will exact their revenge on the country's rulers.

A few miles away, in the country's "parliament", the MPs exercised some of their few remaining privileges of oversight. The "plebgate" affair represented a case in which the feared police, "the Met" were grilled over their actions. Normally the Met is given a free hand to suppress dissent and ensure stability across the lower castes, but in "plebgate" the police were caught on CCTV and audio recordings making false accusations about one of the rulers. The thought that "the Met" could turn on their masters clearly terrifies them: the grilling of the police chiefs represents the public part of a power struggle to define who exactly is in charge.

Alongside this, the heads of the state security apparatus were interviewed over the increasingly embarrasing revelations that they had been intercept the electronic communications of the populace of the country, "the subjects" as they are known. This comes as no surprise to the rulers, who recognise that with the mainstream media being part of the oligarchy, any form of organised dissent will be online. Monitoring of facebook and google is part of this -during the 2011 civil unrest, calls even were made by the press and politicians to disable some of these communications channels. Again, the rulers have to walk a fine line between appearing concerned about these revelations, while avoiding worsening those relationships which are critical for keeping the small hereditary elite in power.

Given the interdepencencies between the rulers, the press and the state security forces, no doubt these cracks will soon be painted over. Even so, irrespective of the public facade, it may be a while before the different parts of what is termed "the establishment" trust each other again.

Maverick and Applications

One action this week was a full OS/X roll on three boxes; fairly event free. The main feature for me is "better multiscreen support". There's also now an OS/X version of Linux's Powertop; as with powertop is more of a developer "your app is killing the battery" than something end users can actually do anything with -other than complain.

The other big change is to Safari, but as I don't use that, it's moot.

The fact that its a free upgrade is interesting -and with Safari being a centrepiece of that upgrade, maybe the goal of the upgrade is to accelerate adoption of the latest Safari and stop people using Firefox & Chrome. The more market share in browsers you have, the more web sites work in it -and as Safari is only used on a macs, it can't have more desktop browser market share than the market share apple have in the PC business itself. A better Safari could maximise that market share -while its emphasis on integration with iPads and iPhones rewards people who live in the single-vendor-device space, making us owners of Android phones feeling left out.

One offering that did get headlines was "Free iWork", but that turns out to be "Free on new systems"; if you have an existing OS/X box, you get to pay $20 or so per app -same as before.

Except, if you go to the apple app store, the reviews of the new suite from existing users are pretty negative: dumbed down to the point where users with existing spreadsheets, documents and presentations are finding things missing -where in keynote a lot of the fancy "make your presentations look impressive" features are gone.

They're not going to come back, now that iWork is a freebie.

If the NRE costs of maintaining iWork are now part of the cost of the Mac -and OS upgrades are going the same way. Even if apple maintain market share and ASP margins over Windows PCs, the software stack costs have just gone up.

Which means those applications have gone from "premium applications with revenue through the app store", with a business plan of "be compelling enough to sell new copies as well as regular upgrades from out existing customer base", to "bundled stuff to justify the premium cost of our machines".

That's a big difference, and I don't see it being a driver for the iWork suite being enhanced with features more compelling to the experts.

Where apple are likely to go is cross-device and apple cloud integration, again to reward the faithful single-vendor customers. Indeed, you do get the free apple iCloud versions of the iWork apps, which look nice on Safari -obviously. Apple's business model there: upsell storage, does depend on storage demand, but the harsh truth is, it needs a lot of documents to use up the 4GB of free storage. Photographs, now, they do take up space, which clearly explains why the new iPhoto has put work in iPhoto to iCloud sharing. Yet it does still retain Flickr sharing, which, with 1TB of storage, must be a competitor to iCloud for public photos, while facebook remains a destination for private pics.

I wonder whether that Flickr uploader will still be there the next time Apple push out a free update to the OS and applications
 
[photo: a line of Tuk Tuks, Tanzania/Kenya Border]

Hadoop 2: shipping!

The week-long vote is in- Hadoop 2 is now officially released by Apache!

Anyone who wants to use this release should download Hadoop via the Apache Mirrors.

Maven and Ivy users: the version you want to refer to is 2.2.0

The artifacts haven't trickled through to the public repos as of 2013-10-16-11:18 GMT -they are on the ASF staging repo and I've been using them happily all week.

This release marks an epic of development, with YARN being a fundamental rethink of what you can run in a Hadoop cluster: anything you can get to run in a distributed cluster where failures will happen, the Hadoop FileSystem the API for filesystem access  -be it in Java or a native client- and data is measured by the Petabyte.

YARN is going to get a lot of press for the way it transforms what you can do in the cluster, but HDFS itself has changed a lot. This is the first ASF release with active/passive HA -which is why Zookeeper is now on the classpath. CDH 4.x shipped with an earlier version of this- and as we haven't heard of any dramatic data loss events, consider it well tested in the field. Admittedly, if you misconfigure things failover may not happen, but that's something you can qualify with a kill -9 of the active namenode service. Do remember to have to >1 zookeeper instance before you try this -testing ZK failure should also be a qualification process. I think 5 is a number considered safer than 3, though I've heard of one cluster running with 9. Nobody has admitted going up to 11.

This release also adds to HDFS

1. NFS support: you can mount the FS as an NFS v3 filesystem. This doesn't give you the ability to write to anywhere other than the tail of a file -HFDS is still not-Posix. But then neither is NFS: its caching means that there is a few seconds worth of eventual consistency across nodes (\cite{Distributed Systems, Colouris, Dollimore & Kindberg, p331}).,
2. Snapshots: you can snapshot some of a filesystem and roll back to it later. Judging by the JIRAs, quotas get quite complex there. What it does mean is that it is harder to lose data by accidental rm -rf operations.
3. HDFS federation: datanodes can store data for different HDFS namenodes, -Block Storage is now a service- while clients can mount different HDFS filesystems to get access to the data. This is something of primarily of relevance to people working at Yahoo! and facebook scale -everyone else can just get more RAM for their NN and tune the GC options to not lock the server too much]

Hadoop 2 also adds is extensive testing all the way up the stack. In particular, there's a new HBase release coming out soon -hopefully HBase 0.96 will be out in days. Lots of other things have been tested against it -which has helped to identify any incompatibilities between the Hadoop 1.x MapReduce API (MRv1) and Hadoop 2's MRv2, while also getting patches into the the rest of the stack where appropriate. As new releases trickle out, everything will end up being built and qualified on Hadoop 2.

Which is why when you look at the features in Hadoop 2.x, as well as headline items "YARN, HDFS Snapshots, ...", you should also consider the testing and QA that went into this -this is the first stable Hadoop 2 release -the first one extensively tested all the way up the stack. Which is why everyone doing that QA -my colleagues, Matt Foley's QA team, the Bigtop developers, and anyone else working with Hadoop that built and tested their code against Hadoop 2.1 beta and later RCs -and reported bugs.

QA teams: your work is appreciated! Take the rest of the week off!

[photo: Deamz and Soker at St Philips, near Old Market]

Hadoop: going way beyond Google

Lars George has some nice slides up on Hadoop and its futures Hadoop is dead, long live Hadoop! -but it shows a slide that I've seen before and it concerns me

It's on p39 where's a list of papers published by google tying them in to Hadoop projects, implying that all Hadoop is is a rewrite of their work. While I love Google Research papers, they make great reads, we need to move beyond what Google's published work, because that strategy has a number flaws

Time: with the lag from google using to writing about it being 2+ years, plus the time to read and redo the work, that's a 4 year gap. Hardware has moved on, the world may be different. Googles assumptions and requirements need to be reassessed before redoing their work. That's if it works -the original MR paper elided some details needed to get it to actually work [1].

Relevance outside google. Google operate -and will continue to operate- at a scale beyond everyone else. They are doing things at the engineering level -extra checksums on all IPCs- because at their scale the probability of bit errors sneaking past the low-level checksums is tangible. Their Spanner system implements cross-datacentre transactions through the use of GPS to bound time. The only other people doing that in public are telcos who are trying to choreograph time over their network.

Its good that google are far ahead -that does help deal with the time lag of implementing variants of their work. When the first GFS and MR papers came out, the RDBMS vendors may have looked and said "not relevant", now they will have meetings and presentations on "what to do about Hadoop". No, what I'm more worried about is whether there's a risk that things are diverging -and its cloud/VM hosting that's a key one. The public cloud infrastructures: AWS, Azure, Rackspace, show a different model of working -and Netflix have shown how it changes how you view applications. That cloud is a different world view: storage, compute, network are all billable resources.

Hadoop in production works best on physical hardware, to keep costs of storage and networking so low -and because if you try hard you can keep that cluster busy, especially with YARN to run interesting applications. Even so we all use cloud infras too because they are convenient. I have a little one-node Hadoop 2.1 cluster on a linux VM so I can do small-scale Hoya functional tests even when offline. I have an intermittent VM on rackspace so I can test the Swift FileSystem code over there. And if you do look at what Netflix open source, they've embraced that VM-on-demand architecture to scale their clusters up and down on demand.

It's the same in enterprises: as well as the embedded VMWare installed base, OpenStack is getting some traction, and then there is EC2, where dev teams can implement applications under the radar of ops and IT. Naughty, but as convenient as getting a desktop PC was in the 1980s. What does it mean? It means that Hadoop has to work well in such environments, even if virtual disk IO suffers and storage gets more complex.


Other work. Lots of other people have done interesting work. If you look at Tez, it clearly looks at the Dryad from MS Research. But there's also some opportunities to learn the Stratosphere project, that assume a VM infrastructure from the beginning -and build their query plans around that.

Google don't talk about cluster operations.

This is really important. All the stuff about running google's clusters are barely hinted at most of the time. To be fair, neither does any one else, it's "the first rule of the petabyte club".

[Boukharos08] GCL Viewer - A study in improving the understanding of GCL programs. This is a slightly blacked out paper discussing Google's General Config Language, a language with some similarities to SmartFrog: templates, cross-references, enough operations to make it hard to statically resolve, debugging pain. That paper looks at the language -what's not covered is the runtime. What takes those definitions and converts it to deployed applications, applications that may now span datacentres?

[Schwarzkopf13]: Omega: flexible, scalable schedulers for large compute clusters. This discusses SLA-driven scheduling in large clusters, and comes with some some nice slides.

The Omega paper looks at the challenge of scheduling mixed workloads in the same cluster, short-lived analytics queries, longer processing operations: PageRank &c, and latency-sensitive services. Which of course is exactly where we are going with YARN -indeed, Schwarzkopf et al [2] cover YARN, noting that it will need to support more complex resource allocations than just RAM. Which is of course, exactly where we are going. But what the Omega paper doesn't do is provide any easy answers -I don't think there are any, and if I'm mistaken then none of the paper's authors have mentioned it [3].

Comparing YARN to Omega, yes, Omega is ahead. But being ahead is subtly different from being radically new: the challenge of scheduling mixed workloads is the new problem for us all -which is why I'm not only excited to see a YARN paper accepted in a conference, I'm delighted to see mentions of Hoya in it. Because I want the next Google papers on cluster scheduling to talk about Hoya [4].

Even so: scheduling is only part of the problem: Management of a set of applications with O(1) ops scaling is the other [5]. That is a secret that doesn't covered, and while it isn't as exciting as new programming paradigms for distributed programming [6] it is as utterly critical to datacentre-scale systems as those programming paradigms and the execution and storage systems to run them [7].

Where else can the Hadoop stack innovate? I think the top of the stack -driven by application need is key -with the new layers driven by the new applications: the new real-world data sources, the new web applications, the new uses of data. There's also the tools and applications for making use of all this data that's being collected and analysed. That's where a lot of innovation is taking place -but outside of twitter, LinkedIn and Netflix, there's not much in the way of public discussion of them or sharing of the source code. I think companies need to recognise the benefits of opening up your application infrastructure (albeit not the algorithms, datasets or applications), and get it out before other people open up competitive alternatives that reduce the relevance of their own project.



[1] I only wrote that to make use of the word elision.
[2] The elison of the other authors to "et al" is why you need 3+ people on a paper if you are the first author.
[3] This not because I'm utterly unknown to all of them and my emails are being filtered out as if they were 419 scams. I know John from our days at HP Labs.
[4] Even if to call me out for being mistaken and vow never to speak to me.
[5] Where the 1 in the O(1) has a name like Wittenauer.
[6] Or just getting SQL to work across more boxes.
[7] Note that Hortonworks is hiring, we'd love people to come and join us on these problems in the context of Hadoop -and unlike Google you get to share both your work and your code.

[Photo: Sepr on Lower Cheltenham Place, Montpelier]

I for one will not defend our nation's critical infrastructure home wifi base stations

Apparently the UK government is planning to spend lots of money building a "Cyber National Guard", which means that a "reserve guard" of civilians will be available on call when the military needs them to organise "cyber strikes" against enemies or defend the national infrastructure.

Or as the Daily Mail (remember, it's not a real paper and has a history of supporting fascism) says:

A new ‘Cyber National Guard’ of part-time reservists will be open to computer whizzkids who cannot pass the current Territorial Army fitness tests, on the basis that press-ups do not aid computer skills. ‘A TA for computer geniuses’, as Mr Hammond called it.

He poured scorn on ‘crude and bonkers attacks by armchair generals’ who have criticised him for cutting the number of soldiers – and made it clear conventional forces faced more cuts in the switch.

In theory I meet some of the criteria, even if I am fitter than the sneering prejudices of the Daily Mail would think.

• I know a lot about computers
• I was the first person ever to publicly demonstrate a signed ActiveX-in-IE security hole.I know how to use WireShark to read HTTP and TCP conversations.
• I used to be able to read x86 ASM (there was some in the film Elysium, entertainingly enough)
• 1. I'm good at breaking things, including large Distributed Systems.
• I'm down as co-author on a patent on how to use a subverted battery to talk to a subverted BIOS to get all your keystrokes without the OS getting a lookin

However, this does not make me suitable for some national-cyber-guard-thingy because if you look at the one documented instance of a nation state committing a (peacetime) over-the-net attack on another nation state, Olympic Games, it's clear that this project took person-years of effort to come up with a virus so subtle it could make use of multiple 0-day exploits to get into Windows, then trickle over to the SCADA-managed industrial machinery by way of USB sticks that were neither near empty or near full (to make the loss in capacity less obvious). Once there, recognise the characteristics of an iranian enrichment centrifuge, change their spin rates to destroy them -all the while reporting valid parameters to the ops team. That's not a activity of some weekend developers: That is the R&D spend of a goverment, the integration of gained knowledge of the Iranian enrichment process, the ability to write code to destroy it -the testing of that on real hardware, and the transport mechanism using 0-day exploits and a forged signing certificate. That is what the future of inter-nation-state conflict over the net looks like, and it doesn't depend on script-kiddies running metasploit.

The classic reserve forces model trains people, part time, to be soldiers, a process which roughly consists of

1. learning how to avoid getting killed.
2. learning how to shoot at things and people
3. learning how to follow orders even when the outcome involves a non-0 zero probability of becoming a KSI statistic.
4. learning how to use skills 1 & 2 to achieve goals the person shouting the orders wants to achieve.

That training and learning scales well, as shown in the twentieth century by two global conflicts and the Korean war. Teaching people how to code malware to infiltrate and damage other government's national and commercial infrastructure does not. What does that leave? Botnets are designed to be O(1) scaling, so you don't need regiments of engineers there. Unless it is just script-kiddie work rummaging around opposing computing facilities -but that's something best done in peacetime, to a relaxed schedule (which is presumably why the Chinese govt. do appear to have associates doing that).

As for myself, unless I am needed to write JUnit tests to break the North Korean missile program, well, I'm not useful.

Maybe, therefore, it's not military attack the army wants, it's defending the nation's critical infrastructure.

Which is what exactly?

Because if its the network, then what you need is not on skills in things like configuring telco-scale switches, it's having that network set up with as much intrusion and anomaly detection as possible, with the people on call to handle the support calls. You aren't going to be able keep part time computer people around to field calls on that if all they know about network security is that turning off UPNP on the home router is a good idea.

No, network management at the government scale is a skill for the few, not the many. That doesn't mean that those people who do have to look after corporate and national intranets shouldn't be up to date with current tools and thinking w.r.t. defending critical network security from nation states, of which a key one is don't buy routers from a company owned by the Chinese Army.

Beyond the network, well there's the many desktops out there. I can just about defend my home set of machines by Romanian Botnet gangs, primarily by disabling most browser plugins, updating flash weekly and not running code I don't trust. I also have to defend my home passwords from an 11 year old. Neither skill will give me a chance to defend my machines against a nation state, not unless the attack by the state in question involves a small boy looking over your shoulder as you type in the password to give him extra time on the machine. In that case -and only in that case- would my advice -use some uppercase characters and learn to touch type- would be of use.

But going round locking down PCs by deleting security risks like Acroread? Enforcing password policies? These are IT dept things, not something for a team of reservist-cyber-warriors. Even there though, the threat posed by foreign governments sending spear-phished PPT documents containing ActiveX controls with 0-day exploits in them (ActiveX is derived from Ole Control Extensions which is built on top of Object Linking and Embedding, all atop the COM common object model). Once the unsuspecting slideware gets opened, whatever payload it goes on to download.


Where does that leave? If the outer network is the netops, the deskopt the PC IT dept, what's left? The applications.

It means designing and building applications that don't let you steal millons if you attach a KVM switch to an employee's desktop.

It means designing databases apps so that SQL injection attacks never work, irrespective of how the input comes in, and validation of that data at entry time, in flight and when it is stored in the database -so that corruption is detected.

It means having some way of backing up application state securely, so that if damage is done to it, then you can recover.

It means thinking about security in a more complex way than generic "user" -with different levels of access to different things, and it means having a defensible audit trail so that if someone were to download large quantities of an organisations files and stick them up
on wikileaks or hand them to a paper -at least you know what's been taken.

From that perspective, people like myself are more generally useful if we do actually make the things we code as securely as possible, and put in audit trails on the off chance that we can't. And it implies that the government would be better of spending money teaching us to understand Kerberos as well as other aspects of secure computing, rather than having some pool of script-kiddies whose skill stops at metaspoit and wireshark, and whose networking understanding stops at home routers. Of course there the problem lies that some of the course matter needed to keep those foreign nation states at bay, "don't trust HTTPS or SSH with 1024 bit keys" may start the audience asking questions you don't want asked.

Overall, these proposals -as presented in the newspapers- appear to be naively unrealistic and based on a complete misunderstanding of how you can both attack and defend core computing infrastructures -as well as who is capable and responsible for doing so.

Why then do they appear?

I can see a number of possibilities

1. Null Hypothesis: Politican is clueless, and said things they don't understand;
2. Politician reported someting realistic, but reporter clueless and reached a conclusion that makes no sense
   
3. Both Politican and Reporter clueless leading to a complete misunderstanding, where the reporter didn't realise
4. the politician was talking bollocks and then exaggerated even that
5. Politician knew that what he was talking about was utterly unrealistic, but did it in an attempt to distract the audience and justify other actions

Given the structure of the talk -to offset cuts in the core "shoot things, achieve goals ordered to achieve, come back alive" bit of the army and reserve forces -he's probably doing #5: divert and justify those actions. But the scary thing is: he may actually believe what he's been saying.

How to update an ubuntu 12.x box to protoc 2.5.0 and so build Apache hadoop 2.x+

Here are some working notes on how to push up an Ubuntu 12.x machine to a version of protoc that builds Hadoop. It is a shame that protobuf.jar doesn't include a protoc compiler
Actually it's also a shame that protobuf.jar isn't backwards compatible across versions. I'm
now ooking at Apache Thrift and thinking "maybe I should adopt that in my own code".


Run these as root:

cd ~
apt-get -y remove protobuf-compiler
curl -# -O https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
gunzip protobuf-2.5.0.tar.gz 
tar -xvf protobuf-2.5.0.tar 
cd protobuf-2.5.0
./configure --prefix=/usr
make
make install
protoc --version

Then log in as you and again run

protoc --version

The output of the --version commands must be "libprotoc 2.5.0". Any error; any different version: your build of Hadoop won't.