Well this whole Prism business has been quite interesting from a personal perspective, especially this “metadata” business.
A long time ago, though not actually that far away from where I am now, I worked as a junior technical writer at a software company called Harlequin (later Xanalys). I worked in the intelligence systems division, and had been involved in some very cool projects for things like crime mapping, network analysis, and homicide case management. We had some great news clippings on the office walls of crimes solved using our technology.
One day my supervisor informed me that I needed to update the manual for one of the company’s more popular but least-liked products, something called CaseCall. Everyone within the company hated CaseCall as far as I could tell, and after a few days working on it I could tell why.
What CaseCall did was basically automate its way around some otherwise quite sensible restrictions on police extracting metadata from telecommunication providers.
In principle, any investigating officer could get in touch with any provider and ask for details about who-called-whom over a particular time period and analyse the data, but in practice not many did because the law put in place a number of steps you needed to go through to have your request approved.
What CaseCall did was turn things like “The absence of this information will prejudice this investigation because…” into a drop-down list of boilerplate non-answers so that the officer could press the submit button at one end, and the service provider press the accept button at the other, and the metadata could flow into the very clever analysis tools that the company had developed (and indeed still sell today).
Harlequin won the first Big Brother award for a product in 1998 for CaseCall. Sadly, no-one from the company went to collect it, as it would have looked great in the office:
Product: Software by Harlequin that examines telephone records and is able to compare numbers dialled in order to group users into ‘friendship networks’ won this category. It avoids the legal requirements needed for phone tapping.
(Friendship network analysis in 1998! Pretty good, huh? Mark Zuckerberg would have been about 14 around then.)
Basically, what we were doing was avoiding the whole business of phone tapping and collecting content, and instead going after metadata. After all, the metadata was usually sufficient to identify important network nodes, identify useful patterns of behaviour, and corroborate other types of information acquired by other means, such as interviews and field officer reports.
The metadata was actually in many ways more useful than the “data” (the content of the phone calls in this case), which would have taken a lot of work to transcribe and analyse, and may not have actually provided much more analytic value than the metadata alone. (It was great to read this little sketch by Cory Doctorow about metadata today which kind of makes the same point.)
So don’t underestimate “metadata”..!