Double a penny once a day and you reach $1 million in less than a month. That is what exponential growth looks like with a base of two. As contact chaining steps through its hops, the social graph grows much faster. If the average person calls or is called by 10 other people a year, then each hop produces a tenfold increase in the population of the NSA’s contact map. Most of us talk on the phone with a lot more than 10 others. Whatever that number, dozens or hundreds, you multiply it by itself to measure the growth at each hop.
Contact chaining on a scale as grand as a whole nation’s phone records was a prodigious computational task, even for Mainway. It called for mapping dots and clusters of calls as dense as a star field, each linked to others by webs of intricate lines. Mainway’s analytic engine traced hidden paths across the map, looking for relationships that human analysts could not detect. Mainway had to produce that map on demand, under pressure of time, whenever its operators asked for a new contact chain. No one could predict the name or telephone number of the next Tsarnaev. From a data scientist’s point of view, the logical remedy was clear. If anyone could become an intelligence target, Mainway should try to get a head start on everyone.
Mainway’s purpose, in other words, was neither storage nor preparation of a simple list. Constant, complex, and demanding operations fed another database called the Graph-in-Memory.
In this extract from Dark Mirror, Barton Gellman discusses the NSA’s development of Mainway, the database developed by the NSA uncovered by Edward Snowden. He explains the way in which contact chaining was used to develop a comprehensive social graph.