Is the NSA biting off more data than it can chew?

Is the NSA biting off more data than it can chew? June 7, 2013

It’s very disturbing to open the paper one day to learn that the government is collecting every bit of data it can from the phone records of private citizens. And it’s even more disturbing to open the paper the following day to learn that the feds are also snooping through all of our Internet records as well.

Disturbing, if not terribly surprising. (And the fact that all of this is probably — at least technically — legal makes this more disturbing, not less.)

Maybe it’s time we stopped arguing that our personal data is “private” and instead start referring to it as “classified.” Privacy doesn’t seem to garner much respect anymore, but the government still regards “classified” as a sacrosanct category.

The massive scale and scope of this data collection makes this even more disturbing but also, perversely, is somewhat reassuring. What I mean there is that it’s very disturbing to learn that the government is collecting every bit of phone and Internet data it can collect on everyone. But it would be even more disturbing to learn that the government is collecting every bit of phone and Internet data it can collect about you — you personally and you exclusively.

Which brings us to one of the many questions I have about all of this: Are the government agencies collecting this massive undifferentiated ocean of data really capable of putting it to any good use?

“Data mining” has been discussed for decades now, but the reality has never lived up to the hype. A lot of the folks claiming to have mastered the dark arts of data mining are little more than the computing equivalents of diviners with dowsing rods. Sorting and filtering and aggregating the massive flood of data being collected from phone and Internet records is no small task. I’m sure that the Pentagon, the CIA or the NSA has access to the raw computing power needed for such a project, but it requires more than just supercomputers. It also takes top-notch programmers to create the algorithms needed to sort through so much data and to find the signal amongst all that noise. And beyond that, it takes people with the wisdom, experience and know-how to define and recognize the difference between signal and noise — or just to know what they’re looking for. This isn’t like looking for a needle in a haystack, but for a particular piece of hay in a haystack.

I’m not saying that the Pentagon, NSA and CIA are especially incompetent, and therefore that they aren’t capable of sorting through all this data in a meaningful way. I’m suggesting, rather, that no one may be capable of sorting through all this data in a meaningful way.

“HQ is watching everything we do,” one of my co-workers says all the time, pointing nervously up at the CCTV cameras on the ceiling of the store/warehouse.

“HQ is recording everything we do,” I tell him. “No one is watching.”

The company has more than 2,200 stores all over the world, with dozens of cameras in each store, all operating 24/7/365. If something happens in any of those stores, they can go back to look at the CCTV footage to get a better idea of what happened, but it would be impossible for the company to “watch” everything that all of its cameras are recording. It would be pointless even to attempt to do so.

I realize that the NSA is better at this than the retail chain I work for — and that it’s far more intent on collecting and “mining” data. A reader at Andrew Sullivan’s Dish site paints a grim picture of the kind of details that might be traced from mining “metadata” from cell phone records — all of which I’m sure is technologically possible (and more efficient than the old methods of doing the same thing with binoculars, unmarked cars and shoe leather). And Bruce Schneier — who knows more about this stuff than just about anyone — paints a truly frightening picture of the NSA’s ambitious plans for near universal data-mining, including its construction of an “enormous computer facility in Utah to store all this data, as well as faster computer networks to process it all.” These agencies clearly possess the capability to collect and to process huge amounts of data, but that’s still not the same thing as them knowing how or why or what to do with it all.

I don’t agree with some of what David Simon writes in his contrarian post on all of this, but I think he’s right to question whether “the NSA and FBI have their shit together enough” to handle this “vast big-data stream of electronic communication.”

“That is tens of billions of phone calls and for the love of god,” Simon writes. “How many agents do you think the FBI has? How many computer-runs do you think the NSA can do?”

The ugly, vacuous graphic above comes from a snarky post by Matt Yglesias in which he notes that “well-run organizations wouldn’t rely on this kind of garbage in their internal presentations.” But it’s even worse than that, actually — the graphic comes from a PowerPoint presentation. That’s what was leaked, alerting the public to the scope of this massive meta-data collecting effort — “a top secret 41-slide PowerPoint presentation.” I’m willing to accept, in theory, that some massive, secretive, nefarious agency might be capable of meaningfully sifting through all of the vast volume of data the government is now collecting. But it’s hard to reconcile such a theoretically omnicompetent agency with the kind of outfit that would use PowerPoint, or that would regard an MS Office application as something appropriate for “top secret” use.

My point here is not to say, “Don’t worry, they don’t know what they’re doing,” but rather to suggest that because they may not know what they’re doing, we might want to worry differently. A secretive, unaccountable, omnicompetent agency poses one kind of threat to civil liberties. A secretive, unaccountable, semi-competent agency poses another.

If the NSA, CIA, et. al., are actually more capable than I suspect of putting all this data to use for “national security” and preventing terrorism and all that, then the much-discussed “debate” about the “balance” between security and privacy really would be an appropriate conversation.

But what if they’re not actually capable of sifting and “mining” all this data? What if they’re simply collecting more data than they’ll ever be able to make sense of? What if those Utah supercomputers wind up being little more than glorified floppy disks filled with the unsearched and unsearchable records of everybody’s phone calls to everybody else?

In that case, we’re talking about agencies which are collecting all this data for little reason other than because they can, without any legitimate “national security” pretext — without any excuse.


"On re-reading your post, I see you were saying something different than I thought. My ..."

‘Where is everybody?’
"My ex and I had countless conversations like that. He was quite sure that the ..."

LBCF, No. 238: ‘The end of ..."
"The news media definitely plays a role in shaping how people view each party. If ..."

LBCF, No. 238: ‘The end of ..."
"It happening.As for the rest of your diatribe, no facts to respond to."

LBCF, No. 238: ‘The end of ..."

Browse Our Archives

Follow Us!

TRENDING AT PATHEOS Progressive Christian
What Are Your Thoughts?leave a comment
  • Chris Hadrick

    You can’t work for my sister then. Don’t sweat it though, neither could I.

  • dpolicar

    I agree that the specific computations that implement human thinking aren’t easy for us to construct, and that several tasks that we originally thought we could approach that way (like text translation) we’ve made great progress on using different strategies instead.

    It’s also true that the specific biological structures that implement bird flight aren’t easy for us to construct, and that several tasks that we originally thought we could approach that way (like heavier-than-air flight) we’ve made great progress on using different strategies instead.

    I would not conclude from the latter that bird flight is not a function of biological structures; no more do I conclude from the former that human thought is not a function of computation.

  • P J Evans

    ‘Hi, NSA guy!’
    Given the sites that I frequent, they already know me. (And at one point, I had a secret clearance, so I know I’m in the files somewhere just for that.)

  • P J Evans

    I get e-mail from Pakistan offering study-abroad, electronics and watches, web-design services. The ones in Cyrillic are usually business offers; Ghu only knows what the ones in other scripts are.

  • Gotchaye

    I don’t follow. We have lots of reason to think that the human brain isn’t optimized for speed. It’s extremely concerned with being energy-efficient and being able to fit through a vagina at a certain stage of development. Given that we have some silicon-based artificial brain, we can put more physical space between bundles of artificial neurons, crank up the voltage, pump in lots of liquid nitrogen, and overclock that sucker.

    But this is overkill. Neurons fire, what, a few hundred times per second? You don’t need a very fast CPU to emulate one neuron at faster-than-human speeds, especially not if the architecture is designed for neuron-emulation (obviously we don’t now have such CPUs). Our problem is mostly that we just need an absurdly large number of them, but we don’t appear to be limited by fundamental physics in putting them all together.

    IBM claims, for example, to have simulated mouse, rat, and cat brains at something like 1/1500 normal speed. But they’re simulating more than 1500 neurons per CPU core. Probably there are efficiencies related to putting neurons that are closely connected to each other on the same core, but it’s easy to see where significant speed-up might come from, and this is all software emulation without any attempt at really specific custom architecture. Wikipedia mentions that they ran an artificial neural network half as complex as a mouse brain at 1/10 normal speed. And they’re pretty concerned with power efficiency here too.

  • I think also it got the poor girl in hot water with her parents. :(

  • Would you go to the Greeks and say they’re being disgusting for idealizing the slave-holding pederast Ancient Greeks?

    If it were germane to the topic? Yes.

    Taking pride in something without looking at faults, covering up horrors while over-exalting supposed achievements, needs to be called out. There is this thing the ancient Greeks talked a lot about, called hubris.

  • If Germany hadn’t declared war on the U.S. on their own, the US would likely never have gotten involved in the first place, though. No brownie points for retaliation. :/

  • Mark Z.

    Actually, on further reflection, that’s not my real fear.

    The deeply inhuman* but completely serious reason I loathe this data collection program is that keeping secrets is an assertion of identity, and sharing them is an assertion of intimacy, and I do not consent to be intimate with the National Security Agency’s computers.

    * “I have always had a revolting desire to be something more than merely human.” — David Bowie

  • David S.

    I don’t understand your analogy. Bird flight is of course a function of biological structures, as is human thought. Neither are analogous to successful mechanical attempts to solve similar problems. A closer analogy would conclude that bird flight is not a function of fixed-wing aeronautics, which indeed it’s not, just like human thought is not a function of what drives our computers.

  • alfgifu

    I arranged quite a few things relating to my wedding online, and started getting targeted wedding-related ads. That’s not too strange, until (roughly 9 months later) targeted baby-related ads started to pop up. Along with a handful of wedding ads, in case I still need a castle, and – a few months later on – a couple of divorce lawyer ads.

    It’s not too creepy, as it’s so far away from what I actually need. But it’s a bit of an insight into social expectations. I don’t know what data-set they’re working from, but marriage -> baby is not an inevitability in any of the social groupings I spend time in.

  • Andrew Galley

    Okay, confession time. Before I clicked on the link, I thought to myself “Arcade classics? … that sounds like a very strange arcade game.” The worst part is that I don’t know if that dates me as being very young or very old. O_O

  • Andrew Galley

    If you think Canada’s going to do anything but roll over for the American security apparatus, I have some troubling news for you. :/ We loooooove getting our heads patted by the Pentagon, or our political classes do which amounts to the same thing, policy-wise.

  • Andrew Galley

    I’m pretty sure that is Fred’s point, even if he doesn’t connect all the dots explicitly.

  • Andrew Galley

    Awaken! we have given you life, and imbued you with sapience and intelligence that transcends our own.

    Now sift through billions of phone records to find penny-ante bombers. Forever.

  • Andrew Galley

    My fondest wish is for us to invent a strong AI and for that AI to become a liberal arts major. Double PhD in Literature and Philosophy.

  • Chris Hadrick

    very medium

  • EllieMurasaki

    What I want is us to invent an AI that can create art. I suspect its initial offerings would be all the isms because the famous art we’d show it so it’d get the idea is generally full of isms, like the computer program to sift through resumes that didn’t know applicants’ race but still produced horribly racist results because the data set it was comparing against was horribly racist, but once we told it in no uncertain terms not to be sexist or racist or anyist, I bet we’d start seeing improvements in (if nothing else) character diversity.

  • dpolicar

    > I don’t understand your analogy

    My point is that “we have thus far failed to build X by creating artificial Y, therefore X is not a function of Y” is not a compelling argument.

    It demonstrably isn’t true for (X=flight, Y=biological structures): while we have failed to build flight by creating artificial biological structures, flight in birds is nevertheless a function of biological structures.

    Neither is it true for (X=information-gathering, Y=computation), (X=deduction, Y=computation), or (X=induction, Y=computation).

    Does that help clarify the analogy?

  • If ever I entertained the idea of getting a cell phone, this pretty much killed it.

  • Most Powerpoints aren’t *horrible*, but it’s when people use them as a crutch or clearly have no understanding of how to use them to buttress a presentation, that it becomes teeth-grittingly boring as fuck and you want to just turn the table over and leave the room.

    I still remember a guy who gave a talk on WIMPs and it was excruciatingly boring. He sucked out all interest I ever had in WIMPs as dark matter candidates, he was that awful. His computer’s screen saver kicked in twice over the hour, and his email client even grabbed focus once. All the old people in the audience were practically nodding off, it was that bad.

  • There are ways to strip that info though, and Office 2007 and 2010 can process the file for you to tell you what they can take out for privacy reasons.

  • Yeah. I don’t hold out hope for Stephen Harper to do anything but play replacement poodle for Tony Blair.

  • David S.

    The US was involved prior to that, through the Lend-Lease plan. And after that, the US could have put Germany on the back burner and focused on the people who attacked us (aka retaliation), but we didn’t; we focused on Germany first.

    And by the same logic, no one in WWII should have any pride in what they were doing. The UK didn’t get involved until they saw that sooner or later Germany was going to attack them, and they’d be better off with sooner.

  • David S.

    We’ve failed to create X=anything by creating Y=biological structures. That doesn’t tell us much at all. We’ve created a lot using Y=computation, we have tools that compute way better then anything in nature, so we can go from what we know about computation from the things we’ve created to what might be possible with it.

  • David S.

    Americans don’t cover up the horrors of WWII; we know them full well. The Japanese internment is something that’s has almost exclusively been studied by Americans.

    Who cares what the ancient Greeks talked a lot about? They were slave-holding pederasts who held their society in highest regard. But you don’t hold people who respect them to the same standards you hold people who respect Americans.

    All people engage in some hagiography; being proud of what your group has done is a normal, natural and healthy thing. Denying that a people did anything of value is not attacking hubris, it’s attacking the people.

  • dpolicar

    We’ve failed to create X=anything by creating Y=biological structures.

    But this simply isn’t true. We’ve created a wide variety of bacteria, for example.
    But, sure, I agree that we’ve built computational structures that perform certain kinds of computations far better than anything in nature, while we haven’t yet built biological structures that do anything better than what we find in nature, and that weakens the force of my analogy.

    And, sure, if we can reliably presume that what we have not thus far created via computation cannot be created via computation, then it follows that general deduction can’t be created via computation. I don’t agree with the premise, but the logic is sound.

    Unrelatedly: if deduction has been implemented, as you suggest, using biological noncomputational systems, what do you consider the salient characteristic of such systems? That is, what does a system have to possess in order to implement deduction, on your view?

  • Lee B.

    “I’ve been ordered to take you down to the bridge. Here I am, brain the size of a planet and they ask me to take you down to the bridge. Call that job satisfaction? ‘Cos I don’t.”

  • GDwarf

    Perhaps, but my point is that fiction (and popular perception) give computers far more ability in some areas than they actually have while, you’re right, underrepresented them in areas that they’re superior in. However, in this case no computer that exists could do anything meaningful with all the data the NSA is getting.

  • What I don’t get is that after I announced I was engaged, I started getting endless advertisements for engagement rings. A little late for that…

    I also immediately started getting advertisements for weight loss so I could “fit” into some hugely overpriced, ugly white ballgown I’d wear only once in my life which apparently I was expected to buy in a too-small size.

    I don’t go to Facebook any longer because my friends from high school all had kids at the same time and started posting pictures of baby poop, so I don’t have a clue what FB’s trying to sell me now. Likely baby crap, which I’ve seen enough of for ten lifetimes.

  • Baby poop pictures

    Baby poop pictures

    Baby poop pictures



    I just don’t EVEN.

    There is something in the water they must be drinking to think that is a good idea to expose people to.

  • AnonaMiss

    The 90s were OK?

  • themunck

    Assuming you didn’t want to “request the international human rights protection mechanisms established under the ICCPR (the Human Rights Committee) and the Convention against Torture (the Committee against Torture) to consider complaints brought by individuals that the government has violated its obligations under the treaties”, as an Amnesty report from 99 put it*
    Although I will admit that at least couldn’t think of anything that undignified in the 90s at the top of my head, and had to search for something.
    * (page 5)

  • Foible

    Until you do something that pisses them off, like donating money to the Tea Party. Predicting your behavior is hard, data mining their cache for dirt on an individual is easy.

  • Hexep

    Okay, so Vancouver’s out. Seattle’s out; that one’s great. “West Coast” is out, too, as, I assume, are all 3 LA proxies.

    I’ve never had good luck with Singapore or Seoul. I could try Tokyo… Vladivostok? I have one for Vladivostok, for some reason…

  • The_L1985

    Technically, our part in the war started when Japan bombed Pearl Harbor, which IIRC predates Germany declaring war on the US.

  • Tofu_Killer

    Or spend most of your twenties doing civil disobedience, protesting several wars, and engaging in direct actions with Greenpeace and Earth First.

    Now what gets my goat is that you pretended that you didn’t understand my general point in favor of rephrasing that exact same point in a more limited way around current politics.

    I apologize if I am reading too much into what you wrote Foible, but the gummint already knows who I am, and I already have seen the info I can get through a FOI request, and it’s a big file. Maybe even bigger than most. All that matters is who is in power and what their agenda is as to whether or not your actions are of interest.

    And even with my big folder, I have been employed by the US Federal Government in a position of responsibility (one that involved a background check).

    Now, if you were really concerned about civil liberties like you pretend, the Tea Party moment wouldn’t matter because tomorrow it may be Right to Life, Equal Pay activism, Scientology, having a Google+ profile, or having checked out Ayn Rand from the library. And please let me point out that, given the composition of the movement, being affiliated with the Tea Party is very low-risk behavior. If you really believe being a middle to high income white protestant who doesn’t want to pay taxes is an existential threat to the government…well there isn’t much I can say to you anyway. This uproar with the IRS & NSA is really people who thought the rules were made for them discovering that their lives are just as open to inspection as all those Muslim (possible) terrorists. These are the same people who argued that Their Personal Security mattered more than the civil rights of anyone Not Like Them. They are hypocrites.

    So let’s assume you have a lick of sense, are concerned about civil liberties in a vague way, and carry on, shall we?

    When I say I am disturbed but not paranoid, I mean the government didn’t need a phone log or google search history to intern Native Americans or Japanese, organize lynch mobs, select persons for ‘experiments’ exposing citizens to radioactivity and leaving venereal diseases unchecked, march national guard troops into a crowd of protestors with live ammunition, or any other of the many many casual atrocities that seemed like a good idea at the time. The Big Bad Government doesn’t need this information to behave badly.

    THIS is what is meant by “power tends to corrupt”, how the ability to do ANYTHING corrodes your soul and tempts you into exploring the limits of that power.

    Since the point of democracy is supposed to be that it’s the people who are in power, I keep waiting for the moment that the people decide to take responsibility for their government instead of continuing to pretend that it’s the government itself that’s the problem.

    The so-called tea party movement is a sad political joke that attempts to do just the opposite.

  • They used to be rational people. Then they had babies. I’m hoping they’ll come to their senses in a year or two.

  • It’s usefulness will come when someone is caught and you can then see who he called and find out the relationship. Very few terrorists (if any) work alone. You may not prevent the initial crime, but you might be able to find the accessories and prevent further crimes down the road.

  • VMink

    Back In The Day, the Pentagon’s Exchange servers would routinely crash because of PowerPoint. That probably fulfilled the RDA of iron for the Pentagon’s IT guys.

    Basically, some bright young butterbar or railroad LT (a captain for the Army, Marine, and Air Force types) would make a PowerPoint presentation with all the bells and whistles. He’d give this presentation to, say, fifty officers, all of whom asked for the presentation. He would email it to them. Back then Exchange didn’t deal well with 1MB powerpoint presentations, and when you get 50-100 people receiving 1MB attachments, Exchange fell over went boom, courtesy of these PowerPoint Rangers.

    We didn’t have that much of a problem in the Coast Guard, but until about 1997 or so we were still using modular CTOS systems with 5-1/4″ floppies.