Eli Pariser - The Filter Bubble

I returned to the question about using algorithms to shift sentiment. “How does someone game the system when it’s all about self-generated, self-reinforcing information flows? I have to think about it more,” Rendon said, “But I think I know how I’d do it.”

“How?” I asked.

He paused, then chuckled: “Nice try.” He’d already said too much.

The campaign of propaganda that Walter Lippmann railed against in World War I was a massive undertaking: To “goose-step the truth,” hundreds of newspapers nationwide had to be brought onboard. Now that every blogger is a publisher, the task seems nearly impossible. In 2010, Google chief Eric Schmidt echoed this sentiment, arguing in the journal Foreign Affairs that the Internet eclipses intermediaries and governments and empowers individuals to “consume, distribute, and create their own content without government control.”

It’s a convenient view for Google—if intermediaries are losing power, then the company’s merely a minor player in a much larger drama. But in practice, a great majority of online content reaches people through a small number of Web sites—Google foremost among them. These big companies represent new loci of power. And while their multinational character makes them resistant to some forms of regulation, they can also offer one-stop shopping for governments seeking to influence information flows.

As long as a database exists, it’s potentially accessible by the state. That’s why gun rights activists talk a lot about Alfred Flatow. Flatow was an Olympic gymnast and German Jew who in 1932 registered his gun in accordance with the laws of the waning Weimar Republic. In 1938, German police came to his door. They’d searched through the record, and in preparation for the Holocaust, they were rounding up Jews with handguns. Flatow was killed in a concentration camp in 1942.

For National Rifle Association members, the story is a powerful cautionary tale about the dangers of a national gun registry. As a result of Flatow’s story and thousands like it, the NRA has successfully blocked a national gun registry for decades. If a fascistic anti-Semitic regime came into power in the United States, it’d be hard put to identify gun-holding Jews using its own databases.

But the NRA’s focus may have been too narrow. Fascists aren’t known for carefully following the letter of the law regarding extragovernmental databases. And using the data that credit card companies use—or for that matter, building models based on the thousands of data points Acxiom tracks—it’d be a simple matter to predict with significant accuracy who has a gun and who does not.

Even if you aren’t a gun advocate, the story is worth paying attention to. The dynamics of personalization shift power into the hands of a few major corporate actors. And this consolidation of huge masses of data offers governments (even democratic ones) more potential power than ever.

Rather than housing their Web sites and databases internally, many businesses and start-ups now run on virtual computers in vast server farms managed by other companies. The enormous pool of computing power and storage these networked machines create is known as the cloud, and it allows clients much greater flexibility. If your business runs in the cloud, you don’t need to buy more hardware when your processing demands expand: You just rent a greater portion of the cloud. Amazon Web Services, one of the major players in the space, hosts thousands of Web sites and Web servers and undoubtedly stores the personal data of millions. On one hand, the cloud gives every kid in his or her basement access to nearly unlimited computing power to quickly scale up a new online service. On the other, as Clive Thompson pointed out to me, the cloud “is actually just a handful of companies.” When Amazon booted the activist Web site WikiLeaks off its servers under political pressure in 2010, the site immediately collapsed—there was nowhere to go.

Personal data stored in the cloud is also actually much easier for the government to search than information on a home computer. The FBI needs a warrant from a judge to search your laptop. But if you use Yahoo or Gmail or Hotmail for your e-mail, you “lose your constitutional protections immediately,” according to a lawyer for the Electronic Freedom Foundation. The FBI can just ask the company for the information—no judicial paperwork needed, no permission required—as long as it can argue later that it’s part of an “emergency.” “The cops will love this,” says privacy advocate Robert Gellman about cloud computing. “They can go to a single place and get everybody’s documents.”

Because of the economies of scale in data, the cloud giants are increasingly powerful. And because they’re so susceptible to regulation, these companies have a vested interest in keeping government entities happy. When the Justice Department requested billions of search records from AOL, Yahoo, and MSN in 2006, the three companies quickly complied. (Google, to its credit, opted to fight the request.) Stephen Arnold, an IT expert who worked at consulting firm Booz Allen Hamilton, says that Google at one point housed three officers of “an unnamed intelligence agency” at its headquarters in Mountain View. And Google and the CIA have invested together in a firm called Recorded Future, which focuses on using data connections to predict future real-world events.

Even if the consolidation of this data-power doesn’t result in more governmental control, it’s worrisome on its own terms.

One of the defining traits of the new personal information environment is that it’s asymmetrical. As Jonathan Zittrain argues in The Future of the Internet—and How to Stop It, “nowadays, an individual must increasingly give information about himself to large and relatively faceless institutions, for handling and use by strangers—unknown, unseen, and all too frequently, unresponsive.”

In a small town or an apartment building with paper-thin walls, what I know about you is roughly the same as what you know about me. That’s a basis for a social contract, in which we’ll deliberately ignore some of what we know. The new privacyless world does away with that contract. I can know a lot about you without your knowing I know. “There’s an implicit bargain in our behavior,” search expert John Battelle told me, “that we haven’t done the math on.”

If Sir Francis Bacon is right that “knowledge is power,” privacy proponent Viktor Mayer-Schonberger writes that what we’re witnessing now is nothing less than a “redistribution of information power from the powerless to the powerful.” It’d be one thing if we all knew everything about each other. It’s another when centralized entities know a lot more about us than we know about each other—and sometimes, more than we know about ourselves. If knowledge is power, then asymmetries in knowledge are asymmetries in power.

Google’s famous “Don’t be evil” motto is presumably intended to allay some of these concerns. I once explained to a Google search engineer that while I didn’t think the company was currently evil, it seemed to have at its fingertips everything it needed to do evil if it wished. He smiled broadly. “Right,” he said. “We’re not evil. We try really hard not to be evil. But if we wanted to, man, could we ever!”

Most governments and corporations have used the new power that personal data and personalization offer fairly cautiously so far—China, Iran, and other oppressive regimes being the obvious exceptions. But even putting aside intentional manipulation, the rise of filtering has a number of unintended yet serious consequences for democracies. In the filter bubble, the public sphere—the realm in which common problems are identified and addressed—is just less relevant.