Eli Pariser - The Filter Bubble

If personalized persuasion works for products, it can also work for ideas. There are undoubtedly times and places and styles of argument that make us more susceptible to believe what we’re told. Subliminal messaging is illegal because we recognize there are some ways of making an argument that are essentially cheating; priming people with subconsciously flashed words to sell them things isn’t a fair game. But it’s not such a stretch to imagine political campaigns targeting voters at times when they can circumvent our more reasonable impulses.

We intuitively understand the power in revealing our deep motivations and desires and how we work, which is why most of us only do that in day-to-day life with people whom we really trust. There’s a symmetry to it: You know your friends about as well as they know you. Persuasion profiling, on the other hand, can be done invisibly—you need not have any knowledge that this data is being collected from you—and therefore it’s asymmetrical. And unlike some forms of profiling that take place in plain sight (like Netflix), persuasion profiling is handicapped when it’s revealed. It’s just not the same to hear an automated coach say “You’re doing a great job! I’m telling you that because you respond well to encouragement!”

So you don’t necessarily see the persuasion profile being made. You don’t see it being used to influence your behavior. And the companies we’re turning over this data to have no legal obligation to keep it to themselves. In the wrong hands, persuasion profiling gives companies the ability to circumvent your rational decision making, tap into your psychology, and draw out your compulsions. Understand someone’s identity, and you’re better equipped to influence what he or she does.

Someday soon, Google Vice President Marissa Mayer says, the company hopes to make the search box obsolete. “The next step of search is doing this automatically,” Eric Schmidt said in 2010. “When I walk down the street, I want my smartphone to be doing searches constantly—‘did you know?’ ‘did you know?’ ‘did you know?’ ‘did you know?’” In other words, your phone should figure out what you would like to be searching for before you do.

In the fast-approaching age of search without search, identity drives media. But the personalizers haven’t fully grappled with a parallel fact: Media also shapes identity. Political scientist Shanto Iyengar calls one of primary factors accessibility bias, and in a paper titled “Experimental Demonstrations of the ‘Not-So-Minimal’ Consequences of Television News,’” in 1982, he demonstrated how powerful the bias is. Over six days, Iyengar asked groups of New Haven residents to watch episodes of a TV news program, which he had doctored to include different segments for each group.

Afterward, Iyengar asked subjects to rank how important issues like pollution, inflation, and defense were to them. The shifts from the surveys they’d filled out before the study were dramatic: “Participants exposed to a steady stream of news about defense or about pollution came to believe that defense or pollution were more consequential problems,” Iyengar wrote. Among the group that saw the clips on pollution, the issue moved from fifth out of six in priority to second.

Drew Westen, a neuropsychologist whose focus is on political persuasion, demonstrates the strength of this priming effect by asking a group of people to memorize a list of words that include moon and ocean. A few minutes later, he changes topics and asks the group which detergent they prefer. Though he hasn’t mentioned the word, the group’s show of hands indicates a strong preference for Tide.

Priming isn’t the only way media shape our identities. We’re also more inclined to believe what we’ve heard before. In a 1977 study by Hasher and Goldstein, participants were asked to read sixty statements and mark whether they were true or false. All of the statements were plausible, but some of them (“French horn players get cash bonuses to stay in the Army”) were true; others (“Divorce is only found in technically advanced societies”) weren’t. Two weeks later, they returned and rated a second batch of statements in which some of the items from the first list had been repeated. By the third time, two weeks after that, the subjects were far more likely to believe the repeated statements. With information as with food, we are what we consume.

All of these are basic psychological mechanisms. But combine them with personalized media, and troubling things start to happen. Your identity shapes your media, and your media then shapes what you believe and what you care about. You click on a link, which signals an interest in something, which means you’re more likely to see articles about that topic in the future, which in turn prime the topic for you. You become trapped in a you loop, and if your identity is misrepresented, strange patterns begin to emerge, like reverb from an amplifier.

If you’re a Facebook user, you’ve probably run into this problem. You look up your old college girlfriend Sally, mildly curious to see what she is up to after all these years. Facebook interprets this as a sign that you’re interested in Sally, and all of a sudden her life is all over your news feed. You’re still mildly curious, so you click through on the new photos she’s posted of her kids and husband and pets, confirming Facebook’s hunch. From Facebook’s perspective, it looks as though you have a relationship with this person, even if you haven’t communicated in years. For months afterward, Sally’s life is far more prominent than your actual relationship would indicate. She’s a “local maximum”: Though there are people whose posts you’re far more interested in, it’s her posts that you see.

In part, this feedback effect is due to what early Facebook employee and venture capitalist Matt Cohler calls the local-maximum problem. Cohler was an early employee at Facebook, and he’s widely considered one of Silicon Valley’s smartest thinkers on the social Web.

The local-maximum problem, he explains to me, shows up any time you’re trying to optimize something. Say you’re trying to write a simple set of instructions to help a blind person who’s lost in the Sierra Nevadas find his way to the highest peak. “Feel around you to see if you’re surrounded by downward-sloping land,” you say. “If you’re not, move in a direction that’s higher, and repeat.”

Programmers face problems like this all the time. What link is the best result for the search term “fish”? Which picture can Facebook show you to increase the likelihood that you’ll start a photo-surfing binge? The directions sound pretty obvious—you just tweak and tune in one direction or another until you’re in the sweet spot. But there’s a problem with these hill-climbing instructions: They’re as likely to end you up in the foothills—the local maximum—as they are to guide you to the apex of Mount Whitney.

This isn’t exactly harmful, but in the filter bubble, the same phenomenon can happen with any person or topic. I find it hard not to click on articles about gadgets, though I don’t actually think they’re that important. Personalized filters play to the most compulsive parts of you, creating “compulsive media” to get you to click things more. The technology mostly can’t distinguish compulsion from general interest—and if you’re generating page views that can be sold to advertisers, it might not care.

The faster the system learns from you, the more likely it is that you can get trapped in a kind of identity cascade, in which a small initial action—clicking on a link about gardening or anarchy or Ozzy Osbourne—indicates that you’re a person who likes those kinds of things. This in turn supplies you with more information on the topic, which you’re more inclined to click on because the topic has now been primed for you.