Mark Changizi - Harnessed - How Language and Music Mimicked Nature and Transformed Ape to Man

I couldn’t find but a handful of such scores, and I did not have the chutzpah to acquire scores of my own. So I gave it up. I could have pushed harder to find data, but it seemed clear to me that, despite its initial promise, sex was far too narrow to possibly explain music. If music sounded like sex, then why isn’t all music sexy? And why does music evoke such a wide range of emotions, far beyond those that occur in the heat of sex? And how can the simple rhythmic sounds of sex possibly have enough structure to explain musical structure? Without answers to these questions, it was clear that I would have to take sex off the table.

Enough with the things I don’t think can explain music (heartbeats, speech, and sex)! It is about time I begin saying what I think music does sound like. And let’s edge closer to that by examining what music looks like.

Believe Your Eyes and Earworms

It is natural to assume that the visual information streaming into our eyes determines the visual perceptions we end up with, and that the auditory information entering our ears determines the events we hear. But the brain is more complicated than this. Visual and auditory information interact in the brain, and the brain utilizes both to guess what single scene to render a perception of. For example, the research of Ladan Shams, Yukiyasu Kamitani, and Shinsuke Shimojo at Caltech have shown that we perceive a single flash as a double flash if it is paired with a double beep. And Robert Sekuler and others from Brandeis University have shown that if a sound occurs at the time when the images of two balls pass through each other on a screen, the balls are instead perceived to have collided and reversed direction. These and other results of this kind demonstrate the interconnectedness of visual and auditory information in our brain. Visual ambiguity can be reduced by auditory information, and vice versa. And, generally, both are brought to bear in the brain’s attempt to guess about what’s out there.

Your brain, then, does not consist of independent visual and auditory systems, with separate troves of visual and auditory knowledge about the world. Instead, vision and audition talk to one another, and there are regions of cortex responsible for making vision and audition fit one another. These regions know about the sounds of looks and the looks of sounds. Because of this, when your brain hears something but cannot see it, your brain does not just sit there and refrain from guessing what it might have looked like. When your auditory system makes sense of something, it will have a tendency to activate visual areas, eliciting imagery of its best guess as to the appearance of the stuff making the sound. For example, when you hear the sound of your neighbor’s tree rustling, an image of its swaying, lanky branches may spring to mind. The mewing of your cat heard far away may evoke an image of it stuck high up in that tree. And the pumping of your neighbor’s kid’s BB gun can bring forth an image of the gun being pointed at Foofy way up there.

Your visual system, then, has strong opinions about the likely look of the things you hear. And, to get back to music, we can use the visual system’s strong opinions as an aid in gauging music’s meaning. In particular, we can ask your visual system what it thinks the appropriate visual is for music. If, for example, the visual system responds to music with images of beating hearts, then it would suggest, to my disbelief, that music mimics the sounds of heartbeats. If, instead, the visual system responds with images of pornography, then it would suggest that music sounds like sex. You get the idea.

But to get the visual system to act like an oracle, we need to get it to speak. How are we to know what the visual system thinks music looks like? One approach is to simply ask what visuals are routinely associated with music. For example, when people create imagery of musical notes, what does it look like? One cheap way to find out is simply to do a Google (or any search engine) image search on the term “musical notes.” You might think such a search would merely return images of simple notes on the page. However, that is not what one finds. To my surprise, actually, most of the images are like the one in Figure 16, with notes drawn in such a way that they appear to be moving through space. Notes in musical notation don’t look anything like this, and actual musical notes have no look at all (because they are sounds). And yet we humans seem prone to visually depict notes in lively motion.

Figure 16. Musical notes tend to be visualized like this, a clue to their meaning.

Could these images of notes in motion be due to a more mundane association? Music is played by people, and people have to move to play their instruments. Could this be the source of the movement-music association? I don’t think so, because the movement suggested in these images of notes doesn’t look anything like an instrument being played. In fact, it is common to show images of an instrument with the notes beginning their movement through space from the instrument: these notes are on their way somewhere, not tied to the musician’s key-pressing or back-and-forth swaying.

Could it be that the musical notes are depicted as moving through space because sound waves move through space? The difficulty with this hypothesis is that all sound moves through space. All sound would, if this were so, be visually rendered as moving through space, but that’s not how we portray most sounds. For example, speech is not usually visually rendered as moving through space. Another difficulty is that the musical notes in these images are usually meandering, but sound waves don’t meander—sound waves go straight. A third problem with the notion that sound waves are the basis for the visual metaphor is that we never see sound waves in the first place.

Another possible counterhypothesis is that musical notes are visually depicted in motion because all auditory stimuli are caused by underlying events that involve movement of some kind. The first difficulty, as with sound waves, is that not all sound, by a long shot, is visually rendered as in motion. The second difficulty is that, while it is true that sounds are typically generated by movement of some kind, it need not be movement of an entire object through space. Moving parts within the object may make the noise, without the object going anywhere. In fact, the three examples I gave at the start of this section—leaves rustling, Foofy mewing, and the BB gun pumping—are noises without any bulk movement of the object (the tree, Foofy, or the BB gun, respectively). The musical notes in these images, on the other hand, really do seem to be moving their whole selves across space.

Music is like rustling leaves, Foofy, BB guns, and human speech, in that it is not made by bulk movements through space. And yet music appears uniquely likely to be visually depicted as notes moving through space. And not only moving, but meandering. When visually rendered, music looks alive and in motion (often along the ground)—just what one might expect if music’s secret is that it sounds like people moving.

A Google image search on “musical notes” is one way to try to discern what the visual system thinks music looks like. Another is simply to ask ourselves: what is the most common visual display shown during music? That is, if people were to make videos to go with music, what would the videos tend to look like? Luckily for us, people do make videos to go with music! They’re called music videos, of course. And what do they look like? The answer is so obvious that it hardly seems worth noting: music videos commonly show people moving about, usually in a manner that is time-locked to the music, very often dancing. As obvious as it is that music videos typically show people moving, we must remember to ask ourselves why music isn’t typically visually associated with something very different. Why aren’t music videos mostly of rivers, avalanches, car races, windblown grass, lions hunting, fire, or bouncing balls? It is because, I am suggesting, our brain thinks that humans moving about is what music should look like . . . because it thinks that humans moving about is what music sounds like.