Visualizing Music

How do you visualize music? Can you turn an auditory experience into a visual one? Scores are one way of doing just that.

An example orchestral score

While a score requires training to read fully, you can follow along when it’s paired with the music. That was part of the allure of early computer music programs like the Music Construction Set.

In terms of representing music visually, many people default to the conventions of a score: higher pitches are higher on the paper, with each instrument getting its own line. Even projects like A Bicycle Built for Two Thousand (which is possibly the creepiest rendition of “Daisy Bell” ever) follow that convention.

Scores represent each note individually, but they don’t capture the timbre of the separate instruments. Instead, they merely name what instruments are to be played, with similar instruments’ staves grouped together. Anita Lillie created visual representations that are like a score, but with timbre to color, producing some neat videos in the process.

What if we move away from trying to capture the individual notes and instead try to represent larger components of a song? The band Pomplamoose videotapes themselves recording their multitracked songs and assembles the footage into a single video. The result is a visual representation of the song’s layers. For instance, you see Nataly Dawn acting as her own backup singers. Their visual representation is organized around each track in a song.

Lasse Gjertsen’s “Amateur” is similar in spirit, with one notable exception: he videotaped himself playing each drum hit and piano note separately, then sliced that video up and combined it into a new song. It’s a hybrid between the traditional score and what Pomplamoose is doing, since Gjertsen’s video is presenting each individual note or event as it happens, but constructed from separate events instead of a continuous performance.

Then there’s Kutiman. For “Thru You”, he assembled his songs from pre-existing YouTube clips rather than composing a song and then videotaping himself or others playing it. The separate video clips are a visual representation of the building blocks he’s used to create his bricolage.

Finally, consider Girl Talk’s mashup album “Feed The Animals”. Gregg Gillis used samples like instruments, layering them on top of each other to create a new composition. Soon after the album’s release, fans created music videos compiled from those of Gillis’s source samples.

Given my love of visual representation, it’s no surprise that this topic fascinates me. Part of it is the whole “dancing about architecture” problem: it’s hard to translate an experience from one sense to another. That’s why, when it’s done well, I find it especially rewarding.

16 thoughts on “Visualizing Music

    1. I did! I hadn’t seen it until college, I think, and I fell in love with it. I actually had meant to reference it in this post and forgot.

  1. I often think about what the notes look like on the page when listening to music. Interesting post.

  2. Your post is part of a dialogue I have with my students every semester in the Music since 1945 class I teach, namely how do we represent music for others. Most of the history of modern Western music has been how to present music visually so the sounds the composer intends come out of the performer. Morton Feldman once said that the degree to which the visual look of the score determines the sonic outcome is the greatest secret in Western composing, and I tend to think he’s right. I may have to put together a response post because I too find this fascinating, just from a different outlook.

  3. Yeah, I really came at it from an audience point of view, in how you can represent music visually to non-performers. The composer-to-performer link is a tricky and twisty one, and I don’t feel qualified to address it, so I’d love to hear your thoughts on it.

  4. I, too, have found the videos like the one by Pomplamoose incredibly compelling. John Wilson linked to a couple more like this in a recent post. The one by Julia Nunes is especially mesmerizing to me. : )

    Oh, and don’t forget the one you pointed us to a long time back that had the objects going by a train window in time with the sounds in the music. I can’t remember artist or title, but I remember the visual. 🙂

    The thing that’s really interesting to me: the extent to which videos like this can influence my enjoyment of the music. I’ve had cases where I’m pretty sure I wouldn’t have given the whole song a listen without the video. As an example that’s really only tangentially connected to your real point, I didn’t really start liking Steely Dan’s music until I saw some footage of them playing. For some reason, Fagen was just interesting to watch. 🙂

  5. Geof, I don’t think that’s weird at all. I find that I’m visual in some aspects, but not particularly so in others. I can’t help wondering if the degree to which we visualize in particular fields rises from our introduction and early experience with that field, or perhaps our experiences over time. Kinda like the commercial in which the composer sits at the piano with–what, composer’s block?–until he notices birds on telephone wires and riffs on the theme they visually represent by their positions on the wires.

  6. A lot of times, the “visuals” I experience are those from when I had my most memorable hearing of the song. If I hear “Seether” by Veruca Salt (or pretty much anything off the Tank Girl Soundtrack) I think of the brick buildings in downtown St. Louis. “That Old Wind” by Garth Brooks makes me see US 27 passing by my windshield in Florida which I’d driven quite a bit with that CD playing. Hearing a song generally combines with the visuals I am experiencing and imprints it on my mind. Fall on Your Sword’s “Shatner of the Mount” will inevitably bring up visuals of the Marriott Atrium Ballroom with people loaded in for the Patrick Stewart panel (even more so than the Nimoy-Shatner panel, but that’s likely because I was pretty nervous prior to Nimoy-Shatner and wasn’t really paying a lot of attention to anything besides “Don’t screw up” 🙂

    There are some songs, however, that I experience in different ways for reasons I’m not entirely clear on. When I first heard “Sadness” by Enigma I clearly saw the music video I would make for the song. I make music videos from time to time and have to be inspired by a song and movie combination (eg. “Wouldn’t it be awesome to put together Sia’s Breathe Me with The Abyss?”, “Wouldn’t it be awesome to match Fight Club with Ministry’s N.W.O.?”, etc.), but this is typically only for songs that really resonate with me. It appears there’s a hierarchy of songs: songs I don’t remember; songs I associate with a specific time and a place; and songs that impose themselves on my consciousness to the point I feel the need to do something creative with them.

    But my experiences are solely concerned with pre-existing music, not the creation of new music along with the accompanying visuals. . . probably because I can process individual elements and figure out how they go together easier for visual things than I can for audio things. I have to think I’m primarily a visual thinker and internally represent sound as images. Probably because of too much MTV rotting my brain.

  7. Surely you can see the problem of using musical scores to attempt to turn an auditory experience into a visual one? Aside from the fact that the score doesn’t contain information for the quality of each tone, the main patterns of relationships between the notes are represented largely in symbolic form rather than in absolute form. For example, the physical space between two notes on the score does not tell you precisely how far apart they really are in terms of semi-tones. A flat or sharp symbol may be placed next to them to modify the distance. Thus our brain has to do extra work to translate the symbols on the score into the actual pattern. Perhaps some people can do this quickly enough to experience the pattern as they would a recording but it would take much training; far more than that required for sight-reading.

  8. There’s problems with any method of representing an auditory experience as a visual one. Any translation will lose something in the process.

  9. Why should any visual translation necessarily loose something? The sound waves simply embody a pattern. The pattern is very complicated if you consider the nuances of the waves of particular tones from particular instruments. But nevertheless, they are patterns which could quite easily be displayed visually. The question is whether or not the brain would respond in the same way to seeing the pattern instead of hearing it. Perhaps the part of the brain that can make sense of patterns of this sort will only take input from the ears.

  10. You’re going to lose something in visual translation because you’re going to simplify it to meet your criteria of creating a pattern that you can experience in a manner similar to hearing the music. You could, for instance, record a performance digitally into a lossless format and then print out the hexadecimal numbers that correspond to that digital pattern, but can you extract the music from the pattern? I’ve known people who can approximate hearing music when they read a score, by drawing on their experience of what instruments sound like and knowledge of how to read a score, but that’s very difficult even though a score is a greatly simplified translation of the music.

  11. Ok here is an example of how you could present the exact same information in absolute (non-symbolic) form: for each instrument and each voice or line being played one each instrument you could display a dot on a screen. The dots can move up and down with frequencies and amplitudes of oscillation which are the have the same relationships to each other as the actual sound waves that the real instruments would produce. This would visually convey the exact same information as the original recording. This particular way of doing it wouldn’t be very effective, I just thought of it off he top of my head. The brain’s processing systems responsible for interpreting frequencies and amplitudes of oscillation are probably not available for information coming in from the eyes. Having said that, I suppose the eyes are sensitive to differences in the frequencies of light waves. So the different frequencies of the tones could be easily translated into different frequencies of light waves (in other words different colours). Differences in the amplitudes of the sounds waves can be translated into different intensities of light.

Comments are closed.