Visualizing Music

How do you visualize music? Can you turn an auditory experience into a visual one? Scores are one way of doing just that.

An example orchestral score

While a score requires training to read fully, you can follow along when it’s paired with the music. That was part of the allure of early computer music programs like the Music Construction Set.

In terms of representing music visually, many people default to the conventions of a score: higher pitches are higher on the paper, with each instrument getting its own line. Even projects like A Bicycle Built for Two Thousand (which is possibly the creepiest rendition of “Daisy Bell” ever) follow that convention.

Scores represent each note individually, but they don’t capture the timbre of the separate instruments. Instead, they merely name what instruments are to be played, with similar instruments’ staves grouped together. Anita Lillie created visual representations that are like a score, but with timbre to color, producing some neat videos in the process.

What if we move away from trying to capture the individual notes and instead try to represent larger components of a song? The band Pomplamoose videotapes themselves recording their multitracked songs and assembles the footage into a single video. The result is a visual representation of the song’s layers. For instance, you see Nataly Dawn acting as her own backup singers. Their visual representation is organized around each track in a song.

Lasse Gjertsen’s “Amateur” is similar in spirit, with one notable exception: he videotaped himself playing each drum hit and piano note separately, then sliced that video up and combined it into a new song. It’s a hybrid between the traditional score and what Pomplamoose is doing, since Gjertsen’s video is presenting each individual note or event as it happens, but constructed from separate events instead of a continuous performance.

Then there’s Kutiman. For “Thru You”, he assembled his songs from pre-existing YouTube clips rather than composing a song and then videotaping himself or others playing it. The separate video clips are a visual representation of the building blocks he’s used to create his bricolage.

Finally, consider Girl Talk’s mashup album “Feed The Animals”. Gregg Gillis used samples like instruments, layering them on top of each other to create a new composition. Soon after the album’s release, fans created music videos compiled from those of Gillis’s source samples.

Given my love of visual representation, it’s no surprise that this topic fascinates me. Part of it is the whole “dancing about architecture” problem: it’s hard to translate an experience from one sense to another. That’s why, when it’s done well, I find it especially rewarding.