TSO: Text to Sound Objects

Posted by devin

I’ve obsessed for the past few years over a project that basically involves turning letters into tones, words into chords, sentences into melodies, and paragraphs into complete musical thoughts. It’s difficult to know where to begin, but here is an example of what I imagine:

The house is red. It has a window, a door, and a chimney. Three people live there. It’s in a nice neighborhood.

Let’s take the first sentence, and map the key of C to it:

T h e  h o u s e  i s  r e d.
A B C  B D G B C  F B  E C D.

So how do we make this work? This stage isn’t really all that complicated. What becomes interesting now is interpreting the random values we’re assigning to this sentence using various perspectives from the music theory side of things, to make a sound structure to build upon. Our first sentence defines domain in which other words and sentences will be interpreted.

A B C could be played in any variation as single notes, in harmony (A C would probably sound most recognizable), or perhaps we’ll just pick one note from that word and use it. How “house” will act in this context brings another level of complexity to the table. “The” defines the starting point arbitrarily, and “house” must use what it contains to work with “The”. Depending on the type of music you’re trying to reproduce, there are many many paths one might take in deciding what “house” will mean to “The”. I’ve only taken a limited amount of college level music theory, but I’ll take a stab at how I might proceed. For the purposes of demonstration, let’s call “The” a chromatic A B C musical phrase.

B D G B C contains a G chord (G B D (the V chord in the key of C)). If no possible chord existed in this case we could do the same as we did with “The”, only this time, it would be keeping in mind what “The” represents musically. Our domain is a little more narrowly defined now that we have a starting point. I could give you a bunch of examples of how we could proceed, but I’m going to try and keep this post quick and dirty. Let’s call “house” a G chord.

Our current work at this point in case I’m losing you:

A B C (G chord)

Now we tackle “is” in this context (F B). B is the leading tone in the key of C, and happens to come before the final word in the sentence, so let’s use it.

A B C (G chord) B

Finally, “red” translates to E C D. We’re heading home with the leading tone, so let’s use C and E in harmony.

If you’re still reading, we have a sentence!

A B C (G chord) B (C E).

Check out how it sounds by listening to midi representation.

I’m going to put this to bed for the night and edit to include the details I know I’m forgetting. This idea has been in my head for a long time and committing all of those fleeting thoughts to paper is going to be a challenge. Thanks for reading. I hope anyone who has experience or thoughts on this subject will comment. Input is much appreciated.