2D Fourier Transform on Music

For a long time, I have thought about what to do to visualize music. The need for this comes when posting music recordings onto YouTube, which is a very nice platform for distributing music. Just showing a black screen does not really do justice to the visual medium, and so one has to think on what to put there to the music. Many music pieces have there a still image, or a collection of images which are relevant to the music. This is fine, but does not use fully the possibilities which video is offering. Ideally, one would have a video that is tightly synchronized to the music and complements the acoustic impression with a visual one - as it is done in popular music videos, since the time MTV has pushed this forward ("video killed the radio star" comes to mind). But for classical music there is really not very much in terms of appropriate music visualizations: there are videos of the performers, there are those series of smartly arranged still images, but rarely is there a video which truly complements the audio and the music in terms of visualizing directly the music and its elements.

In the past there has been some work on this, and I am especially fascinated by the work that Oskar Fischinger has done in the 1920-40s. Sometimes a few of his films have been shown on Turner Classic Movies in between regular movies. He really tried to take music and visualize it with abstract shapes which represent the music.

On most mediaplayers, there are automatic visualization tools which show some automatically generated animations, based on the music audio. However, I have never found a satisfying visualizer, because most of the visualizations appear to be some random motions, on top of a bit of wave analysis.

That kept me thinking, and then in December 2011 I had the idea to use the Fourier Transform to detect periodicities in music. The Fourier Transform (FT) is a well known approach for analyzing audio: it is generally used to create spectrograms and show the frequency content in music. FT detects periodicities in signals, and this is usually used to detect the frequencies which are present in an acoustic signal. Music is based on such frequencies, which relate to musical notes. But music has also "slower" periodicities", not related to the high frequency of sound (>20 Hz), but related to the measures and the beat in music. So I decided to apply the FT on music with a larger window: for a sound spectrum, the FT is usually applied for a time window of 100 ms. But for revealing longer-period periodicities in music, I would apply the FT for a time window of several seconds.

To make the computations easier, I decided to not use the sound wave itself, but the abstract notation of music as input. This is a series of discrete note events, which have pitch and volume. In music sequencers, the notes are usually shown on a piano plot roll, which has as horizontal axis the time, and as vertical axis the pitch resp. the note frequency. A FT can be applied on this piano plot roll, using the 2-dimensional approach for FT that is used in image processing (noise reduction, feature detection).

The figure to the left here shows the 2D Fourier Transform of the piano roll plot of Gustav Mahler's "Urlicht" (4th movement of Symphony No.2). In this plot, the horizontal axis shows spatial frequencies of the rhythm and the beat distribution. The vertical axis indicates the distribution of intervals and chords. There are clearly some peaks to be seen, which indicate predominant intervals in this music. Most of the relevant data are around the central axis (x=0) in the FT plot, which is the axis with small rhythm elements, indicating long notes (or pauses). Along this central vertical axis, the musical chord and interval elements can be seen as distinct points or regions.

This 2D FT can be re-transformed back into a piano roll plot, by again applying the Fourier Transform algorithm. I have tried this, and indeed it comes out correctly. However, this back-transformation also needs the phase of the FT (something that is also numerically computed as part of the transformation, using complex numbers). If this phase is not present, then the resulting piano plot roll is overlaid with its mirror image, due to symmetry. It would be quite beneficial to find a way to remove this mirror image, without using the phase of the FT. Then, the inverse transformation could be used to edit the music in its Fourier space. This would provide completely new ways of creating music.

Overall I think that this 2D FT has quite a significance for music: it is invariant to pitch and tempo (at least if the music time is used as reference, instead of real time), and this can lead to unique fingerprinting of music or musical phrases, which can be used for identifying music. Also, it can provide new ways of creating music by editing its Fourier Transform. And last but not least, I have found a new way of visualizing music. At least, the FT algorithm provides a new dataset (2D array) as a basis for an interesting visualization that directly represents the musical content.

I have published this approach recently at the Music, Mind, Invention workshop in Ewing, NJ. The officially published paper for this is here, and the presentation which I gave is here, although that presentation does not have me speaking, so it may be a bit hard to understand the context and background.

I plan to develop this further, make a few videos with this approach as visual accompaniment of my music, and I will work towards releasing the software as a toolkit so that it can be used by others. Lots of work to do then.

Opening of Workshop "Music, Mind, Invention"

At the evening before the workshop "Music, Mind, Invention" began (held at The College of New Jersey in Ewing), a concert was given in the Center for the Arts, where pianist Joanna Chao performed music by mostly contemporary composers. Here is the program she played:

Joshua Fineberg: "Tremors" (1997)
J.S.Bach: Aria to Goldberg Variations
Fred Lerdahl (2004, 2009): "Three Diatonic Studies", with the movements "Chasing Goldberg", "Cyclic Descent", and "Scalar Rhythms"
Robert Young McMahan: From the "Six Miniatures for Piano" (2011-12) the first two: "Andante" and "Crisp, Strident". This was a world premiere.
Joshua Fineberg: "Veils" (2001)

A very appropriate opening of the workshop, followed by a reception with the opportunity to chat with the participants.

After the Music Workshop

The workshop at the Bridge Arts Gallery in Castleford on 2.October went well. We had five participants with a varied background: some with professional music experience, others with a general interest in music and arts. The first half of the workshop I presented the general concepts of making music with a computer. After lunch with good fish&chips from a nearby outlet we continued to explore the use of a sequencer for music making. I demonstrated some of the sounds of the Garritan Personal Orchestra sample library. Then I played the first movement of Gustav Mahler's Symphony No.1 through the sequencer. The participants, most of them in general not very fond of classical music, found it very enlightning to see the visual indication of the music structure in the piano roll: each voice had its own color, and one could observe the up and down motion of these voices. A teacher saw the great possibilities of this for introducing children to music. Also the intuitive note editing of sound duration through the length of the note bars on the piano roll view appeared to the participants as a major step forward from the traditional 5-line note staff view.

We did not have time to actually start a joint collaborative composition of our own, as I had originally envisioned - we may do this at another time.

Free Workshop: Composing Music Using Your Computer

2.October 2010, 10:30 – 15:00
Bridge Arts, Sagar Street, Castleford, WF10 1AF

Content

In this workshop the participants will learn about the tools for making music with the help of a computer. Those tools are nowadays available to anybody and allow the creation of music in ways that were previously only available to music professionals. This includes sequencing of music in abstract notation, the sound generation, the inclusion of acoustic effects, and the final recording of the music onto a CD.
The presenter Reinhold Behringer will give an introduction about those computer tools and will demonstrate them on a custom-built computer MAESTRO-2 as an example of a music computer system. The principles of sequencing and the use of samples will be explained and demonstrated with Cakewalk SONAR 8.5 and the Garritan Personal Orchestra 4.
The workshop will then focus on the classical orchestra and its instruments. Through demonstration of the individual orchestral instrument samples the acoustic space of the orchestral sound will be explored. As an example of a classical music sequence, the Symphony No 1 of Gustav Mahler, whose 150th birthday was celebrated this year (2010), will be shown as a computer score, and with this work as an example, the various techniques of electronic digital instrumentation will be demonstrated.
In the second part of this workshop, the participants will collaboratively compose a piece of music, using these computer tools. This will result in a music recording which each participant will take home as a CD.

Prerequisites brought by the participants: none, except an appreciation of music.

Schedule

10:30 – 12:30 Introduction to Orchestral Computer Music
12:30 – 13:00 Lunch Break
13:00 – 15:00 Collaborative Music Composition

About the Presenter

Reinhold Behringer is Professor of Creative Technology at Leeds Metropolitan University. His technical area of expertise is artificial intelligence, computer vision, and automatic driverless cars. His interest in classical music already began at an early age, and he played piano and joined the high school orchestra playing cello. In 1993 he began to use computer technology to create "renditions" of classical music, and in 1998 he founded the "Virtual Philharmonic Orchestra". He has created computer-based renditions of many classical orchestral music works, including Prokofjev's Symphony Classique, Ravel's Le Tombeau de Couperin, several waltzes by Johann Strauss, and most recently Mahler's Symphony No.1. In addition he has collaborated with opera singers to create two duets from Puccini's operas using electronic digital orchestration.

www.virtualphilharmonic.co.uk

Book Your Place

To book a place on this workshop please call Bridge Arts at 01977 556741.
Alternatively you can call Bohemian Books on 01977 517262 or email info@bohemianbooks.co.uk