Penfield Mood Suite: Bioofedback/Audio Visualisation Software Development
| September 30th, 2011One of the first things I really got into in Processing was writing audio visualizers. This arose from an initial experiment into audio waveform rendering, which led to further experimentation with realtime audio visualisation using the techniques discussed by Antony Mattox. At the same time I developed on the techniques explored by Dan Shiffman in Learning Processing. Shiffman’s book explores many areas of text based programming for visual design. However, it doesn’t deal with audio visualisation using FFTs, the topic that I wished to focus on interest multimodal feedback systems. During my research, I have attempted to apply the Mattox code (which, as he suggests, can be used as the foundation of many systems for audio visualisation) to the Shiffman examples, working through the book, using selected exercises from the book to build a variety of audio reactive visual programs.
There is a clear artistic and commercial interest in the use of audio-responsive visual software, evident in my ability to get a number of gigs as a VJ (despite how much I loathe the term). These initially started out at house parties and exhibition after parties, followed by club nights and more commercial jobs. Using these gigs, I attempted to find out what audiences and performers found appealing in such software as well as what they expected from them. I have attempted to apply the findings from my research to my final chosen method of data visualisation. These expectations and appealing elements are discussed below.
The audio data should be the dominant variable in the sketch. Even if other controllable variables are to be used, audio data should be applied to them in some form. The audio spectrum data, which can be retrieved via fft.spectrum[], can easily be applied to any variable: the size of geometric shapes, the increment by which an object moves, or simply the colour data of the drawn image. The spectrum can also be used to affect image tinting, contrast, brightness and gamma levels through the image.adjust library. This idea has been central to my design and development process, attempting to apply this spectrum data to as many variables as possible within the sketch, making the final program as audio responsive as possible. Although much of my visual software includes other control interfaces (mouse, keyboard), these functions are included to offer a degree of user interaction and control. This originates form my own experience using the software in a live capacity, as I wanted to be able to switch between various modes without closing the program. However, these controls are largely used for such a process, with the audio data affecting most elements of the sketch.
The visualisation should be highly audio responsive and operate smoothly. The system of audio visualisation, when being used in realtime, has to run smoothly in realtime. Although a bit of lag is maybe not so noticeable to the general public, it is very noticeable to musicians and performers. For use alongside live audio, the software must respond quickly for optimum effect. As a result, various labour intensive elements of the sketch should be reduced, such as the number of audio frequency bands that are to be visualised. Alternatively, the software can be run on an increasingly powerful computer with plenty of RAM to allow more frequency bands to be visualised.
The system should be visually interesting, but still readable. Much of my experience working with this particular software has shown such linear systems as the colour organ to still be popular for audio visualisation. Those systems based on Shiffman’s nested push and pop example have proved popular and highly visually engaging. However, any attempt to analyse the audio data in such a manner is difficult. The audio visualisation is based more in movement than in geometric shapes corresponding to different frequency bands. Similarly, those visuals based in the adjustment of a pre-rendered image are difficult read as the audio is not so much visualised as the audio data used to adjust another visual image. . In contrast the familiarity of the colour organ model has proved easier to read, amongst general audiences and musicians alike.
In designing the final visualisation method, I tried to use these findings as well as further user & audience feedback to develop the system. The audio data is applied to a broad number of variables that affect colour, shape size and layout. In order for the visual software to run smoothly and to filter out those frequencies where there is little data for visualisation, I have significantly reduced the number of frequency bands. In terms of the general visual form, I have taken the linear method of audio visualisation modelled on the colour organ from Mattox’s audio visualisation code. I have then applied it to a variety of different methods detailed in Shiffman’s work, such as the wave, and experimented with these forms. The result is a relatively familiar system for audio visualisation that offers multiple ways of visualising the data within this form. As a result, the user is offered plenty of choice in how they wish to visualise the data.
The final visualisation can be seen as highly derivative of the colour organ model. However, this accessibility proved highly popular with test audiences and users, whilst the varieties of data presentation in the linear form allowed for a wide variety of different visual objects. The software went through multiple rewrites and updates before it reached its final form, so there is some cleaning and labelling to be done. I chose to use a variety of organic and inorganic looking functions, based on waves, double helixes and the colour organ. Using the booleans and mapped keys, users can interact with the software, turning various shapes on and off as well affecting the plotting of the data and the colour.