Tomi Knuutila: Dances with Datafeeds

Information has been used in artistic practises for quite some time. For simplicity’s sake in this post I mostly talk about information (or data) visualisation and sonification (changing data to sound). This should be thought of as a smaller subset of a bigger, amorphous field of creative practices, often referred to as Information Arts, or Art and Science collaboration. In this post I’m mostly ignoring the various discussions for and against art and science collaborations, and assuming that I can take this field as a more or less granted, existing and well established field in which many contemporary artists work with. For the curious, to dive deeper in this topic I suggest the great book Information Arts by Stephen Wilson — or browsing through the 50+ years of archives of the Leonardo Journal, although many good sources exist.

Although algorithms have been used in visual art and music-making throughout centuries — e.g Musical dice games, (in which small precomposed parts of music were combined together to produce a final composition by throwing the dice) were quite popular throughout the 18th century. Of course information visualisation has also a long history in science popularisation and it has been vital in explaining numerical data in books, magazines and newspapers. The current era information usage in arts rose in popularity in the mid 20th century as composers, painters and other artists started exploring methods, such as controlled randomness, chance and mathematical patterns and algorithms to produce (aleatoric) music, visual art and even choreographic movements. These techniques can be seen paradoxically on one hand supporting many avant-garde movements’ ideals of diminishing the originality, talent and ingenuity of the artist, but on the other hand the methods introduced new expressive possibilities, techniques and tools to artistic practice. Artists such as John Cage, Merce Cunningham and Marcel Duchamp who pioneered these methods are currently regarded more or less as genius.

For me these two aspects are interesting: producing visuals and sounds – of which I don’t have complete control – is an artistic practice I feel comfortable with, and getting interesting and unpredictable results is something which, for me, nourishes the creative output. So essentially I see working with machines, technologies, software and programming as tools with which I iteratively collaborate. I have used algorithms, data and information to create melodies and drum patterns, manipulate music, change shapes, colours, size and appearance of graphical elements, animating them, and even using data to determine video editing cut points. While many people argue that computers, algorithms or AI cannot be creative, I would credit them for a huge part in many things which I do. Some experiments in moving image and sound can be found in my ongoing 100 video art works project.

Climatable – Working with Climate Change Data

In 2009 I premiered my interactive installation Climatable. While the installation was mainly built as a testing ground for sensor-based interaction focusing on simplicity, it worked also as climate change information visualisation and sonification tool. As climate change is quite a complex topic, quite a lot of research was needed in order to find out what kinds of various indicators were used in science, although many of these were already familiar as graphs seen in articles and news. Out of the multitude I finally chose four different data sets. Some data sets had to be rejected since there were too many gaps in the measurements, some was from a too brief time period, and in some rare cases I also had to fill in some gaps by calculating an average value out of surrounding values. It can be argued that I chose data sets which supported my initial gut feeling that the data is escalating. Then again, as framing a photograph is an act of power, so is selecting a dataset, and it is good to be aware of this. It is impossible or at least foolish to put together a visualisation or sonification of all possible data sets related to climate change data, and some selection had to be made. The chosen data sets for me were representative, interesting, consistent, descriptive and expressive: representing one side of the phenomenon; interesting what they told about the phenomenon, consistent so that there were no gaps and descriptive about how climate is changing and expressing how climate change sounds and looks like in a different way than typical news article graph.

A derivative of the project is the sound of climate change minimalistic video, a media art work or a musical piece, which in my opinion illustrates the five selection criteria. As the video states, it uses global annual mean temperature data from Nasa Goddard institute between 1880-2014 (please listen until the end, or at least jump to the 1950’s and listen onwards from there to get the point):

After turning the data into sound and hearing the change in the last two decades or so for the first time I was astonished, and still am when I hear the piece. It is worth stating again that the music (or sound in this case) is not “composed”, improvised, manipulated manually, or played according to some instructions found in the data. The sound is manipulated based on the data, one could almost say the sound is the data (although this would be quite and exaggeration. In this case what I have defined is the sound file to be played / looped, and what it sounds like when the data is at its lowest and highest values. So, yes I know it will sound drastic, but when does it sound drastic or mellow I really don’t know beforehand. If you checked out the video you know when the drastic things happen.

Practical Workflow

So let me illustrate more clearly the workflow I used to create the sound piece. This might get a bit technical, but I hope you’ll bear with me:

1. Find a dataset which is interesting. I use the Global Land Ocean Temperature Index as an example here, it is different from the one used to create the sound example — a bit newer and uses both land and surface data, but works just as fine. If you don’t know if there is data about the whatchamacallit which interests you, Google Dataset Search is a good place to go.
2. Find out if there is a version of the data suitable for image or sound manipulation. I prefer to work with as simple data documents as possible, in the site above one can find this text file.
3. Strip unnecessary things from the datafile, prepare it to suit your software / creative environment. The final file can be a text file, xml, programming environment specific file (basically a text file), a spreadsheet file, or even just copy+paste the raw data to the program. Typically data values are separated e.g. with commas, empty spaces, tabs, line breaks, or semicolons. The data preprocessing can be done manually, by programming or by a combination of these two. The creative software or program should be able to read the data, and do things like count how many values there are, find out what is the minimum and maximum, etc. Think about it as a super simple excel sheet (actually excel-files can be used by many programs, or at least csv-files (comma separated values) from excel).
4. How complex is the data? Our example datafile is two-dimensional (time, in this case a year and value, in this case the temperature anomaly from average from 30-year base period 1951-1980), and thus we have some options: e.g. move in time and display (visualise, change in to a sound) a value after a short period, or move in space, and do the same. What about more complex data, with time, geographical locations and multiple values (income rate, employment status, alcohol usage etc)? Many of these can be represented simultaneously, but it can create chaos. Traditional 2D graph can illustrate relationships between surprising factors, as the Gapminder tool illustrates. But is it interesting in an artistic point of view?
5. Choose how to represent the data: with a musical note? A typographic pattern? Time position in a video file? A photograph’s colour hue value? Theatre light change? A choreographic movement chosen from Laban’s set? A position in space? A scale of a hand movement? All of these and more? The possibilities are endless…
6. Does the data range make sense? Most likely not. Consider the piano: in 88 keys the pitch goes from low to quite high. In the global mean temperature values the changes are quite small, from lowest -0.49 degrees below mean to highest 0.99 degrees above. Other data set values might range from -220.5 to 305.8 , which is way beyond the regular music scale. So the values have to be adjusted to a suitable range. There are many ways one can go around this: in the first case we have to shift the values to be higher than zero and then multiply with a suitable value. In the latter case one could perhaps add 300 to all values and play it out as tones, using the value as the sound frequency in hertz, or compress the values to the values you like (e.g. 1-88 in the case of a piano or 0-127 in case of midi, or something else). In programming environments it is typical to change the range to be between 0 and 1 since after that it is easy to multiply values of the dataset to any value between 0 and the target maximum. For that one needs to subtract the minimum value from the value coming from the data set and divide this with the range (the difference between maximum and minimum, in this case 0.99 -(-0.49) = 1.48. In programming terms this might look something like (currentValue – minimumValue) / (maximumValue–minimumValue). So -0.49 becomes 0 and 0.99 becomes 1, and all the values between these sit along nicely between these two (there is an infinite amount of numbers between 0 and 1). If the target maximum is 88 as in the piano keys, just multiply the final value in the range after the adjustment with 88 — or to be more exact with 87 and add one, since there is no key 0 in the piano, and the value determines which note to trigger. If you want to use 3 octaves in starting from the middle C, the multiplication would be * 36 + 40 since, we want to have values from 0 to 36 to play the notes in three octaves and we add these to the Middle C, the key number 40 on a 88-key piano. This example uses both black and white keys, playing notes using on the major c-scale only would require a bit more math fiddling — or manipulation in the audio software side. This data adjustment is done in programming, either by writing the code in a programming language such as Python, Java, using an environment targeted towards creative expression such as Processing or OpenFrameworks, or using a node-based programming environment such as Isadora or TouchDesigner. I have been working with Apple QuartzComposer for the last 10 years, a tool which I know almost by heart, but which unfortunately hasn’t received support in the last operating systems.
7. If necessary, choose how to carry the data from your software to another. You might have a perfect tool, which lets you display graphics, render 3D images, manipulate sound files, play midi notes, control lights, give out commands to actors and dancers on a stage, but most likely you need to do things in 2-3 software at the same time. One software might read and format the data file, another one change it to numerical data in a suitable data range and send them out as midi messages, and a third one will receive those signals and change them to music. Some alternatives and possibilities are midi messages: midi notes and control channel messages can be used for different things. OSC (Open Sound Control) is a newer format with a bit more resolution than with midi, and can be used between many multimedia software. In the Sound of Climate Change example the messages are sent from QuartzComposer via midi control channel messages to Ableton Live, and an audio file is manipulated with a filter or two using the values coming from Quartz, the midi range being 0-127. Although more interesting animated graphics in 2D- or 3D-space would be possible to create in Quartz, this time I wanted to focus on the sound and just displayed the current year.
8. How do you navigate within the dataset? In the example we change from year to year after a certain time period. In the installation version there was a physical slider with which the participants could select a year and hear and see the corresponding value. Perhaps values could be distributed to different locations on the stage or on a dancer’s body? To various objects on the stage? So the data can control the actor and the actors can control the data, or both.

Lastly I leave you with a musical piece related to the Sars0-CoV-2, or the Covid-19 and its genome sequence. It is an “Positive-sense single-stranded RNA virus”, and online one can find quite many scans of the same virus (over 400 by the time of writing this post). The genetic code, the genome sequence data is represented it two ways: the nucleotide sequence, ie. bases in the molecule with letters a, c, g, and t (although rna typically is said to have a,c,g and u? My knowledge about dna and rna sequencing is limited, so please correct any mistakes I might make) and the Protein sequence translation, in which three to five bases form a codon, which is related to one of the 20 different proteins, or a stop character. These are represented by ascii letters, e.g genes 27749..27880 translates as MIELSLIDFYLCFLAFLLFLVLIMLIIFWFSLELQDHNETCHA. The NCBI (National Center for Biotechnology Information) site provides some tools to see and do these translations. Thus we have at least two kinds of patterns to work with: the four bases, which I have used as melody notes in the example below and as a drum machine , and the translated proteins, which could be intrepreted as chords, more complex melodical patterns, rules to guide tempo, or dynamics etc to create more complex musical melodies — as Markus J. Buehler obviously has done in the beautiful musical pieces which can be found on his SoundCloud page. So we do give some control to the pattern, but still make creative decisions.

This blog post is written for TaikaBox dance research days, instead of being in Oulu in person and meeting with dance and media artists — the meeting was cancelled due to the virus threat. As you might realise I am not a dance expert although I have worked with them somewhat. The interesting thing for me to find about is whether using data in an artistic practice is seen as an interesting method to try out, especially when we step outside of representing it with media — via text, graphics or sound. For me, hearing the sound of climate change: the escalation in the audio output, in the seemingly small looking measurement changes, was truly a moment in which I experienced climate change in a new way, unlike the graphs I had seen. Perhaps using data to control the dancer around the stage, or to guide choreographic movements is an idea which never should have been brought forth, but who knows? Not at least without trying something out.