— Edmund Nesveda

Multi-user interaction

Still with Tina Gonsalves and the “Feel” series, it’s interesting to see how it evolves in time, starting with single user interactive installation and ending with multi-user collaborative interactive installation Interesting trajectory, worth considering in my dissertation research and preparation for an interaction workshop.

Tina Gonsalves, The “Feel” series: An Overview:

The “Feel” series (2005-2007) are an interconnected progression of short films and interactive sketches aiming to sense, translate and provoke the psycho-physiology of the audience.

Synopsis of “Feel Inside”:“FEEL: INSIDE” is a psycho-physiologically responsive video installation synthesizing art, neuroscience and technology. The project explores new, more embodied languages of active and emotional communication, investigating the inter-relationship of the internal body and the external world. The audience enters a darkened room. The audience is presented with a large video pro-jection of the artist’s face. A camera sensros the movement of the audience. The emotional expression of the artist’s face is reflective of the audience’s emotional state, appraised in a general way by audience movement. If the audience is calm, her face becomes still and meditative. Alternatively, anxious movement of the audience is mimicked in the anxiousness of the artist’s face. Over time, continued anxiousness leads to the artist crying, and the ence is confronted with a private and intimate moment usually only privy to close friends family. The audience realizes they have caused the upset, but then are asked to work out how to cure it.

Building of “Feel Inside”: With the second prototype, a focus was to move beyond the viewer being dressed with sensors and directed to sit in a chair as they were in “Feel Trace”. To capture a more ‘naturalistic interaction scenario, I wanted the audience to move freely with in the gallery space. After the potent images of “Feel Trace” I aimed to develop softer, more emotional and poetic audiovisual content, hoping this would elicit more empathy from the viewer/s.

The use of movement to trigger video in interactive design has been used quite often in the past. As a result of this, today, at interactive exhibitions, you can often see participants waving their arms around wildly (in non-naturalistic ways) in order to interact with the work. However, the interaction design of “Feel Inside” asks the viewer to be still. I was suggesting that by being still, the viewer may develop a sense of ‘presence’ both to themselves, and to the environment. Presence has been defined as the sense of being present in a particular environment,[i] often divided into ‘subjective presence’ and ‘objective presence.’[ii] With my experience of meditation and yoga, stillness allowed ‘a sense of presence’ to myself and to the world around me. It was in this state, that I felt more ‘empathic’ with others. I wanted to create an interaction design that amplified this feeling of ‘empathy’ I felt when still. When discussing this idea with Critchley, he agreed that “to some extent, attention on bodily processes requires ‘psychomotor’ disengagement from the environment. There is a fair amount of data relating to respiratory control in meditation and yoga, for example, impacting on heart rate variability. Simplistically, the autonomic nervous system is partitioned broadly into sympathetic (fight and flight ) and parasympathetic (rest and digest) bodily control systems, the latter becoming prominent during stillness”[iii]. Does this mean in order to become more aware of our own body we needed to switch off from the environment?

Research has suggested that one does not often think about their level of presence in the real world. They feel it[iv]. With the video footage used in “Feel Inside”, I sought to prompt a sense of empathy and catharsis in the viewer. The video was shot three years before during a residency at the Banff Centre for New Media. It was taken after an exhaustive weeklong shoot. My collaborator at the time, began asking me personal questions, while I my face looked into the camera. Over a period of 10 minutes, I started of laughing and ended up crying.

To create the sensing mode, I used a video camera to monitor the movement of the audience. After the failures of using Flash with “Feel Trace”, the software was written using Max MSP/Jitter[v].

Conclusion of “Feel Inside” The piece was tested four times, mostly at Wellcome Department of Nueroimaging (WDIN) at UCL (shared with the neuroscientists), and once at the Beta_Space at the PowerHouse Museum in Sydney. Parts of the work were successful.

Were the viewers taken on a journey or controlling the journey? “…there was an interaction which grabbed hold of you as you became more engaged – mainly because there seemed to be an obvious response to what one was doing in relation to the installation. I felt more in control of what was going on than I had when I experienced Feel Trace”.[vi] Initial audience responses showed that using movement as an interaction method lead to a stronger sense of engagement that what was achieved in “Feel Trace”,

It appeared that the interactive model promoting notions of stillness, allowed people to resonate more deeply with the images and sound of the installation. Most reported that they found the interaction more ‘poetic’. However, both spaces provided a difficult context to evaluate the audience experience of the work, as the piece required no movement in order to be interactive. When showing the work in the Powerhouse Museum in Sydney, the audience would experience the work in context of the other more interactive and ‘louder’ works in the space. Some participants slowly discovered the interaction and narrative, giving at least five minutes to the work. Others madly waved at the work, anticipating a stronger reaction. A percentage of the audience walked out of the exhibition space with in thirty seconds, not willing to spend more time investigating the work.

Some participants talked about the vulnerability felt by watching the crying portrait video. This was a success, as it seems (through self reporting) that the video did elicit emotional reactions in the user. On reflection, the narrative of “Feel_Inside” was too complicated for the basic interaction design. Also, if the project were to be exhibited again, the size of the display would be changed in order to make the work more ‘feel’ more intimate.

The coding for the piece was created in Max MSP/ JITTER with engineer David Muth. Using this choice of software was robust, with very few crashes. MaxMSP worked superbly with video and limited real-time effects. Using Max MSP was considered a definite success.

Tina Gonsalves. 2010. Empathy and interactivity: Creating emotionally empathic circuits between audiences and interactive arts. Comput. Entertain. 8, 1, Article 3 (November 2010), 14 pages. DOI=10.1145/1857940.1857943 http://doi.acm.org/10.1145/1857940.1857943

Read More

Single user interaction

I looked at Tina Gonsalves for inspiration when I stared experimenting with the pulse sensor, as a way to trigger interaction using bodily responses, rather then conscious choices. It’s powerful work but feels personal and private, maybe not the kind of interaction people would like in large shared spaces. Still, worth considering for dissertation, maybe in a rethinked space.

Tina Gonsalves, The “Feel” series: An Overview:

The “Feel” series (2005-2007) are an interconnected progression of short films and interactive sketches aiming to sense, translate and provoke the psycho-physiology of the audience.

Synopsis of “Feel Perspire” : Feel:Perspire is a psycho-physiologically responsive video installation, using sweat to trigger footage.The participant’s sweat is monitored using a bio-sensor attached to their finger. When the participant becomes calm, images of clouds verge into abstraction, becoming quite blurry and reminiscent of Rothko’s paintings. If the participant became more nervous, the clouds grow to be more stormy and violent, enveloping the participant in their fury.

Building of “Feel Perspire” : From the feedback gathered, I came to the conclusion that “Feel Insula” achieved the naturalistic interaction and engaging content I was searching for. However, I felt the interaction mode of reading movement was too generic and didn’t allow for the more ‘personal’ sensing mode I initially envisaged. This lead to “Feel Perspire”, a psycho-physiologically responsive video installation, using galvanic skin response (sweat) to trigger footage.

We used the sensor mode of the Galvanic Skin Response (GSR) to trigger video sequences. GSR provides a continuous and immediate response, giving participants a sense of control, creating a biofeedback loop. Biofeedback is a technique in which people are trained to improve their health by learning to control certain internal bodily processes. While using the technologies, there is a shift of the subconscious experiences of heart rate, breathing and nervous system activity to a level of cognitive awareness. Therefore, through reflection, participants slowly learn to identify, sense and eventually coordinate the physiological behaviours being monitored.

I developed and tested a few video databases. The initial video footage for the project was taken from a helicopter. I shot out the window of the moving craft. While shooting, I focused on a horizon. Originally, I wanted to use this in a simple narrative. For example, if the participant’s GSR reading was calm, the footage continues to fly out to a horizon. If the GSR reading rose, the footage would to crash to the sea. After testing, this scenario was deemed to be too potent for the viewer, often enticing them to become more nervous than relaxed

I finally arrived at time-lapsed cloud footage. If the participants relaxed, the footage would blur and become ‘rothkoesque’. If the participant become stressed, storms would rolls in, enveloping the participant in their fury.

At the core of Feel_Perspire was a Max/MSP patch that controlled the computation for the system. We ran the patch on a G5 Apple Macintosh computer under Mac OS X. The output of the GSR was read into the patch through the analog audio input. This signal was used to control the experience of the participants, directly triggering the video narrative. The programming was initially created by Max MSP expert, David Muth. It was difficult to attain a smoothness of video with the constant triggering from the GSR data. The video did not work with programming, leading to a conundrum of how to take the project further.

After three months reflection, I began working with programmer, Evan Raskob, a MAX MSP expert based in London. I altered the concept, working with some stock cloud footage, real-time effects and multiple video channels. I wanted to investigate the use of hue and saturation as an expressive modality, so it was essential that this these effects could be achieved with fluid live effects. Itten, in his book, the Art of Colour, expresses how the mixing of pure color with either white, black or gray to form gradients gives ‘expressive’ power. From the perceptual literature, Valdez and Mehrabian, in their book, Effects of Color on Emotions found that a significant amount in the variance of subject’s emotional response to color came not from the color’s hue, but from the brightness and saturation levels of the color.[i]

Enabling real Real time effects of speed, colour and scale effects was an important step in the research. This created a very smooth and responsive video narrative when triggered by GSR

Conclusion of “Feel Perspire” : We tested the work three times. The initial consumer GSR monitor we implemented in “Feel Perspire” proved troublesome, often dropping out and not giving sensitive enough readings, creating frustration in the participant more than anything else. The following GSR reader (usually used for experiments in laboratory settings) was very large, old and bulky, though extremely sensitive to subtle changes in sweat.

The sensor was attached to the user’s fingers, and dressing the sensor was quite a delicate process. This also hindered the natural interaction scenario, as the user had to remain still and sit in an assigned chair. Furthermore, The GSR monitor needed to be tuned in manually 30 seconds after the participant was hooked up to the monitor, making it difficult to work with in public exhibition contexts. As with “Feel_Trace”, the act of attaching the sensor catalysed an arousal in the body. Adjusting the interface on the monitor was difficult and time consuming. Although the readings proved sensitive, the machine only took a reading every three seconds. Another limitation of using GSR is that it primarily monitors arousal, and does not define emotions. Movement of the body and the variability of GSR data sets across multiple participants provided much difficulty.To tackle this, the GSR monitor needed to be calibrated for each participant.

The simple imagery, using real time effects that responded to the GSR reading, provided a fluid biofeedback interaction scenario. Using GSR as a sensor seemed to result in an engagement “…I wanted to do it for a long time. Feeding back physiological signal on a big screen is really great – one feels good involvement in biofeedback process. This is achievable with a small computer screen but the difference may be like watching movie on the TV or in the cinema”. Initial observations demonstrate that participants felt that the video work was analogous to their psychophysiological state, and giving a sense of control. Past research has shown, that when the participant has a modality of control in the environment they experience a greater sense of presence.[ii] “The most unsettling part of the artwork is that after some time, I started to feel like I could control the content of the feedback, while remaining unable to explain how. Falling into the skies accompanied by stormy winds or quietly floating with resting noise, the artwork transported me throughout neatly intermingled settings related to my feelings”. The interaction design assumed that when the GSR level rose, this meant you were ‘stressed’ triggering video of a storm clouds and loud noise to fill the exhibition space. When discussing the project with Picard, she stated a higher GSR reading could mean you were stressed or happy, and I had not creative a narrative that catered for happiness. The limited sensing modality didn’t allow us to differentiate the data to denote an emotional feeling, only ‘aroused’ or ‘calm’.

Tina Gonsalves. 2010. Empathy and interactivity: Creating emotionally empathic circuits between audiences and interactive arts. Comput. Entertain. 8, 1, Article 3 (November 2010), 14 pages. DOI=10.1145/1857940.1857943 http://doi.acm.org/10.1145/1857940.1857943

Read More

Starting work for final project, right now in testing stage with Kinect and PD. The 2 methods for getting tracking data into PD: Synapse+OSC messages and pix_openni PD objetc developed byMatthias Kronlachner.

Conclusions so far: Synapse needs calibration pose for skeleton tracking, pix_openni works without it more of less accurate. I’m inclined towards the pix_openni as it’s suited for a more natural interaction. A interaction workshop is needed to research the most natural way of interaction with the work

Read More

I’m using pix_data to get pixel data from video, and use that to trigger sound effects using bang!

Read More

Inspired by Nicolas Collins


Read More

I guess using your heartbeat to control the pitch an oscillator will result in a degree of randomness. I can live with that.


Read More

Experimenting with light sensors in our new location.


Read More

The most appropriate research method identified for this project is Video Ethnography, mostly due to the ambiguities and the open-endednes of images as Grimshaw (2004, p4) argues: “these qualities are…what the observational filmmaker attempts to render, viewing them not as a preliminary to anthropology ‘proper’ but as constitutive of a different sort of knowledge practice”. This research is not intended to be qualitative research or data collection, but a collection of video stories, that put together, which will yield a bigger picture. In this case the journey is the goal not the destination, as moving from one subject to another, from one village to other, step-by-step, through these individual stories, a map of the community or in many cases what was left of that community, can be pieced together.

The technique employed, described as observational cinema, according to Grimshaw (2009, p4), “it was grounded in the ethnographic encounter itself, It was fundamentally cinematic, not literary…it involved the abandonment of ethnographic interpretation in favour of a different kind of logic-one that was filmic rather than derived from written text. “

Previous visual work in this area was done mostly in journalistic style, usually in photo reportage format, (the article about Peter Trimper in the online version of Adevarul (2012), first caught the attention on the subject, is clearly a journalistic piece.) While there are other articles in the media, and they are indeed valuable resources for finding potential stories, there is a clear limitation to that approach in terms of depth as well as structure of the material. Getting go know your subjects and build a personal relation with them, gain their trust, and making them feel collaborators, is part of the ethnographic methodology that in the end will yield a more complete and compelling picture. Getting in and out, asking a few questions can work if you want to find facts, but it will not reveal much information about the person, especially in the case of a visual based research.

978x0 978x0 (1)PANO_20140805_185932IMG_20140805_190031

Read More

Useful resource for interactive documentaries




Read More

.Following the signing of the Treaty of Passarowitz in 1718, after the Austrian Empire defeated the Ottoman Empire, the Banat Region is transferred back to the Habsburg administration. In response to the effects of the prolonged war that lead to depopulation the new administration decides to carry out a massive colonisation in order to revive the region, as Batt (2010, p4) explains: “However, the impact of the protracted military campaigns between 1683 and 1716 devastated the region. Towns and villages were destroyed, the rich and fertile land neglected, and the population dispersed.”

In three waves between 1718 and 1787 around 200.000 Germans from Rhineland, Alsace, Swabia, Bavaria and Austria arrive in Banat, settling alongside Romanians, Hungarians, Serbians and Jews, in the major urban centres but also establishing new villages across the region. The map below shows the extent of the colonisation together with the percentage of the German ethnics within the localities.

Census data shows a peak of German population in the 1930, when it made up for around 24% of the total population.

That was followed by a sharp decline after the WW2, when tens of thousands were sent to labour camps in the former USSR as war compensation according to Rusnac (2009 online).

Another cause of decline is the secret ‘repatriation’ deal between the Romanian and West-German governments under which in the 1968-1989 timeframe West-Germany paid the Romanian government between 2000 and 10000 deutsche mark for each migrant, according to their qualifications, reveals Adevarul(2014 online). Apparently this was part of a secret agenda as: “The Ceaușescu regime was set on a course of ‘national communism’ that aimed at homogenizing Romanian society both socially and ethnically. In Banat, meanwhile, Romanians had become some 83 per cent of the population of the two Banat counties.”Batt (2010,p12)

Another sharp decline followed the fall of the communist regime in 1989 when everyone was free to leave. The 2011 Romanian Census shows a mere 13129, less than 1%, ethnic Germans living in the region. Recensamantromania.ro(2011 online). As seen from the Census data, the ethnic composition of the region has suffered a dramatic shift since WW2, as there was a constant flow of population from east to west, with the German ethnics leaving the country, and Romanians from the Moldova and Oltenia Regions being encouraged to move west to Banat, with the result that today ethnic Romanians are the overwhelming majority.

Many villages became deserted and still are today, in others, ethnic Romanians from other regions were encouraged to migrate in order to repopulate. In many cases the change was so sudden that even the original German name of the localities was lost suggests, Rusnac (2009 online).

Nevertheless, the many newcomers largely managed to integrate themselves in their guest communities, keeping the spirit, to some extent, of the multicultural society that existed before, as Batt (2010, p21) suggests, referring to the Banat myth and regarding multi-culturality: “The ‘Banat myth’ is above all the story of a society of immigrants, and thus is, in principle, open to all who come to share in its relative prosperity and advantageous border location, providing they are seen as adopting the local ‘work ethic’ as their own.” The ‘Banat myth’, of peaceful   co-existence of so many different ethnic and religious groups, is not really myth but as many historians had demonstrated was and still is reality, as Neumann (1997,p4) confirms: “Banat was a model of peaceful co-existence between 1800 and 1950, an example to others of mutual understanding and the creation of interethnic and interconfessional relations. Istvan Bibo’s book (1981), A kelet-europai kissallamok nyomorusaga, provides eloquent testimony of this.”

The overall contribution of the German community to the development and prosperity of the region over the last 200 years cannot be overstated according to Rusnac (2009 online). Batt (2010, p7) goes even further suggesting that: “the Germans were everywhere the leading ethnic group, and their overwhelming cultural hegemony can still be felt in Banat today. They drained the swamps, introduced innovative farming methods, viticulture and new crops such as maize, potatoes and tobacco, transforming empty grasslands and fever-ridden swamps into smiling fields within a generation”

A list of personalities from this ethnic group is impressive, including two Nobel Prize winners: Herta Muller and Stefan Hell. Wikipedia (online)


Read More

Recent Comments