Modern Viewer

The process of a spectator spending hours in front of a single image absorbed in its narrative is being replaced in the digital era by a process in which the spectator expects “the stimulus of continual visual transformation” (Vaughan, 2005:5).

Idea progression

Found this very interesting implementation of Flash that just so happened to coincide with reading that I’ve been doing.

Coming form a photographic background the image/ photograph to me still has a resonant truthfulness/ reality or power over the viewer. However, from its invention people/ artists/ photographers have played with this idea of reality. This has resulted in the questioning of the photographic image as fact.

There has always been discourse on the ‘truthfulness’ of the photograph, even more so now with digital technology. Now with digital technologies where hyperrealities can be created I am interested in drawing upon this. Hyperrealities have many critics who mention that these are a means of escape from actuality or a form of fantasy. I wish to draw upon the themes of ‘truthfulness’ and escape by creating a power struggle between the image and the viewer. I wish to highlight that these discourses are forever in flux as our technologies change.  My initial idea is to create some sort of little brother system that is integrated within an environment that is recognisable as safe or traditional within the medium of photography. Questioning who is the viewer and the viewed.

There is a natural suspicion of environments that seem to be ‘alive’, and aware of the visitors presence It is unnerving to feel that free and spontaneous actions are being recorded and that they have become active agents within an unseen world” (Colson, Richard The Fundamentals of Digital Art p64)

Code Play 2

Sketch for drawing matrices from James Davoll on Vimeo.

video of image render … still need to link to video capture for dynamic image that is never finished. the idea of a computer continually drawing using pixel analysis to make connections within screen.

All Eyes on You

All Eyes On You from Britzpetermann on Vimeo.

Thanks got to Adrian for brining this to my attention. Very apt for what I have been playing about with!

Code Play

Been looking at processing again using Christopher Webbs – spider web drawing as a start to try and understand how to create some play with rendering the viewer by webcam. Christopher Webb uses matrices and webs of lines that colour match…..been playing about just with jpegs but need to see how the video renders come out….sound linking??? maybe have the detail in response to ambient sound..have some of integer controls with sensors (temp/movement/pressure etc etc the list goes on!!!)

//Original Code form Christopher Webb - Spider Web Drawing.....OpenCV webcam input needed
PImage reference;//image library

int SPEED = 10000;//speed of render
int TESTS = 10;
//the percent that colours must have in common to link together
float ACCURACY = 100;
float DISTANCE = 100;

//stops from getting stuck on a single colour
int MAXTRIES = 20000;//how many colour swtaches/matches

int locTest;
color colTest;
int xTest;
int yTest;
float testTot;

color col;
int loc;
int x;
int y;

int count = 0;
int tries = 0;
boolean newPix = true;

void setup(){
  reference = loadImage("test.jpg");//load image
  strokeWeight(1);//size of web stroke....

void draw(){ 

  if(DISTANCE > 35) DISTANCE -= .1;
  if(SPEED < 4000) SPEED ++;

  for(int i = 0; i < SPEED; i++){

    if(newPix || tries > MAXTRIES){
      newPix = false;
      tries = 0;
      //choose a random pixel to test for
      locTest = int(random(reference.pixels.length));
      //grab its colour/position
      colTest  = reference.pixels[locTest ];
      xTest  = locTest  % reference.width;
      yTest  = locTest / reference.width;   

      //generates the value for that colour
      float rV = colTest >> 16 & 0xFF;
      float gV = colTest >> 8 & 0xFF;
      float bV = colTest & 0xFF;
      testTot = rV + gV + bV;

    //choose a random pixel
    loc = int(random(reference.pixels.length));
    col = reference.pixels[loc];

      x = loc % reference.width;
      y = loc/ reference.width;
      if(dist(xTest,yTest,x,y) <= DISTANCE){
        //enable to make spiderweb like drawings

    //resets counter
    if(count > TESTS){
      newPix = true;
      count = 0;


//tests colour similarities
boolean colorTest(color _col){

  float rV = _col >> 16 & 0xFF;
  float gV = _col >> 8 & 0xFF;
  float bV = _col & 0xFF;
  float tot = rV + gV + bV;

 //check similarity
 float perc = (testTot/tot) *100;

 if(perc >= ACCURACY ){
   return true;
 }else return false;

Another cute idea form Niklas Roy

Little Brother Code

Ive been working on some new code for my little brother sketch so that it can track not only x value but y value as well. The only way I can think of doing this is to create a sequence of images that can be animated together to cover the full arc of 180×180*. Guessing at this stage that it will require a pretty large grid of images to perform a smooth animation. Had a look at project by Ben Mandeberg called look here very similar in idea but using mouse position rather than viewers position. So back to web camera code to figure out viewers face x/y position.


After a good conversation with Rich about projector mapping (brought about by the recent shimmer and lumiere festivals) it seems that v4 has some real potential on the interactive side

– 1 you can attach sensors to it and have them give serial inputs for integers!

-2 you can use opencv plug ins!

A nice example of wat can be achieved

Project Development


The process of a spectator spending hours in front of a single image absorbed in its narrative is being replaced in the digital era by a process in which the spectator expects “the stimulus of continual visual transformation” (Vaughan, 2005:5). I want to explore this new need of the viewer to be continually stimulated as inspiration for a new responsive Digital Facial Image (DFI) hopefully my DFI will respond to the viewers presence and actions creating a unique responsive interaction.

Brother will be produced using a practise based method, utilising and reflecting upon my previous projects, little Brother (fig.1) and Emblematic (fig.2). Both these projects were carried out using a practised based research method. These DFI systems explore separate interactions that the spectator can experience with a responsive artwork, but they share a common thread. Little Brother and Emblematic explore the spectators relationship with computers and their environment and invite the spectator to become a collaborator of the work.

Little Brother is a DFI system that follows the spectator around a room. This system uses a mixture of hardware and software including Processing comprising of a hidden webcam and a monitor. This work visualises the concerns of George Orwell’s 1984. It highlights the modern trend of CCTV and the evasion of privacy. The concerns are intensified and given new meaning with the use of a female figure as the source of confrontation. Little Brother explores the role of technology in modern society by deliberately highlighting questions of privacy. Art spaces traditionally allow the viewer to look at and study the artwork, not have the artwork study them. The resulting confrontation of the spectator being watched and role reversal within the traditional gallery space has an unsettling affect upon the spectator. They realise they are being watched and they themselves are also watching.

Fig. 1 little Brother

Whilst developing Little Brother I experimented with numerous combinations of technology. I researched and tested several sensors including IR, sonar and light. These sensors all worked with differing degrees of success. It was however the use of a webcam as a sensor that both worked more accurately and embodied the sense of surveillance I wished to achieve with the project. During the making process and display of Little Brother I gathered feedback from spectators of the system, these insights helped develop the piece. The main feedback topic was that the spectator felt they needed to understand the relationship between themselves and the screen instantly, this information highlighted the need to keep the themes simple and the surveillance obvious. It also raised issues with the system reliability. It became clear that faults occurred when more than one spectator was within the field of view. The face detection library became confused as to which spectator to prioritise and created a fault in the process. These tests also showed that when the spectator left the webcam’s field of vision the system lost its reference point and therefore repeated the last moments. This was reported to affect the spectators experience of the piece as they could still see the DFI and understood this as a fault. However, when these limitations were not experienced by the spectator the piece successfully visualised that act of being watched and allowed for the spectator to explore this relationship.

For the production of Brother I have reflected upon the feedback attained from Little Brother. I have devoted time to research other methods and achieve a more smooth tracking mechanism that will allow Brother to be more accurate in locating the spectators position thus improving the spectators experience. I wish to continue to use a webcam as the main sensor of the piece as it visualises the concept of surveillance. It was also raised during feedback that the webcam was instantly recognisable to the majority of the spectators and that they instinctively knew that they were being watched.  These details have therefore pushed my research into discovering other facial libraries which are able to be implemented with Processing. These new libraries have included OpenCV, JMyron and FaceDetect.

Emblematic, a collaboration with Adrian Park and the Northern Stage, uses a twitter feed to ascertain how society is feeling. By monitoring this feed the emotions that are being expressed in the tweets are ascertained. These emotions are then represented visually by playing videos that reflect the sentiments of the text. These videos match Ekman’s (1967) six universally recognised emotional states; anger, disgust, fear, happiness, sadness and surprise. Emblematic works using a mix of hardware and software. A tweet feed is passed through an API to rate the emotional value of the text, this then feeds a Processing sketch that selects a relevant video from a prerecorded library of slow motion video clips. Emblematic uses the geo-tags of every Newcastle around the Globe, of which there are 36,  to filter the tweets. This means that the emotions of each Newcastle are represented and provides Emblematic with a global data source.

Still from Emblematic

The resulting experience of Emblematic has several affects. By slowing the emotional expressions down we have allowed the spectator to glimpse the nuances that we as humans make when we communicate. The scientific term for these subtle nuances is ‘emblems’ and they are universally recognised. The spectator freely interacts with the system and understands the emotional response that the DFI is displaying. These emotional reactions are innate to all humans and therefore the viewer can be said to be looking into a mirror of themselves. Every individual makes similar movements in order to communicate emotion through their facial expressions and so each spectator will understand them and recognise themselves within them.

During the production of Emblematic user studies were carried out that allowed spectators to interact with the system in its early stages.These studies highlighted the need of the spectator to understand what the display was depicting as well as allowing the spectator to know that their contribution (tweet) had been received. These suggestions led to development of the project and with each user test, a new version was evolved. The feedback was positive with emphasis placed on the ease of the spectator to reflect upon their own mood as they could recognise that of the DFI.

Brother will further my research into the spectators interaction with the computer DFI. I want to invite the spectator to use their body to explore their relationship with these DFI’s, physically demonstrating their contribution to the artwork, as collaborator. I want to achieve what Ponty (2002) describes as our conscious experience becoming sedimented into our bodily gestures. I wish to achieve this by bringing together elements of Little Brother and Emblematic, as well as, new ideas brought about through my continuing research and reflective e-journal. During my research into face detection systems I have come across several other viable libraries. I have been experimenting with tweaking these libraries to deal with recognising facial emotions, such as a smile. This will require further development, Open CV has thus proven to be more reliable with a broader array of personalised options. The integration of face detection along with emotion recognition will allow Brother to evolve its response to the spectators presence. For example, if the viewer approaches and smiles at the DFI, the DFI will smile back. Providing a stimulus for further dialogue to occur between spectator and machine. By using a DFI I hope to build on the idea that the viewer sees a reflection of themselves within the artwork. The spectator as proven by previous user tests, due to their capital, will intuitively understand that the DFI system is reacting to their proximity. Rokeby (1995:133) suggests that “an interactive technology is a medium through which we communicate with ourselves – a mirror”. It is this idea of interacting and exploring relationships with ourselves that I wish to explore, reinventing the childhood game of hide and seek.

Brother will create responses to the spectators presence. Allowing a narrative performance to occur between the two parties, human and machine. Brother will be able to respond emotionally with the viewer inviting the spectator to become a collabrative performer, “the feelings the spectator has for their movements and perceptions in the performance of viewing the artwork are central to their experience” (Van de Vall, 2008 :141). Brother will be able to track the spectator through 180 degrees by utilising a bespoke motor sensor system, allowing a nostalgic performance of hide and seek or cat and mouse to occur. If the spectator moves beyond the webcam’s field of view Brother will pivot and follow them. This pivot system will involve using servo motors and gears to control a mounted monitor. The servo motors position will be controlled using Processing and Arduino communicating via serial. The noise of the motors will instill a greater sense of machine human relationship. By utilising facial emblems and recognisable emotional states to develop an emotional story between the spectator and system will allow me to explore how the spectator responds to emotional stimulus. From the feedback of Emblematic the spectator reflects upon their own emotional state and is in turn affected by the emotional state of the DFI. It is this same dialogue that I wish to achieve.

The main aims of this project are to:

  • Create a striking piece of digital art that explores the spectators interaction with a responsive system

  • Invite the audience to reflect upon emotional spectrum, empathy, democracy and entitlement

  • Animate a highly visible space

  • Generate greater awareness of experimental digital art amongst the general public by involving them in its creation and exhibiting it in a public space

  • Allow personal experiences of the work and encourage engagement

Other digital artists working with DFI and relevant material that investigate responsiveness and emotional interest include Andy Holtin, Tiffany Holmes, Kirsten Geisler and Niklas Roy. Andy Holtin’s Glance explores the relationship between two DFIs and our role as voyeur. Glance plays a narrative of two DFIs being caught looking at each other. However, Glance does not include the spectator and results in the spectator playing a passive role. Tiffany Holmes’ Nosce Te Ipsum utilises the spectator’s movement towards a screen as a tool. The screen depicts a collage and as the spectator approaches, layers of the collage are peeled away. When all the layers are peeled back an image of the spectator themselves is revealed at the base of the collage. This embodies the concept that the spectator is integral to the production of a piece of artwork. A limitation of this process is that it does not change between spectators and the narrative is the same, only the last image varies. Kirsten Geisler’s Dream of Beauty series looks into our relationship with the DFI and how we interact and relate with a responsive screen based work. Niklas Roy’s My little piece of Privacy playfully utilises OpenCV to block the view of passersby from peering into his studio. A curtain moves to track the passerby therefore creating  barrier between his studio and the spectator. Roy creates an interesting motor assembly and movement tracking system to metaphorically embody his views on privacy. This piece is problematic in that the process attracts attention when trying to avoid it. David Rokeby’s Gathering and Sorting Daemon gathers images of moving people outside the building and then sorts these into colours and shapes. The majority of these people however will not know that their image has been captured and incorporate into a piece of art.

Feasibility Study

Several approaches to this project have been explored yet it would be valuable to further research the use of:

  • Processing – with Java, OpenCV, GSVideo. Processing can also speak to Arduino micro processors allowing data to be transferred between the two through the use of serial.

  • OpenCV Facial recognition Library (after testing Processing’s own facial recognition library it become clear that it was not advanced enough as it became confused when more than one face is present or if the face leaves the field of view. The OpenCV library handles these tasks much more accurately with the ability to personalise its sensitivity and functionality).

  • GSVideo plug in library, alternative video library for processing that has proven to more stable and able to handle a higher bitrate.

  • Arduino – microprocessor that can control motors and read sensors. This will be used in controlling the position of the screen allowing spectators to be tracked.

  • Servo or DC motors – further tests need to be carried out as to which is more effective. Servo motors can be told to go to certain positions whereas dc motors can be run forwards and back.

  • Gear Cog system – to allow for the screen to smoothly turn

  • LCD Screen/ Monitor with integrated Webcam.

  • Bespoke Perspex Frame – to hold motor and gear assembly. Hinged to allow screen to pivot.

  • Mini Mac (or equivalent hardware to run system on) – small, high processing power, plenty of RAM for processing sketch and video library.

Install Diagram

The Dialogue Between Spectator, Artist and the Digital Artwork

The Dialogue Between Spectator, Artist and the Digital Artwork


The purpose of this text is to establish what the spectator

brings to the artwork and how this in turn affects their experience. The intention is to explore 20th and 21st century, digital artworks and how their creators utilise the spectator’s capital to establish new participatory experiences and outcomes. In digital artworks the spectator is seen to be user, collaborator and viewer. This essay seeks to investigate the contemporary issues surrounding the spectator’s dialogue with digital artworks using specific examples. Digital art comes in many forms but for the purpose of this essay I am going to concentrate on what Hansen coins Digital Facial Images (DFI) [Hansen, 2004: p127]. These are digitally generated close-up images of a face that are displayed via technology.

The spectator plays a vital role in the reading of any artwork. They are the instigator of the dialogue and willing participant. Barthes explains that the spectator brings to the reading of the artwork their education, their experiences and their ideas [Barthes, 1993: p143]. John Berger echoes these sentiments in his book Ways of Seeing by observing “the way we see things is affected by what we know and believe”[p8]. Berger goes on to relate to Jean-Francois Lyotard suggesting that we are always under the influence of some narrative, that things have already been told to us and we ourselves have already told the story [Berger, 1990: p12]. This idea that the spectator is the author and instigator of their art-viewing experience is not a new concept but it seems to be evolving with technology. Technology allows for even greater scope when it comes to the spectators individual experience of an artwork. New digital artwork or media according to Lev Manovich “is not something fixed once and for all, but something that can exist in different, potentially infinite versions”[Manovich, 2001: p36]. So, not only can the spectator create infinite connections with the artwork the artwork itself can exist in infinite versions, thus emphasising the concept of individual interactions and experiences.

The following digital artists use the spectator’s capital in numerous different ways in their artworks. They draw upon the spectator’s knowledge of the world and how they integrate with it. The interfaces that these artists use are common to the public. The spectator has been exposed to them before and is therefore familiar with their purpose and process.

Daniel Rozin’s work Wooden Mirrors 1999 (fig. 1) resembles the form of a traditional framed mirror placed upon the gallery wall, but rather than a silver coated glass mirror we are confronted by a network of wooden blocks

. These blocks react and respond to whatever passes in front of it. When a person approaches the piece a camera captures their image and changes the photons

reflected by the spectator into pixels. The wooden blocks pivot changing the level of reflection they cast to form the pixelated image of the spectator standing in front of the Mirror. The sound of the tiny motors is directly connected to the spectator’s interaction and highlights the physical change that they are making to the artwork. The piece works on two levels, as a mirror and as a window [Bolter J D, Gromala D., 2003: p33]. Firstly, the spectator recognises the artwork as a mirror, and approaches it accordingly, seeking a reflection. In doing this, they change the network of blocks to form the desired reflection. As Rozin states, “the simple interaction between the viewer and the piece removes any uncertainty regarding its operation, it is a mirror” [, 1999]. Secondly it can be described as a window due to the fact that the interface between the camera and the blocks is transparent. The interface is not seen in it’s own right, like a window, but it is what lies beyond that is seen. The spectator is unaware of or sees-through the technological process used in the artwork and views only beyond the technological function, the mirror. “An interactive technology is a medium through which we communicate with ourselves – a mirror” [Rokeby, 1995: p133]. Rozin can be seen to make a literal interpretation of this through his work.

Fig. 1 Wooden Mirror by Daniel Rozin

Heidegger [1978] talks about the concept of a tool becoming invisible when the user is familiar with it. His concept of the ‘ready-to-hand’ tool is when the tool becomes an extension of the hand and is no longer perceived as an object but a natural extension of ourselves. These sentiments are seen to be true with modern digital technology. In our western society we are immersed in technology everyday and we no longer seem conscious of technological objects. We interact with the computer without thought. It is this idea of transparency seeing through the physical technology to its function, that drives designers today. “Designers…must mix strategies and create an interface that is both transparent and reflective” [Bolter J D, Gromala D, 2003: p68]. In the Digital Artwork Dream of Beauty 2.2 Touch Me, 1999 by Kirsten Geisler (fig. 2) a small framed portrait of a beautiful blemish free woman stares directly back at the spectator. The spectator is invited to touch the portrait both through the title of the piece and accompanying labels. This allows the spectator to physically interact with the Beauty. This is an act usually not associated within the gallery space and so is intriguing to the user. They intuitively understand how a touch screen works due to being exposed to it in everyday life, but perhaps do not know the technological process and components behind it. Heidegger explains this state of transparency by saying that “the World of tools is an invisible realm from which the visible structure of the world emerges” [Heidegger, 1978: p69]. Depending on where the DFI is touched ‘she’ will react with varying emotions, laughing, weeping or even blowing a kiss when touched on the lips. The spectator has control of this virtual woman during the interaction and can manipulate ‘her’ reaction as they wish. Due to the instant response of Dream of Beauty 2.2 the spectator knows that their intervention is what has caused the reaction on the screen. The spectator has power over the narrative of this artwork. Geisler’s work adheres to the theory that “computers can pretend to be intelligent only by tricking us into using a very small part of who we are when we communicate with them” [Lev Manovich, 2001: p34]. Dream of Beauty can only react in predetermined gestures, we cannot change the gestures merely play them in a finite number of mathematical combinations. Manovich goes onto explain this type of interactivity as “branching interactivity” explaining that the user affects a predetermined set off actions, but, they however cannot change these actions. “The artist is seen to create an audience activated choosing mechanism” [Rokeby, 1995: p136] giving the spectator a sense of control (mirror) as the result is due to their intervention.

Fig. 2 Dream of Beauty 2.2 Touch Me by Kirsten Geisler

The author’s own work Little Brother, 2010 depicts a DFI staring back from a screen, it uses a hidden camera to track the position of the spectator and the DFI stares directly at that position. This has an unsettling affect on the spectator when they realise they are being watched. As the spectator moves so does the DFI following them until they are outside of the field of view. Little Brother links the interaction between body and artwork. The spectator recognises that their movements affect the DFI and that they are the tool by which the digital domain and reality are communicating information and the embodied human that they are [Hansen, 2004: p129].

Tiffany Holmes’ Nosce Te Ipsum, 1999 utilises the spectator’s movement towards a screen as a tool. The screen depicts a collage and as the spectator approaches, layers of the collage are peeled away. When the spectator gets to a predetermined distance, all the layers are peeled back and reveal the spectators image at the base of the collage. This embodies the concept that the spectator is integral to the production of a piece of artwork and, indeed, the first piece to the puzzle. “The (new media) artist now attempts to construct an environment, a system of communication and production, a collective event that implies its recipients, transforms interpreters into actors, enables interpretation to mentor the loop with collective action…it places us within a creative cycle, a living environment of which we are already co-authors” [Lévy, 1997: p123]. Holmes deliberately leaves the work ‘incomplete’; it is missing the initial image to create the collage. The piece invites the spectator to walk towards it, this movement physically demonstrates their contribution to the artwork, as collaborator, which is rewarded with the piece itself mirroring the spectator. Interactors themselves become referents of the work. “The works are akin to portraits” [Rokeby, 1995: p153].

Fig. 3 Nosce Te Ipsum by Tiffany Holmes

All of the mentioned artworks function on an immediacy and hypermediacy

level. They encompass both interfaces that the spectator is aware of, and those that they are not. The spectator can only experience real immersion when true immediacy is achieved. This level of transparency, however, does not exist due to the co dependence of both reflective and transparent interfaces, “our two contradictory logics not only exist in digital media today but are mutually dependent”[Bolter J. Grusin R., 2000: p9]. Instead of trying to achieve this impossibility, Andy Holtin accepts it and even draws attention to it in his piece Contraption for the Influence of Breath II, 2010 (fig. 4) . Holtin plays with the idea of hypermediacy and does not try to hide the interface of his work. Cables, fans and sensors are deliberately left on view for the spectator to explore in an attempt to understand their function. The spectator is invited to blow on a sensor that sends a signal to a network of fans that in turn blow plastic bags. The interface of fans blowing plastic bags passes down the room until it reaches a screen showing a DFI. The DFI then blows back towards the spectator, triggering the fans and bags to blow back mirroring the exact process that the spectator has undergone. Due to the openness of the technology employed and the design of the interaction the spectator is under no illusions of transparent technology. It’s as if the illusion has ended and we can see exactly where we fit into these processes, understanding our role as collaborator.

Fig. 4 Contraption for the Influence of Breath II by Andy Holtin

“The activity the exhibition exists for is between viewer and maker”[Baxandall, 1991: p8] but it seems that to some extent the spectator trades their subjectivity for this participation. Interactive artwork interferes with the spectator’s subjective process of interpretation [Rokeby, 1995: p141].

“The experience of even simple artefacts does not exist in a vacuum but, rather, in dynamic relationship with other people, places and objects” [Buchenau, 2000: p422]. The relationship between artist and spectator is evolving into a collaborative process within digital art. The spectator still brings their capitol to the artwork but they are now allowed in to finish the piece or make it behave in a manner that another may not, therefore creating an exclusive experience for both artwork, artist and spectator. This creates the idea that new media objects assuring users that their choices, thoughts and desires are unique [Manovich, 2001: p42].



Barthes R. (1993) Image Music Text. Fontana Press

Baxandall. (1991) Exhibiting Intention Smitsonian Institute Press

Berger J X. (1990) Other Than Itself. Aperture

Berger J X. (1972) Ways of Seeing. Penguin Books Ltd

Bolter J D, Gromala D. (2003) Windows and Mirrors MIT Press 

Bolter J D, Grusin R. (2000) Remediation MIT Press

Hansen M. (2004) New Philosophy for New Media MIT Press

Heidegger M. (1978) Being in Time Wiley-Blackwell

Lévy P. (1999) Collective Intelligences: Mankind’s Emerging World of Cyberspace, Basic Books

Manovich L. (2001) The Language of New Media MIT Press

Rokeby .D (1995) Transforming Mirrors: Subjectivity and Control in Interactive Media in Critical Issues in Electronic Media State University Press

Online References

Buchenau M, Suri J F. (2000) Experiencing Prototyping. 

Available at [Accessed 10/11/2010]

Kwastek K. Interactivity – A Word in Process.

Available at [Accessed 08/11/2010]