Testing Light Tracker/ User Selection

Testing Processing Sketch from James Davoll on Vimeo.

Carried out 3 user tests on the new sketch to track user position and place a beam of light on their location. This is to highlight that they are in control of the interaction and that the system will respond to their movement.


User 1 – fun, nostalgia of child in a game making you move. Also conjured up a starnge sense of being watched.

User 2 – very accurate and obvious that I was the source for where the light would be projected.

User 3 – would have loved if the light would have tracked over a larger space, also jumped sometimes. Would have beeb better if tested with ore than one person.

Thinking advancement


The process of a spectator spending hours in front of a single image absorbed in its narrative is being replaced in the digital era by a process in which the spectator expects “the stimulus of continual visual transformation” (Vaughan, 2005:5). I propose to use this new need of the viewer to be continually stimulated as inspiration for a new responsive Digital Facial Image (DFI), Brother. Brother will respond to the viewers presence and actions creating a unique responsive interaction.


Brother will be produced using a practise based method, utilising and reflecting upon my previous projects, little Brother (fig.1) and Emblematic (fig.2). Both these projects were carried out using a practised based research method. These DFI systems explore separate interactions that the spectator can experience with a responsive artwork, but they share a common thread. Little Brother and Emblematic explore the spectators relationship with computers and their environment and invite the spectator to become a collaborator of the work.


Little Brother is a DFI system that follows the spectator around a room. This system uses a mixture of hardware and software including Processing comprising of a hidden webcam and a monitor. This work visualises the concerns of George Orwell’s 1984. It highlights the modern trend of CCTV and the evasion of privacy. The concerns are intensified and given new meaning with the use of a female figure as the source of confrontation. Little Brother explores the role of technology in modern society by deliberately highlighting questions of privacy. Art spaces traditionally allow the viewer to look at and study the artwork, not have the artwork study them. The resulting confrontation of the spectator being watched and role reversal within the traditional gallery space has an unsettling affect upon the spectator. They realise they are being watched and they themselves are also watching.


Fig. 1 little Brother


Whilst developing Little Brother I experimented with numerous combinations of technology. I researched and tested several sensors including IR, sonar and light. These sensors all worked with differing degrees of success. It was however the use of a webcam as a sensor that both worked more accurately and embodied the sense of surveillance I wished to achieve with the project. During the making process and display of Little Brother I gathered feedback from spectators of the system, these insights helped develop the piece. The main feedback topic was that the spectator felt they needed to understand the relationship between themselves and the screen instantly, this information highlighted the need to keep the themes simple and the surveillance obvious. It also raised issues with the system reliability. It became clear that faults occurred when more than one spectator was within the field of view. The face detection library became confused as to which spectator to prioritise and created a fault in the process. These tests also showed that when the spectator left the webcam’s field of vision the system lost its reference point and therefore repeated the last moments. This was reported to affect the spectators experience of the piece as they could still see the DFI and understood this as a fault. However, when these limitations were not experienced by the spectator the piece successfully visualised that act of being watched and allowed for the spectator to explore this relationship.


For the production of Brother I have reflected upon the feedback attained from Little Brother. I have devoted time to research other methods and achieve a more smooth tracking mechanism that will allow Brother to be more accurate in locating the spectators position thus improving the spectators experience. I wish to continue to use a webcam as the main sensor of the piece as it visualises the concept of surveillance. It was also raised during feedback that the webcam was instantly recognisable to the majority of the spectators and that they instinctively knew that they were being watched.  These details have therefore pushed my research into discovering other facial libraries which are able to be implemented with Processing. These new libraries have included OpenCV, JMyron and FaceDetect.


Emblematic, a collaboration with Adrian Park and the Northern Stage, uses a twitter feed to ascertain how society is feeling. By monitoring this feed the emotions that are being expressed in the tweets are ascertained. These emotions are then represented visually by playing videos that reflect the sentiments of the text. These videos match Ekman’s (1967) six universally recognised emotional states; anger, disgust, fear, happiness, sadness and surprise. Emblematic works using a mix of hardware and software. A tweet feed is passed through an API to rate the emotional value of the text, this then feeds a Processing sketch that selects a relevant video from a prerecorded library of slow motion video clips. Emblematic uses the geo-tags of every Newcastle around the Globe, of which there are 36,  to filter the tweets. This means that the emotions of each Newcastle are represented and provides Emblematic with a global data source.


Still from Emblematic


The resulting experience of Emblematic has several affects. By slowing the emotional expressions down we have allowed the spectator to glimpse the nuances that we as humans make when we communicate. The scientific term for these subtle nuances is ‘emblems’ and they are universally recognised. The spectator freely interacts with the system and understands the emotional response that the DFI is displaying. These emotional reactions are innate to all humans and therefore the viewer can be said to be looking into a mirror of themselves. Every individual makes similar movements in order to communicate emotion through their facial expressions and so each spectator will understand them and recognise themselves within them.


During the production of Emblematic user studies were carried out that allowed spectators to interact with the system in its early stages.These studies highlighted the need of the spectator to understand what the display was depicting as well as allowing the spectator to know that their contribution (tweet) had been received. These suggestions led to development of the project and with each user test, a new version was evolved. The feedback was positive with emphasis placed on the ease of the spectator to reflect upon their own mood as they could recognise that of the DFI.


Brother will further my research into the spectators interaction with the computer DFI. I want to invite the spectator to use their body to explore their relationship with these DFI’s, physically demonstrating their contribution to the artwork, as collaborator. I want to achieve what Ponty (2002) describes as our conscious experience becoming sedimented into our bodily gestures. I wish to achieve this by bringing together elements of Little Brother and Emblematic, as well as, new ideas brought about through my continuing research and reflective e-journal. During my research into face detection systems I have come across several other viable libraries. I have been experimenting with tweaking these libraries to deal with recognising facial emotions, such as a smile. This will require further development, Open CV has thus proven to be more reliable with a broader array of personalised options. The integration of face detection along with emotion recognition will allow Brother to evolve its response to the spectators presence. For example, if the viewer approaches and smiles at the DFI, the DFI will smile back. Providing a stimulus for further dialogue to occur between spectator and machine. By using a DFI I hope to build on the idea that the viewer sees a reflection of themselves within the artwork. The spectator as proven by previous user tests, due to their capital, will intuitively understand that the DFI system is reacting to their proximity. Rokeby (1995:133) suggests that “an interactive technology is a medium through which we communicate with ourselves – a mirror”. It is this idea of interacting and exploring relationships with ourselves that I wish to explore, reinventing the childhood game of hide and seek.


Brother will create responses to the spectators presence. Allowing a narrative performance to occur between the two parties, human and machine. Brother will be able to respond emotionally with the viewer inviting the spectator to become a collabrative performer, “the feelings the spectator has for their movements and perceptions in the performance of viewing the artwork are central to their experience” (Van de Vall, 2008 :141). Brother will be able to track the spectator through 180 degrees by utilising a bespoke motor sensor system, allowing a nostalgic performance of hide and seek or cat and mouse to occur. If the spectator moves beyond the webcam’s field of view Brother will pivot and follow them. This pivot system will involve using servo motors and gears to control a mounted monitor. The servo motors position will be controlled using Processing and Arduino communicating via serial. The noise of the motors will instill a greater sense of machine human relationship. By utilising facial emblems and recognisable emotional states to develop an emotional story between the spectator and system will allow me to explore how the spectator responds to emotional stimulus. From the feedback of Emblematic the spectator reflects upon their own emotional state and is in turn affected by the emotional state of the DFI. It is this same dialogue that I wish to achieve.


The main aims of this project are to:

  • Create a striking piece of digital art that explores the spectators interaction with a responsive system
  • Invite the audience to reflect upon emotional spectrum, empathy, democracy and entitlement
  • Animate a highly visible space
  • Generate greater awareness of experimental digital art amongst the general public by involving them in its creation and exhibiting it in a public space
  • Allow personal experiences of the work and encourage engagement


Other digital artists working with DFI and relevant material that investigate responsiveness and emotional interest include Andy Holtin, Tiffany Holmes, Kirsten Geisler and Niklas Roy. Andy Holtin’ s Glance explores the relationship between two DFIs and our role as voyeur. Glance plays a narrative of two DFIs being caught looking at each other. However, Glance does not include the spectator and results in the spectator playing a passive role. Tiffany Holmes’ Nosce Te Ipsum utilises the spectator’s movement towards a screen as a tool. The screen depicts a collage and as the spectator approaches, layers of the collage are peeled away. When all the layers are peeled back an image of the spectator themselves is revealed at the base of the collage. This embodies the concept that the spectator is integral to the production of a piece of artwork. A limitation of this process is that it does not change between spectators and the narrative is the same, only the last image varies. Kirsten Geisler’s  Dream of Beauty series looks into our relationship with the DFI and how we interact and relate with a responsive screen based work. Niklas Roy’s My little piece of Privacy playfully utilises OpenCV to block the view of passersby from peering into his studio. A curtain moves to track the passerby therefore creating  barrier between his studio and the spectator. Roy creates an interesting motor assembly and movement tracking system to metaphorically embody his views on privacy. This piece is problematic in that the process attracts attention when trying to avoid it. David Rokeby’s Gathering and Sorting Daemon gathers images of moving people outside the building and then sorts these into colours and shapes. The majority of these people however will not know that their image has been captured and incorporate into a piece of art.


Feasibility Study

Several approaches to this project have been explored yet it would be valuable to further research the use of:

  • Processing – with Java, OpenCV, GSVideo. Processing can also speak to Arduino micro processors allowing data to be transferred between the two through the use of serial.
  • OpenCV Facial recognition Library (after testing Processing’s own facial recognition library it become clear that it was not advanced enough as it became confused when more than one face is present or if the face leaves the field of view. The OpenCV library handles these tasks much more accurately with the ability to personalise its sensitivity and functionality).
  • GSVideo plug in library, alternative video library for processing that has proven to more stable and able to handle a higher bitrate.
  • Arduino – microprocessor that can control motors and read sensors. This will be used in controlling the position of the screen allowing spectators to be tracked.
  • Servo or DC motors – further tests need to be carried out as to which is more effective. Servo motors can be told to go to certain positions whereas dc motors can be run forwards and back.
  • Gear Cog system – to allow for the screen to smoothly turn
  • LCD Screen/ Monitor with integrated Webcam.
  • Bespoke Perspex Frame – to hold motor and gear assembly. Hinged to allow screen to pivot.
  • Mini Mac (or equivalent hardware to run system on) – small, high processing power, plenty of RAM for processing sketch and video library.



Install Diagram

OpenCV Face Detection

After a catch up with Tom (cheers for that) it was apparent that I needed to test out some possibilities for tracking the viewer in potential spaces to ascertain their location. Doing this now will save allot of trouble later.

I had initially thought of doing this either with OpenCV Face Detection within Processing or learning a little more about using the Kinect. As I had some previous experience with OpenCV I was more in line to using this approach. After some tests of tracking an ellipse representing the viewer (me) there are some obvious flaws with the OpenCV library. When the face is turned more than 30*(give or take) to the screen the face detection losses the viewer and believes that no viewer is present. If this was to happen during an interaction the connection between system and viewer would be lost and the image showing would jump all over the place. The other problem is that the actual size of the ellipse represented by the viewers distance jumps dramatically as well meaning that it is unstable and would be very troublesome to plot images to.

It was suggested to look again at the Kinect at its use. That or the OpenCV blob detection library with a camera from above.

After a quick test by Tom of the Osceleton library on the Kinect it seems that this could be a much more smooth, stable option. However, it does have limitations in distances that the viewer can be aware. More testing to be completed in the next few weeks.

project mindmap

Further Play

Having further play with how to implement some movement into stills, again the fact that these are stills and not video footage is making the movements far less subtle as well as highlighting the fact that they are separate images. but a test worth while. I’ll probably have to do his in after effects possibly as I have no idea how Processing handles Gifs and whether or not you can control the speed, playhead position etc.


Playing around very crudely and quickly with how to initiate some subtle movements into final piece. Cinemagraph made form crappy stills camera…..video footage works so much better as you can break down subtle movements into frames and get a more subtle effect, plus a tripod would have been useful. This again is an attempt to make the viewer aware of their own presence and create some self reflection. From project meeting it was brought up that subtle movements by themselves will make the viewer self aware.  As a quick sketch I think that this is something that I can incorporate into project but obviously need to work on the technique a bit more and use video footage…even quick shutter and fps will not look smooth.

Reality/ Simulation/ Hyperreality

As a photographer – I come from a background of being absorbed by the debate – from the beginning of the photographic process there have been discussion about the truth of the photograph. Now technology with digital photography things are openly fake, i’m interested to see how much we  are willing to believe. The balance of truth in todays photography.

Critics have said that we use (Robins) this new technology (digital) to escape form the real World and our fears and to substitute an interaction with the World. With digital media hyperrealities are therefor created. I’m interested on how the dialogue of truth has progressed.

Chemical photography may construct a reality but at least it was the result form an encounter with the real World. This process has now been broken down into binary numbers taking the real and representing it as numbers. It therefore suggests that this process can be reversed and that the numbers can create a reality of their own. Where those the truth lie in this ‘derealised’ (Lister, Martin) World.

These hyperrealities and simulations are not there to imitate the real as photography set out to do, but are creations of their own.

flow diagram

Idea Development

My idea is to create an interactive postcard like image. Traditionally a Postcard shows a view that the individual can purchase and claim as their own. They own it, they have seen this. They are not saying that it has seen them. They are in control of it. They buy a postcard and send it as part of their experience of the place, they are fully in control of the event even down to how they describe it and who they send it to.

A view that is bought, translated by an individual and invites others to view their interpretation. I wish to subvert this… the closer they look the less in control the viewer is. Creating an open discourse of who is the viewer and who is the viewed.

From afar the image will take the form of a recognisable seaside scene. As the viewer approaches the image, the image will zoom into a character that looks at viewer and tracks them around the space if they walk back again the image will return to its original format. If they approach another area of the image another area will be uncovered and zoomed into where another character reacts to the viewers position. I visualise this happening in three stages. Each level closer the viewer gets more becomes apparent of what the image holds.

I’m also hoping to touch upon several Irish legends and myth again to play with this idea of hyperreality. The Banshee and the Selkie would be interesting characters to elude to.

Marie Sester – Access