The process of a spectator spending hours in front of a single image absorbed in its narrative is being replaced in the digital era by a process in which the spectator expects “the stimulus of continual visual transformation” (Vaughan, 2005:5). I propose to use this new need of the viewer to be continually stimulated as inspiration for a new responsive Digital Facial Image (DFI), Brother. Brother will respond to the viewers presence and actions creating a unique responsive interaction.
Brother will be produced using a practise based method, utilising and reflecting upon my previous projects, little Brother (fig.1) and Emblematic (fig.2). Both these projects were carried out using a practised based research method. These DFI systems explore separate interactions that the spectator can experience with a responsive artwork, but they share a common thread. Little Brother and Emblematic explore the spectators relationship with computers and their environment and invite the spectator to become a collaborator of the work.
Little Brother is a DFI system that follows the spectator around a room. This system uses a mixture of hardware and software including Processing comprising of a hidden webcam and a monitor. This work visualises the concerns of George Orwell’s 1984. It highlights the modern trend of CCTV and the evasion of privacy. The concerns are intensified and given new meaning with the use of a female figure as the source of confrontation. Little Brother explores the role of technology in modern society by deliberately highlighting questions of privacy. Art spaces traditionally allow the viewer to look at and study the artwork, not have the artwork study them. The resulting confrontation of the spectator being watched and role reversal within the traditional gallery space has an unsettling affect upon the spectator. They realise they are being watched and they themselves are also watching.
Fig. 1 little Brother
Whilst developing Little Brother I experimented with numerous combinations of technology. I researched and tested several sensors including IR, sonar and light. These sensors all worked with differing degrees of success. It was however the use of a webcam as a sensor that both worked more accurately and embodied the sense of surveillance I wished to achieve with the project. During the making process and display of Little Brother I gathered feedback from spectators of the system, these insights helped develop the piece. The main feedback topic was that the spectator felt they needed to understand the relationship between themselves and the screen instantly, this information highlighted the need to keep the themes simple and the surveillance obvious. It also raised issues with the system reliability. It became clear that faults occurred when more than one spectator was within the field of view. The face detection library became confused as to which spectator to prioritise and created a fault in the process. These tests also showed that when the spectator left the webcam’s field of vision the system lost its reference point and therefore repeated the last moments. This was reported to affect the spectators experience of the piece as they could still see the DFI and understood this as a fault. However, when these limitations were not experienced by the spectator the piece successfully visualised that act of being watched and allowed for the spectator to explore this relationship.
For the production of Brother I have reflected upon the feedback attained from Little Brother. I have devoted time to research other methods and achieve a more smooth tracking mechanism that will allow Brother to be more accurate in locating the spectators position thus improving the spectators experience. I wish to continue to use a webcam as the main sensor of the piece as it visualises the concept of surveillance. It was also raised during feedback that the webcam was instantly recognisable to the majority of the spectators and that they instinctively knew that they were being watched. These details have therefore pushed my research into discovering other facial libraries which are able to be implemented with Processing. These new libraries have included OpenCV, JMyron and FaceDetect.
Emblematic, a collaboration with Adrian Park and the Northern Stage, uses a twitter feed to ascertain how society is feeling. By monitoring this feed the emotions that are being expressed in the tweets are ascertained. These emotions are then represented visually by playing videos that reflect the sentiments of the text. These videos match Ekman’s (1967) six universally recognised emotional states; anger, disgust, fear, happiness, sadness and surprise. Emblematic works using a mix of hardware and software. A tweet feed is passed through an API to rate the emotional value of the text, this then feeds a Processing sketch that selects a relevant video from a prerecorded library of slow motion video clips. Emblematic uses the geo-tags of every Newcastle around the Globe, of which there are 36, to filter the tweets. This means that the emotions of each Newcastle are represented and provides Emblematic with a global data source.
Still from Emblematic
The resulting experience of Emblematic has several affects. By slowing the emotional expressions down we have allowed the spectator to glimpse the nuances that we as humans make when we communicate. The scientific term for these subtle nuances is ‘emblems’ and they are universally recognised. The spectator freely interacts with the system and understands the emotional response that the DFI is displaying. These emotional reactions are innate to all humans and therefore the viewer can be said to be looking into a mirror of themselves. Every individual makes similar movements in order to communicate emotion through their facial expressions and so each spectator will understand them and recognise themselves within them.
During the production of Emblematic user studies were carried out that allowed spectators to interact with the system in its early stages.These studies highlighted the need of the spectator to understand what the display was depicting as well as allowing the spectator to know that their contribution (tweet) had been received. These suggestions led to development of the project and with each user test, a new version was evolved. The feedback was positive with emphasis placed on the ease of the spectator to reflect upon their own mood as they could recognise that of the DFI.
Brother will further my research into the spectators interaction with the computer DFI. I want to invite the spectator to use their body to explore their relationship with these DFI’s, physically demonstrating their contribution to the artwork, as collaborator. I want to achieve what Ponty (2002) describes as our conscious experience becoming sedimented into our bodily gestures. I wish to achieve this by bringing together elements of Little Brother and Emblematic, as well as, new ideas brought about through my continuing research and reflective e-journal. During my research into face detection systems I have come across several other viable libraries. I have been experimenting with tweaking these libraries to deal with recognising facial emotions, such as a smile. This will require further development, Open CV has thus proven to be more reliable with a broader array of personalised options. The integration of face detection along with emotion recognition will allow Brother to evolve its response to the spectators presence. For example, if the viewer approaches and smiles at the DFI, the DFI will smile back. Providing a stimulus for further dialogue to occur between spectator and machine. By using a DFI I hope to build on the idea that the viewer sees a reflection of themselves within the artwork. The spectator as proven by previous user tests, due to their capital, will intuitively understand that the DFI system is reacting to their proximity. Rokeby (1995:133) suggests that “an interactive technology is a medium through which we communicate with ourselves – a mirror”. It is this idea of interacting and exploring relationships with ourselves that I wish to explore, reinventing the childhood game of hide and seek.
Brother will create responses to the spectators presence. Allowing a narrative performance to occur between the two parties, human and machine. Brother will be able to respond emotionally with the viewer inviting the spectator to become a collabrative performer, “the feelings the spectator has for their movements and perceptions in the performance of viewing the artwork are central to their experience” (Van de Vall, 2008 :141). Brother will be able to track the spectator through 180 degrees by utilising a bespoke motor sensor system, allowing a nostalgic performance of hide and seek or cat and mouse to occur. If the spectator moves beyond the webcam’s field of view Brother will pivot and follow them. This pivot system will involve using servo motors and gears to control a mounted monitor. The servo motors position will be controlled using Processing and Arduino communicating via serial. The noise of the motors will instill a greater sense of machine human relationship. By utilising facial emblems and recognisable emotional states to develop an emotional story between the spectator and system will allow me to explore how the spectator responds to emotional stimulus. From the feedback of Emblematic the spectator reflects upon their own emotional state and is in turn affected by the emotional state of the DFI. It is this same dialogue that I wish to achieve.
The main aims of this project are to:
- Create a striking piece of digital art that explores the spectators interaction with a responsive system
- Invite the audience to reflect upon emotional spectrum, empathy, democracy and entitlement
- Animate a highly visible space
- Generate greater awareness of experimental digital art amongst the general public by involving them in its creation and exhibiting it in a public space
- Allow personal experiences of the work and encourage engagement
Other digital artists working with DFI and relevant material that investigate responsiveness and emotional interest include Andy Holtin, Tiffany Holmes, Kirsten Geisler and Niklas Roy. Andy Holtin’ s Glance explores the relationship between two DFIs and our role as voyeur. Glance plays a narrative of two DFIs being caught looking at each other. However, Glance does not include the spectator and results in the spectator playing a passive role. Tiffany Holmes’ Nosce Te Ipsum utilises the spectator’s movement towards a screen as a tool. The screen depicts a collage and as the spectator approaches, layers of the collage are peeled away. When all the layers are peeled back an image of the spectator themselves is revealed at the base of the collage. This embodies the concept that the spectator is integral to the production of a piece of artwork. A limitation of this process is that it does not change between spectators and the narrative is the same, only the last image varies. Kirsten Geisler’s Dream of Beauty series looks into our relationship with the DFI and how we interact and relate with a responsive screen based work. Niklas Roy’s My little piece of Privacy playfully utilises OpenCV to block the view of passersby from peering into his studio. A curtain moves to track the passerby therefore creating barrier between his studio and the spectator. Roy creates an interesting motor assembly and movement tracking system to metaphorically embody his views on privacy. This piece is problematic in that the process attracts attention when trying to avoid it. David Rokeby’s Gathering and Sorting Daemon gathers images of moving people outside the building and then sorts these into colours and shapes. The majority of these people however will not know that their image has been captured and incorporate into a piece of art.
Several approaches to this project have been explored yet it would be valuable to further research the use of:
- Processing – with Java, OpenCV, GSVideo. Processing can also speak to Arduino micro processors allowing data to be transferred between the two through the use of serial.
- OpenCV Facial recognition Library (after testing Processing’s own facial recognition library it become clear that it was not advanced enough as it became confused when more than one face is present or if the face leaves the field of view. The OpenCV library handles these tasks much more accurately with the ability to personalise its sensitivity and functionality).
- GSVideo plug in library, alternative video library for processing that has proven to more stable and able to handle a higher bitrate.
- Arduino – microprocessor that can control motors and read sensors. This will be used in controlling the position of the screen allowing spectators to be tracked.
- Servo or DC motors – further tests need to be carried out as to which is more effective. Servo motors can be told to go to certain positions whereas dc motors can be run forwards and back.
- Gear Cog system – to allow for the screen to smoothly turn
- LCD Screen/ Monitor with integrated Webcam.
- Bespoke Perspex Frame – to hold motor and gear assembly. Hinged to allow screen to pivot.
- Mini Mac (or equivalent hardware to run system on) – small, high processing power, plenty of RAM for processing sketch and video library.