Here’s a mockup of my final project. It’s called Camerami. No, it’s not a camera. Yes it will teach you how to use a camera. And perhaps even make you a better photographer (whatever that is).

But I guess that’s the nature of research, particularly when I didn’t come to the course with a particular research topic in mind, nor any experience with practice-lead research. Most of my work on the course so far has arguably been figuring out what practice-lead research means to me and, more importantly, how it relates to my career post-Masters.  I’ve found two things particularly challenging through the course – identifying the common thread through my broad interests (if any), and distilling from those broad interests a topic of research that is small and focussed enough to be  covered adequately in the context of a part time research Masters. I think I am close to  achieving an 80% proof single malt topic now (though, I’d be happy with 60-70% proof, drinkable glass of single malt Scotch right now!).

I’ve just had the necessity to provide in an email a concise summary of what I am proposing as my final research topic. I’m quite satisfied with this as a summary, or at least a snapshot of where my thinking is currently.

The aspect of my research so far that I am going to take forward is the focus on Photography as an act of enquiry rather than the Photograph as an output of the process. The important implication for me being that, if you examine Photography as an act of enquiry and the camera as the primary tool in that act, then changing the tool necessarily changes the nature of the act and enquiry. I find that quite fascinating and the argument leads directly to topics in the domain of HCI, Interaction Design, User Experience Design e.t.c – all topics in which I have a deep interest. So what I’m proposing as a core part of my research is to record the different ways in which different photographers with different cameras explore the same space. I’ll then take the data gathered, examine it and visualise it in a way that, hopefully, reveals interesting insights into the relationship between camera and operator, how the camera affects the nature of the enquiry and how that affects the outcome of the process.

I have also done a lot of thinking of late around what I want to do post-Masters. I’m going to return, with re-ignited passion, to my interests in User Experience  and Information Design and what, in the web development domain, would be called Front End Development. I hesitate to label my interest as that since my interest is actually broader. It may be better described as some combination of motion and graphical programming, programmatic data visualisation and application development. The best label I have that wraps up the whole lot concisely is “Creative Coder”. I’ve also set my sights on applying these skills in the area of eLearning. I think there is untapped potential in the application of these skills to the production of digital learning materials and I plan to explore that potential.

So, I might describe myself as a User Experience Designer and Creative Coder specialising in eLearning.

So, bringing this back round to my practice, I’m proposing to investigate what opportunities the combination of Apple’s iBooks and Creative Coding using HTML+CSS+Javascript based technologies may offer for eLearning content and tools (think Processing.js, Paper.js in iBook form).

And finally, tying the research and practice together, my final submission will be an iBook that combines the results of my research with theory of technology and photography, using interactive digital media in iBook form to create a hybrid photobook and photography guide that contrasts strongly with the majority of derivative photography books to be found on most high street bookstore shelves.

In my continuing effort to understand this thing we call Photography, I’ve been researching visual perception in the hope of better understanding our relationship with photographs. I’ve so far read two interesting books related to the subject: An Anatomy of Thought by Ian Glynn and The Object Stares Back : on the Nature of Seeing by James Elkins. I discuss the first in this post and the second in a later post.

(Note: I read the chapters that I felt were immediately relevant: that is ‘The doors of perception‘ and all chapters in the section Looking at Seeing. This post is based on those chapters.)

I started reading the book with fairly simple and reductionist assumptions about seeing that I suspect many people share – that is that the eye is little more than a mechanism that converts light into an electrical impulse that is sent to the brain which processes and interprets the input. My basic knowledge of the eye based on school biology lessons probably informed this view. I was intrigued to find out that this is not the case and astounded to find out quite how complex the eye actually is and how much processing of incoming visual information happens within the eye before the information is even passed on to the brain.

One of the biggest surprises to me was that the retina is in fact an extension of the brain. Glynn  writes “the retina is the only sense organ that originates as an outgrowth from the developing brain” and that “we should not be surprised, then, that the retina, like other parts of the brain, is involved in information processing” (Glynn, 1999, p.155).

Based on my limited knowledge of the eye, I’d imagined that a photoreceptor simply passed its information directly to the brain independently of other photoreceptors. Glynn explains that, actually, the nerve cells attached to the rods and cones of the retina form a complex, interlinked structure in which the stimulation of one small area of the retina may inhibit neighbouring areas, with the result that “boundaries between light and dark areas of the visual field are sharpened” (Glynn, 1999, p.157). This sounds to me like the ‘edge detection’ of computer vision parlance and is exactly the process photographers frequently employ when post-processing their images in Photoshop.

Glynn provides an example of how sophisticated the preprocessing of image data in the eye may be. He describes how, in 1953, Horace Barlow found that the ganglion cells of the frog retina are very responsive to a small black dot moving rapidly about in the eye’s visual field (Glynn, 1999, p.159). Further, Barlow found that live frogs exhibited a feeding response to the same stimulus (presumably because it resembles a fly). As Glynn observes, “this implied that, in frogs, analysis to extract significant features from the visual scene – analysis of a kind that everyone had imagined could occur only in mysterious ‘centres’ in the brain – can occur to an important extent in the much simpler circuitry of the retina” (Glynn, 1999, p.159). He goes on to point out that “in mammals, less analysis is done in the retina, and much more information is sent to the visual areas of the brain. What is sent, though, is neither a point by point description of the retinal image, nor a summary of crucial features in that image; it is simply the results of the first steps towards an analysis of the retinal image aimed at revealing such features, an analysis which will be taken further by the parts of the brain concerned with vision” (Glynn, 1999, p.159).

For me, one of the key conclusions from this is that, in the process of seeing, a significant degree of processing has happened before we even begin to interpret the visual information – the rational interpretation of the visual data has already been mediated by the eyes. Glynn goes on to discuss optical illusions and “disordered seeing” (Glynn, 1999, p.201), demonstrating how these phenomena are side effects of the neurological processing of visual information. For me, the inescapable conclusion is that, whilst we think that we get a highly detailed and accurate view of the world through our eyes, this perception is actually a very elaborate trick of the mind.

This leads me to wonder whether this may explain some of the power of the photograph and whether photography helps us to overcome some of the shortcomings of our visual system. Clearly, we do not perceive single snapshot images but a constant stream of visual information, from which, our visual system is exceedingly good at extracting the information that is most essential to our survival. But, we can’t take everything in and, as Glynn shows, significant portions of the visual information entering the eye never even make it to the brain. Photography allows us, to a certain degree, to circumvent these limitations of our visual system, freeze a moment in time with a relatively high degree of detail, and study the scene exhaustively.

Another conclusion Glynn’s account of the visual system leads to is that the way we see the world is very different to the way a camera sees it. Generally speaking, the photographic image is more akin to the “point by point description of the retinal image” (Glynn, 1999, p.159) as taught in school biology classes. The implication is that, if a photograph is to match our visual perception more closely, certain processing will need to be done on it and this processing should match as closely as possible the processing that goes on in the retina and visual cortex. In fact, many of the standard transformations made to photographic images, either in or out of camera, are designed to do exactly this: contrast adjustment, sharpening, colour correction are all aimed at transforming the image to appear more like we see the world and less like the camera sees the world. For me, as a lens based artist and practitioner, this is an interesting observation in the context of the significant mistrust associated with processing images. Of course, it goes without saying that this mistrust is very well placed in many contexts (photojournalism, legal evidence e.t.c.), but it may suggest a possible yardstick for quantifying what is acceptible?

Glynn, I. (1999) An Anatomy of Thought – The Origin and Machinery of the Mind. London: Weidenfeld & Nicolson.

We held our inaugural Tinker Tuesday today. The idea is that all the Digital Media Masters students get together on a Tuesday to tinker.

Ben Holden asked me about HTML5 and we got to discussing the relative merits of HTML5+JavaScript compared to Flash+ActionsScript and Java applets compiled from Processing. This led to the topic of Processing JS and Ben asked how one goes about porting a Processing app to Processing JS. As I hadn’t yet played with Processing JS, I set about finding out.

Turns out it’s dead simple: with the processing.js file downloaded and included with the sketch files, all that’s needed is a simple HTML file that includes processing.js and has a canvas element with a property called ‘data-processing-sources’ with its value being a list of all the .pde files separated by spaces.

Check out the example produced in the session.

When I got home I decided to test it in Safari on the iPad. The experience of using a finger rather than the mouse was strangely satisfying and engaging. I was inspired to try and enable multitouch so that more than one finger could be used to spawn sparks. Turns out that is also very straightforward – simply write in a touchMove() method to respond to multitouch events and you’re good to go! I also took the liberty of tweaking the aesthetics and physics.

Check out the multitouch version (it will work in a standard browser but multitouch will only work on an IPad and any multitouch enabled device. Obviously).

Source code for both

Update 10/02/2012: Loren asked about modifying this so it can be loaded from the file system without the need for a web server. I found that including the Processing code as inline code rather than linking to the .pde files solves the issue. I’ve updated the source code zip with an extra file called index-inline.html which demonstrates this working (tested in Chrome, Firefox and Safari on OS X Lion).

[The progress of the project so far has been documented extensively in Evernote. This series of posts pulls together this documentation in a more coherent form and provides an account of the progress so far rather than the documentation itself]

I have recently become interested in the relationship between mental disorder – specifically schizophrenia – and creativity and have discovered some interesting parallels with my thinking around the Poetic Camera.

I am conscious that the relationship between mental disorder and creativity is topical, particularly in the realm of pop psychology. A quick review of literature on the topic is all that is required to reveal that this is a contentious, and sensitive, issue. I don’t intend to use my research to draw any conclusions nor make comment on the matter, but only to consider how current thinking may inform the development of the camera.

In the introduction to their article Relating Schizotypy and Personality to the Phenomenology of Creativity, Nelson and Rawlings summarise the discussion of their research as follows:

These findings indicate that “positive” schizotypy is associated with central features of “flow”-type experience, including distinct shift in phenomenological experience, deep absorption, focus on present experience, and sense of pleasure. The neurologically based construct of latent inhibition may be a mechanism that facilitates entry into flow-type states for schizotypal individuals. This may occur by reduced latent inhibition providing a “fresh” awareness and therefore a greater absorption in present experience, thus leading to flow-type states.

As suggested in previous posts on the Poetic Camera, my aim is to collapse the temporal divide between acquiring a photographic image and editing it further in the digital darkroom. I am aiming to create an experience in which the phenomenological exploration of the world, the interpretation of that experience and the recording of the perceptual response is unified and holistic – a “flow”-type experience. The aspects of this type of experience which Nelson and Rawlings describe – a “distinct shift in phenomenological experience, deep absorption, focus on present experience” – correspond exactly with features of the type of experience I would like to enable with the Poetic Camera, and also with the features typical of pinhole photography.

How might research on schizophrenia, particularly with regard to creativity, inform the design and development of the Poetic Camera?

In the same article, Nelson and Rawlings discuss in detail the concept of latent inhibition. They define it as:

Latent inhibition is a neurologically based concept that describes the phenomenon of attenuated attention to stimuli upon repeated exposure. It is based on the notion of a gating mechanism that allows organisms with complex nervous systems to cease responding to stimuli with no apparent motivational or emotional value.

The less effective this gating mechanism, the more latent inhibition is said to be reduced and the more the organism’s nervous system responds to external stimuli. Nelson and Rawlings state “reduced latent inhibition has been found to be characteristic of the schizophrenia spectrum” and “has also been found among creative individuals…and associated with personality traits that correlate highly with creativity.” They go on to observe “it has been suggested that reduced latent inhibition may facilitate creativity by allowing access to a greater inventory of unfiltered stimuli during early processing, thereby increasing the possibility of combining this information in an original manner.”

It may be argued that traditional camera devices respond to and process only one external stimuli – light. This may be thought of as a high latent inhibition. The Poetic Camera, in it’s current incarnation, forms an image over a period of time (it will almost certainly remain this way) and this provides ample opportunity for the device, and the output, to be exposed to multiple other stimuli, thereby reducing latent inhibition. Examples of stimuli it may respond to and which may influence the output are auditory noise, temperature, and movement, both of the device itself and within the scene in front of it.

Nelson and Rawlings go on to suggest “it is precisely this newness of appreciation, and the associated sense of exploration and discovery, that stimulates the deep immersion in the creative process, which itself may trigger a shift in quality of experience, generally in terms of an intensification or heightening of experience.” It is my hope that by understanding the nature of flow and creativity, especially as revealed through existing research on schizophrenia, and embedding these concepts into the device, it may better afford this type of experience when using it to explore and record the phenomenological world.

It is important to highlight at this point the Context Photography project by Ljungblad, Håkansson, Gaye, and Holmquist. The team built a application to run on Nokia 6600 and 6630 phones which used various sensors on the phone to map input to visual affects in the final still images in a manner very similar to that proposed above. One of the conclusions they arrived at through the project was that these “mappings and representations need to be thoroughly designed, but should still be open for the user to use, interpret and appropriate”. As part of the development and testing of the Poetic Camera, it is intended that the means to ensure any such mappings remain open tested. Needless to say, it is hoped that the Poetic Camera project builds on and adds another perspective to the Context Photography project.

Nelson, B., & Rawlings, D. (4 August 2008). Relating Schizotypy and Personality to the Phenomenology of Creativity. Schizophrenia Bulletin, 36, 388-399. doi:10.1093/schbul/sbn098

Image by Flickr user thoburn (http://www.flickr.com/photos/thoburn/)

[The progress of the project so far has been documented extensively in Evernote. This series of posts pulls together this documentation in a more coherent form and provides an account of the progress so far rather than the documentation itself]

In the course of my research for the essay mentioned in the previous post I discovered Renée Creager O’Brien’s PHD dissertation entitled The Post-Romantic Vision of Pinhole Photographery. I am not a pinhole photographer myself so upon reading her dissertation I discovered an approach and philosophy of photography that seemed to correlate very closely with what I was (or am) trying to achieve with the Poetic Camera.

In the abstract for the dissertation, O’Brien (1998) writes “pinhole photography is a medium that intimates time, mystery, intuition, reflection and a personal vision. It is an approach to imagemaking that creates images based on the flux and flow of occurrences. Its phantasmagoric qualities are closer to dreams than visual reality. Pinhole photography is a holistic process grounded in experimentation. In its method, it deconstructs the camera and image but rediscovers the magical ceremony.” (p.1) Ideas such as intuition, personal vision, holistic, experimentation, and deconstruction of the camera are all ideas and aims the Poetic Camera shares. O’Brien (1998) goes on to write in the introduction “in contemporary art making, the current application of pinhole photographs is an aesthetic choice of artist-photographers. These artists employ pinhole photography not as an art of imitation but as a subjective investigation of the phenomenal world.” (p.3) An “investigation of the phenomenal world” is very much one of the primary purposes of the Poetic Camera, and echoes Flusser’s thinking in Towards a Philosophy of Photography and The Gesture of Photographing (see previous post for more on Flusser).

O’Brien discusses in detail the distinct and unique qualities of pinhole photography both as an experience and also in terms of the resulting images. She highlights the typically long exposures required (p.171) and the qualities of the final image that many might regard as aberrations – for example distortion of perspective, an overall lack of sharpness and vignetting (p.234). These and other characteristics result in a process and results that are unpredictable and susceptible to chance. A high degree of mindfulness and intuition is necessary to succeed in producing interesting images.

It is my intention that continued study of O’Brien’s work and related texts will help to guide the development of the Poetic Camera. Without attempting to simply mimic the results of pinhole photography, I aim to embed the Poetic Camera with similar principles to encourage a creative process of experience and discovery. I will measure the success of the device on the following criteria:

• Does it afford an intuitive, subjective and phenomenological exploration of the world?
• Does it embody unpredictability and chance?
• Does it allow for personal vision and is this vision reflected in the results?
• Are the results aesthetically unique?

O’Brien, R (1998) The Post-Romantic Vision of Pinhole Photographers (Abstract to Unpublished Doctoral Dissertation). Retrieved 24 September, 2011, from http://www.cpalmer.biz/Membership/O’Brien,%20Renee%20Creager/obriendiss.doc

O’Brien, R (1998) The Post-Romantic Vision of Pinhole Photographers (Unpublished Doctoral Dissertation). New York University, New York, United States of America.

After building and experimenting with the Poetic Camera, and liking the results, I felt that I had stumbled across something interesting. But I figured that if I wanted to develop it further, I had to get to the bottom of exactly what it is I liked about it, what makes the device unique and what attributes of the device should be investigated and developed further. So, as mentioned in the previous post, I shelved the device and started thinking about these questions from a theoretical perspective, trying to pick apart the device.

During this time I discovered the writings of Vilém Flusser, particularly The Gesture of Photographing which, in contrast to most other writing and thinking on photography, focusses not on the photographic image but on the act of photography. In The Gesture of Photographing, Flusser examines the act of photographing from a phenomenological perspective. One of the main arguments stemming from this examination is that photography be treated as form of philosophy. By this rationale, by changing various aspects of the gesture, the philosophical perspective is changed – we get a different view and experience of the world.

In my own photographic practice, I’m more concerned with representing my own experience of the world than with reflecting an objective view of the world. Reading Flusser, I realised that, through photography, I was engaged in a philosophical enquiry with the world – observing it in detail, considering it from various different perspectives, interpreting it through different conceptual filters and ultimately forming ideas or theories about the world through this enquiry. I came to the realisation that the tool of my enquiry – the digital camera – was inherently embedded with a programme that was frustrating my enquiry. Ultimately, I realised that what I found so compelling about the Poetic Camera device was that it enabled me to alter the programme of the camera at a fairly fundamental level and in a way that enables me to explore, discover, enquire and learn about the world in a less restrictive manner than a standard digital camera would.

A decided to explore this line of enquiry further in an essay written as part of the theory side of the Digital Media Master of Research programme. The introduction to this essay is included below and the full essay is available for download here: Exorcising the Apparatus or How Not To Be Possessed by the Camera (note that it does not reference Flusser’s The Gesture of Photographing as the English translation written by and very kindly provided to me by Nancy Roth has not yet been published). The essay does make reference to Renée Creager O’Brien’s thesis, The Post-Romantic Vision of Contemporary Pinhole Photographers. Though I have never done any pinhole photography myself, in reading this thesis I realised that there was a high degree of correlation between my approach to photography and the approach of pinhole photographers and between the program (Flusser, 2007, p. 10) of the Poetic Camera and the program of pinhole cameras. The following post discusses these correlations and how they may inform the development of the Poetic Camera.

The form of modern digital cameras reflects a particular preoccupation of photographers, a preoccupation apparent from the dawn of photography: that of obtaining an exact mirror, and some might say, reproduction of the camera’s subject. But the creation of representational images is not the only use of the tool: many use the camera for a more subjective, individual exploration and recording of the world. Mainstream cameras, with their focus on accuracy and objectivity, limit the potential of the tool for these, more poetic, pursuits. Traditionally, the relatively high costs and technical knowledge required by photographers to create their own cameras has limited the scope for alternative uses of the tool. Now, digital cameras, based on software and information processing principles, provide new opportunities for photographers to modify their functions for novel and unique purposes. This essay aims to demonstrate the bias towards accuracy and objectivity in the camera’s evolution and to establish how software may provide photographers with the means to subvert this evolution to create new and interesting forms of photographic tool.

Flusser, V. (2007) Towards a Philosophy of Photography. London: Reaktion Books Ltd.

Poetic Camera is a photography system I created for the physical computing module of the Doing channel of the Digital Media Masters programme. It was born out of a questioning of the fundamental nature of the pixel. Following are examples of the types of questions I set out to explore:

• What exactly is a pixel?
• Why is it square?
• Can it be another shape?
• What information does it contain?
• What information does it not contain and is there any other information it should contain?
• If the information it contains, its shape, its position e.t.c changes, is it still a pixel? Is the image that it represents still a photograph?

The device itself consists of a webcam connected to a computer alongside an Arduino with a couple of switches to provide basic control of the system. The webcam, Arduino and switch assembly are enclosed in a box modelled closely on Kodak’s Box Brownie camera. The styling of the box was chosen to confound expectations of the type of imagery that webcams and computers might produce. The box is connected to the computer by a single USB cable.

The computer runs a processing sketch which accepts input from the webcam and responds to input from the switches to decide how it processes and stores incoming image data. The basis of the image processing is almost entirely derived through various answers to the final question in the list above. The image processing algorithm transforms some or all of the incoming pixels into different shapes and sizes and renders them to a canvas, producing a wide variety of renderings of the scene in front of the camera.

By tweaking various variables in the algorithm, results can be obtained that are sometimes derivative and mimetic of painting styles, sometimes representative, sometimes abstract, sometimes aesthetically pleasing, sometimes not, but almost always interesting and surprising. Below are a few examples of the output, all of the same scene:

The reception of the device amongst coursemates, friends and teachers was overwelmingly positive. Generally people seemed intrigued but the device and it’s results and several people wanted to use the device themselves. On a personal level, I very much enjoyed using the device and exploring my surroundings with it and discovering a unique perspective on the world. What I found especially compelling about it was the ability to tweak and experiment with the image processing algorithm whilst shooting.

A traditional image processing workflow in digital photography incorporates image processing as a stage distinct from the capturing of images – the photographer captures a series of images and then, at some later time, processes them using a computer and image processing software. As a photographer, I am often frustrated by the temporal break between capture and processing. I feel it robs the process of spontaneity, accident, serendipity and surprise – for me, key ingredients in the creative process of image making. With the Poetic Camera I found that the image processing algorithm (and the relatively low tech webcam) resulted in a lack of predicability of the output. This, and the ability to relatively easily make small tweaks to create significantly variable results, sometimes ‘accidental’ and often surprising, resulted in a more complete and fulfilling creative experience. I felt that there was more ‘flow’.

On completion of the project, I shelved the project to all intents and purposes in order to concentrate on and experiment with other ideas and technologies. But, based on the reaction it received and how compelling I found the device and creative process it enabled, I feel there is much scope for further investigation, testing and development. In the intervening period I have thought much about the device from a theoretical and practical level: what various theories and philosophies of technology may reveal about the whys and wherefores of the device, how they may inform its development, what theories and philosophies of images and creativity may be embodied in the device and how these may be embodied in the device in future, and also what technologies may be used in the development and refinement of the device. Inevitably, the outcome of this thinking and question process is a large number of possible manifestations of the device to explore! I feel an important next step in the process is to weed out and select those that best represent the ‘spirit’ of the device as I see it. Which leads to the next line of enquiry: what exactly am I trying to achieve with the device? What is the ‘spirit’ of the device as I see it?

These questions are explored further in the following posts in this series.

With the basics of the patch figured out, I started exploring how I might make it better reflect the Fragmentation theme and the theory behind the idea.

Though the image had ended up being built from ‘fragments’ of a larger image, I wasn’t satisfied that the concept of fragmentation came across clearly enough in the aesthetics. Since one of the idiosyncrasies of Max is the distorted, fragmented sounds and graphics it produces, I decided to try an take advantage of this idiosyncrasy. Since the basis of the piece is about movement around a virtual window to reveal fragmentary views, I decided to try and use speed of movement to introduce fragmentation. After a bit of investigation and exploration, I discovered the jit.scissors object and its brother, jit.glue. I settled on a simple approach of chopping each of the 4 image fragments into 4 more fragments and then using a speed value as the basis for how they’d be reassembled. Back in my Javascript code I translated the movement of the face into a speed value from 1 to 4 which was passed on to a series of switch objects that sit between the jit.scissors and jit.glue objects. These switch objects reroute fragments to various jit.glue objects based on the speed value. The higher the speed value, the more the fragments are re-routed to the ‘incorrect’ outlets. Here’s an example of the result:

It took a while to get the effect right but I really liked the finished effect. It definitely says “fragmentation” and it reflects the theory nicely in that each person sees a different view based on their interaction with it. What I also liked is that, when interacting with the system, the user is rewarded with clearer views by remaining still or moving slowly. For me, photography is very much a contemplative activity and I find that by sitting still in one position, just looking, I begin to see more clearly and more fully. I often get more decent images this way than by moving around and looking at a lot of different things. Unintentionally, my sketch had started reflecting my personal approach to photography and seeing!

With this done and a bit more time in hand, I decided to explore the sound features of Max so set about using sound in much the same way I’d used movement to introduce distortion and fragmentation. Using a similar mapping, I decided to experiment with increased sound levels creating more distortion. I figured this reflected my photographic practice in the same way in that to be still is to be quiet and noise distracts from seeing. I ended up using the sound levels to alter the RGB values of the image fragments such that the brightness of the images is reduced with increased noise. Whilst the result is interesting, I don’t think it’s developed enough and doesn’t seem to fit with the feel of the rest of the piece.

With limited time before the presentation, I finally set about polishing up the aesthetics a little. In order to enhance the effect of seeing a view through window, I decided to very simply place a window frame over the images fragments. Here’s the finished result, with distortion and without:

I’ll shortly upload the patch for reference…

So, moving on from thinking about the Max assignment, I set about figuring out how I might implement such a sketch and then seeing how I might do it in Max.

When learning new tools, I get bored very quickly just following tutorials through from start to finish. Instead, I usually start by quickly scanning through a few tutorials of varying complexity just to get a feel for what is possible. I’m not so concerned with actually understanding how to do things but more with what possibilites and limitations of the tool. I figure out the how later, when I know what I’m trying to achieve.

Back to the task, I very quickly figured it was natural to somehow use face detection and tracking to plot the viewing position relative to my virtual ‘window’. Added bonus: I’ve never done face detection/tracking before so new thing to learn. Yay!

Fellow course mate, Allessandro, pointed me in the direction of Jean-Marc Pelletier’s awesome cv.jit computer vision library for jitter. The library includes an object called cv.jit.faces and the object reference that comes with the download includes an example patch. The documentation states that the object’s output is a “4-plane single row-32″. This, I eventually figured out, is a matrix describing the x and y positions of the top left and bottom right corners of a rectangular bounding box of any face it finds. Once I’d figured out how to grab the first face in the list of faces it finds, I unpacked these coordinates and spat them out as individual send messages to be used elsewhere.

With this done I set about figuring out how to use these values to manipulate the view through a virtual window. The ‘fragmentation’ topic actually inspired my solution: I decided to set up 4 individual jit.pwindows arranged into a block. I then chose one of my own landscape photographs and sliced it up into a 10×10 grid so that I ended up with 100 individual image files in a numbered sequence. The plan was to display a grid of 4 slices in the jit.pwindows in the correct order. I started trying to build the logic to match face tracking values to images slices and quickly grew frustrated that what was becoming a very complex Max patch could be achieved in a few lines of code! Remembering that someone had mentioned that Javascript may be used in Max patches, I investigated this route. Pretty swiftly I had the rudiments of some Javascript to work out what slices to display. I pushed the file names of 4 image slices through the outlets of the Javascript object and used them as sources for the pwindows. With a few tweaks I had it working satisfactorily.

Having tried it out, I was satisfied the result was interesting enough to present in the final Doing exhibition but decided I wanted to work with the implementation and aesthetics a bit more and explore how they might better reflect the theory that had originally led to the idea. That’s the topic of the next post.

Below are a few screen shots of the sketch in action. As I was working on it, I noticed the nice bit of serendipity that, if I left gaps between my pwindows, it created the archetypal window with 4 panes of glass!

Note that the webcam feed is reversed so when my head appears on the right in the image, I’m actually leaning left. I’ve included little diagrams to show my view in relationship to the virtual window.

The source landscape image. Chosen because it was sitting on my desktop!