Archive for the ‘Live Image Processing’ Category

Non-linear narratives in video

Wednesday, March 8th, 2006

I was reading this good article called Segmentation and Reassembly of the Digital Moving Image by Susanne Jaschko, and she was giving examples of different works starting from 1993 Toshio Iwai’s Another Time, Another Space which comings and goings of people through the station were filmed by cameras and were manipulated in realtime by the computer to deform shape, time references and showing a different time-space environment on each monitor, and going with Manovich’s soft cinema which she gives as another example of searching new ways in video. Iwai’s example is good because he is playing with the spatiotemporal attribute of time, and it looks like he is the first one to play with this. Zerrfalten by Stephan Schulz and Nelson Vergara is another example of this play. There is an pdf in english tells in detail the process.The important part is segmentation of the data and re-using this in a new sense. The article is stressing the digital moving image’s liberation in today’s culture and manifesting that it is leading to new artistic models representing the complex concepts of time and space.

Well she is right at some point, because I came to this project actually before reading her article. Khronos Projector by Alvaro Cassinelli is a video time-warping machine with tangible deformable screen.

from the site:

The Khronos Projector is an interactive-art installation allowing people to explore pre-recorded movie content in an entirely new way. A classic video-tape allows a simple control of the reproducing process (stop, backward, forward, and elementary control on the reproduction speed). Modern digital players add little more than the possibility to perform random temporal jumps between image frames.

The goal of the Khronos Projector is to go beyond these forms of exclusive temporal control, by giving the user an entirely new dimension to play with: by touching the projection screen, the user is able to send parts of the image forward or backwards in time. By actually touching a deformable projection screen, shaking it or curling it, separate “islands of time” as well as “temporal waves” are created within the visible frame. This is done by interactively reshaping a two-dimensional spatio-temporal surface that “cuts” the spatio-temporal volume of data generated by a movie.

This project is interesting to me in two ways. First, it is good because it is commenting on the linearity that we are having in our everyday lives in front of the screen. It is showing us a new way of perceiving this phenomenon. The second, it is doing this with a tangible interface. This means a transparent interface between the user and the subject. I am really curious about the algorithm behind the morphing between the different timelines of the videos, they look really slick, it looks like opening a door in timeline. I want to do some experiments on this in jitter.

I came across to Khronos project before but now I have found the time to understand it yet better. It has won the Grand prize at Japan Media Arts Festival.

Golan Levin has an informal catalogue of slit-scan video artworks as he named at this site.

lipp project updates

Monday, March 6th, 2006

We have ordered bunch of 880nm IR leds from super bright LEDs. Also I have come up with this useful information from the jitter list. Our webcam’s sensitivity could be dropped to 10-15% when it is sensing 880nm comparing to sensivity at visible wavelengths. We will see what we can do with our ISight.

Someone on the list suggested illuminating the scene with lots of tungsten lights with many layers of deep red gels. Actually Mark helped me suggesting some different gels for this :
Roscolux Colors #19 x 2 (two sheets)
#83 (one sheet)
#90 (one sheet)
and then putting a Lee #87 filter in front of the camera with nightshot mode. Someone on the list also suggested using a ex-view chipped camera which is very useful for IR lights. Sony had one here. Those informations can be useful in the future.

LIPP & NIME collaboration

Friday, February 17th, 2006

Following Josh’s great idea of collaborating for the end semester show at Tonic, I am keeping the sources that I have found which could be beneficial in terms of think-tank. I was reading Joseph Paradiso’s paper Electronic Music Interfaces, and he is pointing out good ideas approaching how to build a electronic musical interface.

I think the question we have to ask to ourselves is before starting any production, how I am going to approach to it. Is it going to be an extension for a current acoustic instrument,or is it going to be a total new interface? Personally I am kind of curious about two things currently. Using slit-scanning as a real time composition tool and using brainwaves to perform and compose.

I have found couple of good papers around those subjects:
Computer Music Research page of Plymouth University, UK
Interfacing the Brain Directly with Musical Systems: On developing systems for making music with brain signals
Tactile Composition Systems for Collaborative Free Sound
An Informal Catalogue of Slit-Scan Video Artworks by Golan Levin.

Those could be a good start for inspiration.-


Monday, February 13th, 2006

Max Mathews and Bill Verplank is teaching a nice class over Stanford for the last couple of years and I am aware of it now. The class is about building/experimenting new musical controllers, videos seem interesting, syllabus might be checked out periodically.

Joseph Paradiso (he is the director of the Responsive Environments Group)has paper dated 1998 where he describes new electronic music interfaces.


Monday, February 13th, 2006

jit.op object lets you perform mathematical operations on all the data in the jitter matrix at once.

math operation will be determined by the attribute @op and provided by an operator.
ex jit.op @op pass + * – * meaning pass the alpha plane, add to the first plane, subtract from the second and multiply the scalar with the last plane.

you can set the size of the jit.pwindow by sending a size message box to its left inlet.

stop stops the movie playback start starts the movie playback.
vol controls the volume of the movie.
frame goes to a specific frame.

You can get important information about the current movie loaded into the object by querying attributes such as duration timescale and fps. Querying gettime spits current time.

Looppoints will help you to set looppoints for your movie. it takes two attributes the starting time of quicktime units and ending time of it.

random attribute gives a random number between 0-attribute.
select gets object/number from its left inlet and compares this with its attributes and according to that bangs its related outlet.
key spits out the ASCII value of the key you press.

jit.unpack objects unpack the four planes into four different planes separating their A-R-G-B values. (multiplane matrix into individual 1-plane matrices)

gate object allow us to route different inlets according to the values.
coll object contains the permutations which are later usable. // 0, 1,2,3 / 1,1,3,2/ 2,2,1,2,3/… you can then use unpack object and send these to the gate object for their outlets. Thus, you can swap between the values.
line object generate ramps and line segments

we can describe the jit.matrix with names, likewise: jit.matrix bob 4 char 320×240.
interp message blurs the pixels of the movie.


Jitter Tutorials Start

Saturday, February 11th, 2006

At last…
‘s primary job is open a movie and read each frame one after other into RAM for easy access by other jitter objects.

jit.window gets that data in the RAM and shows it as colored pixels on the screen. (external window)
jit.pwindow does the same thing in the workspace.

gives mouse coordinates of the screen with its outlets.
scale scales a input range of values to output range.
scale 0 600 -1 1

rate is the playing speed of the movie. 1 is the default. -1 is reverses the movie.

we can map our mouse coordinates as the playing speed of the movie by sending this value by rate $1 to the inlet of the

so we have setcell object where we can set the exact point of the matrix data and display it on the pjit.pwindow.

it works like this: setcell x y val 0-255.

pack object gets number to be packed and spits them as a pack.
pak is similar to pack but all inlets cause outputs.

here is a small snippet that uses pak and allows you to change the matrix data instantly which you can see on the jit.pwindow plus max window.-

#P window setfont “Sans Serif” 9.;
#P window linecount 1;
#P message 288 67 60 196617 clear \, bang;
#P comment 111 108 100 196617 value;
#P comment 60 108 100 196617 y;
#P newex 35 173 52 196617 pak 0 0 0;
#P number 110 129 35 9 0 255 3 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P number 63 129 36 9 0 12 3 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P number 20 129 35 9 0 15 3 3 0 0 0 221 221 221 222 222 222 0 0 0;
#P message 268 112 129 196617 setcell \$1 \$2 val \$3 \, bang;
#P user jit.pwindow 267 238 162 122 0 1 0 0 1 0;
#P newex 269 192 117 196617 jit.matrix 1 char 16 12;
#P comment 22 108 100 196617 x;
#P newex 191 229 46 196617 jit.print;
#P fasten 2 0 0 0 196 210;
#P connect 2 0 3 0;
#P connect 7 0 8 2;
#P connect 6 0 8 1;
#P connect 5 0 8 0;
#P fasten 8 0 4 0 40 200 167 200 167 108 273 108;
#P fasten 11 0 2 0 293 142 274 142;
#P connect 4 0 2 0;
#P window clipboard copycount 12;

New goodies and ideas

Friday, January 27th, 2006

Again one of the nights that I am lost in the infinite world of world wide web jumping from that source to another. Actually I have discovered really interesting stuff.
I am trying to remember how it all started.. I have found this article about Electronic Music In Turkey after I saw the name Bulent Arel who was one the founders of Columbia-Princeton Electronic Music Center and was curious if he was Turkish, which turned out to be he was and tried to do couple of stuff back in the day.

The other thing and maybe the most valuable one is this blog of Trond Lossius who is a sound and installation artist living in Bergen, Norway. It is one of the deepest source that I have come across in a long time. Especially for a newbie like me who wants to educate himself in spatialisation and sound installations. So I am really curious about discovering new concepts in this sense, one of which is Ambisonic Sound I still haven’t quited visualized on my mind. This is Trond’s post about it which he collected from SurSound mailing list.

Also another thing that I was steered through his posts is this software environment called which is if I quote from the site : is an interactive network performance environment invented and developed by composer and computer musician Georg Hajdu. It enables up to five performers to play music over the Internet under the control of a “conductor.”

I believe it could be a good source for the final project in LIPP, I am thinking about a multi-user video composing environment with sound where different performers perform according to the score they have. I must dig this later, especially I am curious about Cage’s latest works called 5, where he uses time-frame in the same score.
~~~ I haven’t read about this, so I must ~~~

Reading Golan Levin’s Paper

Sunday, January 22nd, 2006

I was reading Golan Levin‘s paper about Computer Vision, and this area seems has lots of potential overall. In the beginning of his paper he gives a historical background of Computer Vision through Myron Krueger‘s works, a figure that coined the term “artifical reality”. It looks like it is the first HCI using camera and computer vision. Another peak point for me about this project, it allows two participants in mutually remote locations to participate in the same projection. That is a great route I should be aware of.

Next piece is Levin‘s collaborative work with Lieberman called Messa di Voce. Basically it is a tracking system of your head which spits out circles according to level of your sound that is transforming through a mic. Since it addresses good points about HCI and possible outcomes of computer vision in this sense, it is a good project.

There are numerous other projects as well but the one I like, since it is using physical computing, is an installation called Standarts and Double Standarts(2004) by Rafael Lozano-Hemmer. This work consists offifty leather belts suspended at waist height from robotic servo-motors mounted on the ceiling of the exhibiton room. Controlled bu a computer vision-based tracking system, the belts rotate automatically to follow the public, turning their buckles slowly to face passers-by. In its conceptual sense, it “turns a condition of pure surveillance into an ‘absent crowd’ using a fetish of paternal authority: thebelt”.

In the second part of the paper Levin explains elemantry computer vision technics which are frame differencing, which attempts to locate features by detecting their movements, background substraction, which locates visitor pixels according to their difference from a known background scene, and brightness thresholding, which uses hoped-for differences in luminosity between foreground people and their background environment.

He has a detailed explanation in here with code samples: Link.

Shiffman’s Reactive and Swarm

Thursday, January 19th, 2006

First the precious link And from the papers :

Can an image behave? In other words, if a digital image is a visual representation of colors (i.e. pixels) on a grid (i.e. screen or piece of paper), what if each element of this grid were able to act on its own? A series of experiments in answering these questions led me to create Reactive, a live video installation that amplifies a user’s movements with exploding particle systems in a virtual space.

Reactive began as an experiment in taking a digital image and mapping each pixel in a three-dimensional space. A low-resolution image (80×60 pixels) is mapped to a grid of 2400 pyramid shapes, each colored according to RGB values from the source image, and each with a “z-axis” position according to that color’s brightness. Suddenly, this still image manifests itself as a floating particle system with a one-to-one relationship between pixels and particles.

That is a really interesting topic to dig. To perceve everything on the screen as composition of pixels. The latest examples that Shiffman showed as at Nature of Code are basically spawned from this idea as well, but applying algorithms based on imitation of Nature. Those were really slick examples by the way.

Again from the Swarm:

Swarm is an interactive video installation that implements the pattern of flocking birds (using Craig Reynold’s “Boids” model) as a constantly moving brush stroke.

Swarm is implemented as a system of 120 boids following the rules outlined by Reynolds. In my system, each boid looks up an RGB color from its corresponding pixel location in the live video stream. If the viewer stands still, his or her image will be slowly revealed over time as the flock makes its way around the entire screen. If the viewer chooses to move during the process of painting more abstract shapes and colors can be generated.

Here we can see he is using Reynolds’ Boids as the drawing tool.