Here’s what it does: Flash grabs the webcam image every 60ms, and sends it using a socket connection to a separate C++ server that runs on your localhost. The server uses the OpenCV software library to detect faces, using a haar classifier cascade. When it detects one or more faces it sends the result (coordinates) back to Flash. There, Flash draws a transparent box or image at the coordinates of the face, like this:

What makes this cool is that you can perform quite complex image analysis (at least something you can’t easily do in Flash), and use this data on a platform that is highly suitable for multimedia development (animation, sound, nice graphics, etc).

The demo only handles one face now, but this could be easily expanded.

Download:

• Binary (1,18 MB)
  
• Source (904 KB)

At the moment there are no installation instructions included. If you need help you can contact me, or wait untill I create the instructions (which I will) :)
A few quick pointers to get you past some obstacles:

• Binary: You probably have to add the folder you run the .swf from as trusted directory in your Flash settings. Otherwise you get a security / sandbox violation error.
• Binary: Start the server before starting the .swf (client). The .swf tries to connect to the server when it starts.
• Binary: The needed OpenCV dll’s and .xml file for face detection are included in the same folder.
• Source: (Server) If I remember correctly, only pthread is a required dependency. UPX is a file packer for releasing, Visual Leak detector can be turned off if you don’t want to use it.
• Source: (Server) Additional dependencies you should include when compiling/linking: wsock32.lib cv.lib cxcore.lib highgui.lib

Last note: This was both my first AS3 and C++ project, I’m always open to suggestions, improvements and cool new ideas. Also, when you find this code useful, I would appreciate it if you’d drop me a line.

Font anti-aliasing is done automatically in the browser when you don’t add any effects to it like a GlowFilter. When running AIR the font is not anti-aliased:

Verdana, Not embedded, Browser

Verdana, Not embedded, AIR

If you want to do effects like alpha and rotation on fonts, you have to embed them into your SWF. This also results in anti-aliasing when you run your application as AIR file:

Verdana, Embedded, Browser

Verdana, Embedded, AIR

Though the strange thing is the 0 (zero) and o (O) are missing a bit on the top right.

Here is a quick example. The Jason Voorhees hockey mask moves along with my head in real time (about 60ms delay). The size of the sprite also changes as you move back and forward.

Edit: Or go on a killing spree in style ;)

One of those is We Feel Fine in which new blog posts are scanned every few hours to collect the feelings of the blogger. Each time someone says ‘I feel’ (or I’m feeling, I felt, …) the emotion is logged, next to the gender, age and location. This data is then used in some really pretty visualizations. Especially the photo’s attached to some of the ‘I feel’ sentences can be quite impressive. You can play around with it at their website.

The other application that really interested me is called Universe. The idea behind it is to draw / write the persons, events, etc on the virtual sky. It’s hard to describe it correctly, so you just have to watch the video below, or check out Jonathan Harris’ website.

Now I’ve finally managed to get some things working on my Flash / C++ computer vision project. Zoran (my project advisor at the University of Amsterdam) helped me a lot on the C++ computer vision part. Using the OpenCV library we’ve managed to put face detection in a Flash application! Input from the webcam is sent via a Socket to a light weight server running localhost. When a face is detected, a message is sent back to Flash.

The boundaries of the face in the image are also known. I just have to think of some kind of protocol to send and retrieve different kinds (and sizes) of data.

The result of the image detection from OpenCV (using the red color channel here):


This is how I display the webcam (AS3 code):

// Show the webcam
var camera:Camera = Camera.getCamera();
camera.setMode(WEBCAM_WIDTH, WEBCAM_HEIGHT, 31);
videoDisplay = new Video(WEBCAM_WIDTH,WEBCAM_HEIGHT);
videoDisplay.attachCamera(camera);
this.addChild(videoDisplay);

Making the connection:

private function openConnection(evt:Event):void
{
	// Connect to the external server
	s = new Socket();
	s.addEventListener(Event.CONNECT, initializeImageFeed);
	s.addEventListener(ProgressEvent.SOCKET_DATA, receiveData);
	s.addEventListener(IOErrorEvent.IO_ERROR,ioErrorHandler);

	s.connect("localhost", SERVER_PORT);
}

The timer that sends the feed every x milliseconds (200ms now for me):

/**
* Takes the data of the webcam and sends it to the server
*/
private function sendData(evt:Event):void
{
	var webcamSendWidth:int = 160;
	var webcamSendHeight:int = 120;
	var scale:Number = webcamSendWidth / WEBCAM_WIDTH;

	var scaleMatrix:Matrix = new Matrix();
	scaleMatrix.scale(scale,scale);

	// Get the bitmap from the webcam
	var now:BitmapData = new BitmapData(webcamSendWidth, webcamSendHeight);
	now.draw(videoDisplay, scaleMatrix);
	var rect:Rectangle = new Rectangle(0, 0, webcamSendWidth, webcamSendHeight);
	var ba:ByteArray = now.getPixels(rect);

	s.writeBytes(ba);
	s.flush();
}

The BitmapData that is being sent consists of 4 channels in this order: Alpha, Red, Green, Blue. That is 4 bytes per pixel, which makes 160×120 x 4 = 76800 bytes (75KB) for the complete image. If you want to do this over the Internet, you could compress the Bitmap. Because I’m running both the Flash application and the server on the same computer, bandwidth usage isn’t an issue and compressing and decompressing would probably take longer than just sending it.

This is my humble blog, where I post some interesting stuff now and then. For the world to see it and to support my own memories.

You can contact me at: marijn @ [the URL of this website].

The photos are being placed around the time line, based on the moment they are taken. You can slide through the timeline and zoom in to the level of displaying seconds. The positioning of the photo’s is done using a so called force based algorithm, which is often used in graph drawing. The photo’s are being attracted to the position on the timeline that corresponds with the time the photo was created. Also, the photo’s repulse each other. These two forces result in the intuitive positioning of the images. The more you zoom, the more clear you can see where the photo should really be.

Used technologies:

• Java
• JOGL (OpenGL for Java)
• Force based algoritme for the positioning of the images.

My teacher and assistent were also happy with the result: we won the ‘First Price’ and the ‘OGL Wisselbeker’ Screensaver

Code is available on request.

Flash is very useful for this. Also, reading the webcam input is a lot easier in Flash than in C. So, what’s the idea: Flash reads the webcam, sends the image data to C++. Then the C++ application sends back the meta information about the images / frames, which Flash uses to draw the game graphics and execute actions in the game.

Zinc
My plan was to use Zinc. With Zinc you can convert a .swf file (Flash) to a .exe (Executable). Zinc is actually a container like a browser, using an integrated Flash Player. What’s nice about Zinc is that you can shape the window of your application like you want, and they provide additional API functionality. With the extra API calls you can access low level system commands like rebooting the PC, or information like the CPU speed. It also provides database connectivity with MySQL and Access.

Using the Zinc API you can also access external DLL’s. So, what I tried to do was use this DLL access to analyse the webcam data in an external C++ application that uses existing Computer Vision libraries.
To test the DLL connection, I wrote tiny test DLL (source). It consists of two functies, Init and Process. Init is a pure dummy function, Process returns a small test string.

Flash
Calling the external DLL in Flash (AS3):

// Create the Bitmap
var bmd:BitmapData = new BitmapData(30, 30, false);
var rect:Rectangle = new Rectangle(0, 0, 30, 30);
var bytes:ByteArray = bmd.getPixels(rect);

// DLL
var myDLL:DLL = new DLL("TestDLL.dll");
parameterIndex = myDLL.addParameter("integer", "120");
parameterIndex = myDLL.addParameter("integer", "120");
myDLL.call("none", "Init");
myDLL.clear();

// Send data to DLL
parameterIndex = myDLL.addParameter("unsigned char*", bytes);
var result:String = myDLL.call("unsigned char*", "Process");
myDLL.clear();
myDLL.close();

Result
And we’re done! You would say… Unfortunately, some time profiling shows even the Init dummy function takes 30ms to execute. Sending a 120×120 bitmap takes 180ms. As 10FPS can be considered a minimum you want for analysis, we can conclude this is too slow to be useful.
The next and final attempt (because of time constraints) is to use a C++ server and use a Socket to communicate between Flash and the server.