Face detection using Flash and C++, revisited

March 6th, 2008

Over the past few months I’ve been asked several times for the code for face detection in Flash. So I’ve taken some time to make a short demo application, grabbed from my original Virtual Mirror game (see my two previous blogs).

Here’s what it does: Flash grabs the webcam image every 60ms, and sends it using a socket connection to a separate C++ server that runs on your localhost. The server uses the OpenCV software library to detect faces, using a haar classifier cascade. When it detects one or more faces it sends the result (coordinates) back to Flash. There, Flash draws a transparent box or image at the coordinates of the face, like this:

ActionScript3 + C++, Face detected

What makes this cool is that you can perform quite complex image analysis (at least something you can’t easily do in Flash), and use this data on a platform that is highly suitable for multimedia development (animation, sound, nice graphics, etc).

The demo only handles one face now, but this could be easily expanded.

Download:

At the moment there are no installation instructions included. If you need help you can contact me, or wait untill I create the instructions (which I will) :)
A few quick pointers to get you past some obstacles:

  • Binary: You probably have to add the folder you run the .swf from as trusted directory in your Flash settings. Otherwise you get a security / sandbox violation error.
  • Binary: Start the server before starting the .swf (client). The .swf tries to connect to the server when it starts.
  • Binary: The needed OpenCV dll’s and .xml file for face detection are included in the same folder.
  • Source: (Server) If I remember correctly, only pthread is a required dependency. UPX is a file packer for releasing, Visual Leak detector can be turned off if you don’t want to use it.
  • Source: (Server) Additional dependencies you should include when compiling/linking: wsock32.lib cv.lib cxcore.lib highgui.lib

Last note: This was both my first AS3 and C++ project, I’m always open to suggestions, improvements and cool new ideas. Also, when you find this code useful, I would appreciate it if you’d drop me a line.

AS3 font anti-aliasing in AIR and browser

August 3rd, 2007

First of all, I’m a noob on fonts in AS3. Just posting some differences between AIR (Adobe Integrated Runtime) and the browser I noticed while developing.

Font anti-aliasing is done automatically in the browser when you don’t add any effects to it like a GlowFilter. When running AIR the font is not anti-aliased:

Verdana, Not embedded, Browser
Font quality Browser

Verdana, Not embedded, AIR
Font quality AIR

If you want to do effects like alpha and rotation on fonts, you have to embed them into your SWF. This also results in anti-aliasing when you run your application as AIR file:

Verdana, Embedded, Browser
Font quality Browser (embedded font)

Verdana, Embedded, AIR
Font quality AIR (embedded font)

Though the strange thing is the 0 (zero) and o (O) are missing a bit on the top right.

Face tracking, adding virtual objects in Flash

July 12th, 2007

In a previous blog I talked about using Flash and C++ for face detection. I’m going to use face detection to determine if the player of the game has the right distance to the webcam and is standing in the middle of the screen. But it’s also lots of fun to attach virtual objects to the face (like done in the more expensive Logitech webcam series), or to put filters and effects on the facial area.

Here is a quick example. The Jason Voorhees hockey mask moves along with my head in real time (about 60ms delay). The size of the sprite also changes as you move back and forward.

Face detection fun

Edit: Or go on a killing spree in style ;)

Face detection fun (keep your hat on)

We Feel Fine & Universe

July 11th, 2007

I came across an interesting TED talk by Jonathan Harris. He talked about a few of the applications he has been working on.

One of those is We Feel Fine in which new blog posts are scanned every few hours to collect the feelings of the blogger. Each time someone says ‘I feel’ (or I’m feeling, I felt, …) the emotion is logged, next to the gender, age and location. This data is then used in some really pretty visualizations. Especially the photo’s attached to some of the ‘I feel’ sentences can be quite impressive. You can play around with it at their website.

The other application that really interested me is called Universe. The idea behind it is to draw / write the persons, events, etc on the virtual sky. It’s hard to describe it correctly, so you just have to watch the video below, or check out Jonathan Harris’ website.

Face detection using Flash and C++

June 24th, 2007

As you could read in a previous blog, I’m trying to create a small webcam game in Flash that supports ‘physical input’ using a webcam. The frames are processed by a C++ server that uses existing computer vision libraries. This way it should be fairly easy to build in cool features like face detection.

Now I’ve finally managed to get some things working on my Flash / C++ computer vision project. Zoran (my project advisor at the University of Amsterdam) helped me a lot on the C++ computer vision part. Using the OpenCV library we’ve managed to put face detection in a Flash application! Input from the webcam is sent via a Socket to a light weight server running localhost. When a face is detected, a message is sent back to Flash.

The boundaries of the face in the image are also known. I just have to think of some kind of protocol to send and retrieve different kinds (and sizes) of data.

The result of the image detection from OpenCV (using the red color channel here):
CV face detection
Read the rest of this entry »

About

June 19th, 2007

I’m a 24 year old Software Engineer at one of the biggest websites in the Netherlands. I’m a former Information Science student at the University of Amsterdam and a large part of the articles on this website are related to my master thesis there.

This is my humble blog, where I post some interesting stuff now and then. For the world to see it and to support my own memories.

You can contact me at: marijn @ [the URL of this website].

Khronos – time based photo browser

May 30th, 2007

Together with my team mate Meindert, I’ve created a cool photo browser for the Information Visualisation course given at the University of Amsterdam. A screenshot is displayed below with the end result.

The photos are being placed around the time line, based on the moment they are taken. You can slide through the timeline and zoom in to the level of displaying seconds. The positioning of the photo’s is done using a so called force based algorithm, which is often used in graph drawing. The photo’s are being attracted to the position on the timeline that corresponds with the time the photo was created. Also, the photo’s repulse each other. These two forces result in the intuitive positioning of the images. The more you zoom, the more clear you can see where the photo should really be.

Used technologies:

  • Java
  • JOGL (OpenGL for Java)
  • Force based algoritme for the positioning of the images.

My teacher and assistent were also happy with the result: we won the ‘First Price’ and the ‘OGL Wisselbeker’ Screensaver :)

Code is available on request.

Winnaar 2007

Zinc DLL interface = slow?

April 30th, 2007

At the moment I’m working on a webcam game as part of my Master thesis research. There are many Computer Vision libraries available in C/C++ (for example with optical flow implementations or face detection), but C++ isn’t very suited for rapid prototyping in respect to interface and game graphics.

Flash is very useful for this. Also, reading the webcam input is a lot easier in Flash than in C. So, what’s the idea: Flash reads the webcam, sends the image data to C++. Then the C++ application sends back the meta information about the images / frames, which Flash uses to draw the game graphics and execute actions in the game.

Zinc
My plan was to use Zinc. With Zinc you can convert a .swf file (Flash) to a .exe (Executable). Zinc is actually a container like a browser, using an integrated Flash Player. What’s nice about Zinc is that you can shape the window of your application like you want, and they provide additional API functionality. With the extra API calls you can access low level system commands like rebooting the PC, or information like the CPU speed. It also provides database connectivity with MySQL and Access.

Using the Zinc API you can also access external DLL’s. So, what I tried to do was use this DLL access to analyse the webcam data in an external C++ application that uses existing Computer Vision libraries.
To test the DLL connection, I wrote tiny test DLL (source). It consists of two functies, Init and Process. Init is a pure dummy function, Process returns a small test string.

Flash
Calling the external DLL in Flash (AS3):

// Create the Bitmap
var bmd:BitmapData = new BitmapData(30, 30, false);
var rect:Rectangle = new Rectangle(0, 0, 30, 30);
var bytes:ByteArray = bmd.getPixels(rect);

// DLL
var myDLL:DLL = new DLL("TestDLL.dll");
parameterIndex = myDLL.addParameter("integer", "120");
parameterIndex = myDLL.addParameter("integer", "120");
myDLL.call("none", "Init");
myDLL.clear();

// Send data to DLL
parameterIndex = myDLL.addParameter("unsigned char*", bytes);
var result:String = myDLL.call("unsigned char*", "Process");
myDLL.clear();
myDLL.close();

Result
And we’re done! You would say… Unfortunately, some time profiling shows even the Init dummy function takes 30ms to execute. Sending a 120×120 bitmap takes 180ms. As 10FPS can be considered a minimum you want for analysis, we can conclude this is too slow to be useful.
The next and final attempt (because of time constraints) is to use a C++ server and use a Socket to communicate between Flash and the server.