« About
We Feel Fine & Universe »

Face detection using Flash and C++

As you could read in a previous blog, I’m trying to create a small webcam game in Flash that supports ‘physical input’ using a webcam. The frames are processed by a C++ server that uses existing computer vision libraries. This way it should be fairly easy to build in cool features like face detection.

Now I’ve finally managed to get some things working on my Flash / C++ computer vision project. Zoran (my project advisor at the University of Amsterdam) helped me a lot on the C++ computer vision part. Using the OpenCV library we’ve managed to put face detection in a Flash application! Input from the webcam is sent via a Socket to a light weight server running localhost. When a face is detected, a message is sent back to Flash.

The boundaries of the face in the image are also known. I just have to think of some kind of protocol to send and retrieve different kinds (and sizes) of data.

The result of the image detection from OpenCV (using the red color channel here):
CV face detection

This is how I display the webcam (AS3 code):

// Show the webcam
var camera:Camera = Camera.getCamera();
camera.setMode(WEBCAM_WIDTH, WEBCAM_HEIGHT, 31);
videoDisplay = new Video(WEBCAM_WIDTH,WEBCAM_HEIGHT);
videoDisplay.attachCamera(camera);
this.addChild(videoDisplay);

Making the connection:

private function openConnection(evt:Event):void
{
	// Connect to the external server
	s = new Socket();
	s.addEventListener(Event.CONNECT, initializeImageFeed);
	s.addEventListener(ProgressEvent.SOCKET_DATA, receiveData);
	s.addEventListener(IOErrorEvent.IO_ERROR,ioErrorHandler);

	s.connect("localhost", SERVER_PORT);
}

The timer that sends the feed every x milliseconds (200ms now for me):

/**
* Takes the data of the webcam and sends it to the server
*/
private function sendData(evt:Event):void
{
	var webcamSendWidth:int = 160;
	var webcamSendHeight:int = 120;
	var scale:Number = webcamSendWidth / WEBCAM_WIDTH;

	var scaleMatrix:Matrix = new Matrix();
	scaleMatrix.scale(scale,scale);

	// Get the bitmap from the webcam
	var now:BitmapData = new BitmapData(webcamSendWidth, webcamSendHeight);
	now.draw(videoDisplay, scaleMatrix);
	var rect:Rectangle = new Rectangle(0, 0, webcamSendWidth, webcamSendHeight);
	var ba:ByteArray = now.getPixels(rect);

	s.writeBytes(ba);
	s.flush();
}

The BitmapData that is being sent consists of 4 channels in this order: Alpha, Red, Green, Blue. That is 4 bytes per pixel, which makes 160×120 x 4 = 76800 bytes (75KB) for the complete image. If you want to do this over the Internet, you could compress the Bitmap. Because I’m running both the Flash application and the server on the same computer, bandwidth usage isn’t an issue and compressing and decompressing would probably take longer than just sending it.

This entry was posted on Sunday, June 24th, 2007 at 1:09 am and is filed under Actionscript 3.0. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

16 Responses to “Face detection using Flash and C++”

  1. theTudor Says:
    August 21st, 2007 at 10:21 am

    Hi,

    I ran across the same problem, i need to use a .lib and it’s only available in C++

    I have to port a small game to AS3, and the physics libraries available for AS3 turned out to not be as full featured as the one (www.ode.org) i had been using. Both APE and Fisix engine do not have static friction, and miss other things i would need.

    Could you tell me what path you took for setting up your sockets (listener?) in C++ ? Did you build it from scratch ? Did you use a sockets lib ? What is the path you advise ?

    (i need to send requests to create objects from the flex/as3 side –
    and from c++/ode side i have send the coordinates for all objects realtime)

  2. Marijn Says:
    September 5th, 2007 at 1:01 am

    @theTudor, sorry for my late reply, I was on vacation starting the 21th.

    I used a small self-written C++ socket library / app by a friend. That was after trying several existing libraries that kept giving strange errors, incomplete data packets, etc. I’m quite sure I was just using them incorrectly as I’m not really familiar with C++.

    If you’re interested / still need it, I can send it. Contact me at marijn @ [the URL of this website].

    I’m hope a socket will be fast enough for real-time physics calculation. It will highly on what you want to calculate and how often. How much data do you need to send?

  3. wiseobject Says:
    September 21st, 2007 at 3:09 am

    Hi Marijn,

    I have the same problem too, need to implement openCV in Flash, and i think your solution is the one i need. Can i ask for your sockets codes?

    kindly send it to wiseobject (at) yahoo.com

    Thanks in advance.

  4. Chris Says:
    October 26th, 2007 at 1:12 pm

    Hi Marijn,

    I would like to ask you if you can send me your socket library cause I have a similar problem.

    My email is demochri@hotmail.com .

    Thank you in advance

  5. theTudor Says:
    October 29th, 2007 at 8:23 pm

    thanks Marijn, i’m just using APE for now, but i’m gonna port it to Motor when it comes out( in December i hope) – (it’s supposed to have static friction)

    but thank you anyway.

    and good luck with your projects :)

  6. RaDmAc Says:
    November 25th, 2007 at 5:29 pm

    Hi Marijn,
    I am doing a project on multimodal biometric system in C++/OpenCV and I was looking for a suitable GUI. Flash is one of the easiest to play with and I wolud like to use it but have no idea how to connect those two. Can you send yours libs to rad.mac(at)wp.pl?

  7. Trigun82 Says:
    February 16th, 2008 at 1:31 am

    Hi marijn,
    whe just talk about that! Please if u got it give it to me.
    I need to do your same work. Thanks!

  8. Yaz Okulu Says:
    March 24th, 2008 at 4:18 am

    does anyone knows if there is any other information about this subject in other languages?

  9. Marijn Says:
    March 25th, 2008 at 12:39 pm

    Not that I know of, sorry. Also check out my latest blog about this subject, with more code and information, http://www.marijnspeelman.nl.

  10. Jaruwan Says:
    April 1st, 2008 at 7:05 pm

    Hey Marijn,
    Are you running DLL files at server side or client side? Can I have your email address please?
    Thanks

  11. Jaruwan Says:
    April 3rd, 2008 at 5:14 pm

    My email is jmesit@cs.ucf.edu. Thank you.

  12. JY Says:
    April 3rd, 2008 at 8:11 pm

    Try OSC as a protocol. It was originally used for audio (alternative to MIDI) but is used in multi-touch applications to detect blobs. Using OSC (TUIO), you’ll be compatible with a bunch of other projects that are using it.

    The main issue with it is that is UDP and not TCP so Flash can not accept it natively. You’ll need to change it into TCP and use a socket connection to send the data to Flash. The trade-off is more compatibility with other projects that are using OSC/TUIO but more work to get the data to something that Flash can read.

    Good luck with everything.

  13. Charles Says:
    May 7th, 2008 at 9:06 pm

    Have any of you ever heard of Red5? I have been recently using flash/flex with red5 to do electronic control. You don’t have to mess with sockets when using Red5 as it uses RTMP protocol. You can simply use a Netconnection call in order to call a server side function from a client side flash application. However, the native language Red5 is written in is Java.

    Although it takes additional overhead, isn’t it relatively trivial to pass data to and execute a c++ app from java? Thats the route I am going to try and take.

  14. Andy Says:
    November 18th, 2008 at 1:48 pm

    You may try this now:
    http://labs.adobe.com/technologies/alchemy/

  15. Junior B. Says:
    August 31st, 2009 at 2:08 am

    I liked reading your blog…keep up the good work.

  16. chew Says:
    March 4th, 2010 at 5:26 am

    Hi Marijn,

    I would like to ask you if you can send me your socket library.

    My email is ccyhaku@hotmail.com

    thanks. :)

Leave a Reply