FLARtoolkit trick: use a colored marker

May 23rd, 2009  |  Published in Uncategorized  |  26 Comments

YouTube Preview Image

Reasons why should we use colored marker instead of standard black and white only marker

Using a black and white only marker is boring

There are more and more AR applications in the world, but seems everybody use only the black square marker. May be the public is not yet bored by the black square yet, but if can use other color for the square, we can integrate the marker into real-life objects more beautifully and more flexible.

Increase marker detection performance

The internal algorithm of AR marker detection has a characteristic that, the less pixels’ color is the same of the marker, faster the detection can be done. When we use black square marker, the detector will waste a lot of time to process non-marker pixels as you can imagine that it is common to have black objects in real-life (especially we the Chinese people have black hair). So if we target less common color, the detector will be able to ignore most of the non-marker area.

How to achieve this in FLARtoolkit

We will create a custom detector class since we need to work at lower level of the FLARtoolkit. If you have not look into the internal classes of FLARtoolkit (because the FLARmanager is so much easier to work with ;-) ), there are two detectors: FLARSingleMarkerDetector and FLARMultiMarkerDetector. As their names suggest, one is for single marker detection and the other one can detect multiple markers. For simplicity, I will go through only the single detection for a red marker (if you want me to cover the other ones too, leave comment).

First we just copy all the codes inside FLARSingleMarkerDetector to a new class, I named it FLARSingleRedMarkerDetector.

To detect markers, we use the detectMarkerLite method, which is:

public function detectMarkerLite(i_raster:IFLARRgbRaster, i_threshold:int):Boolean

IFLARRgbRaster wraps a BitmapData, we can get it by:

var srcImg:BitmapData = (i_raster as FLARRgbRaster_BitmapData).bitmapData;

Originally, the method uses a threshold filter to extract black areas and give the result to other classes to process. As you might think of, we can simply swap the filter code. Originally the black areas will be turned to white pixels and others will be turned black and be ignored. We now want to detect red markers, so just try to turn everything black except the red regions to white. You may use ColorTransform if you want, but I am now obsessed with PixelBender. My version of kernel is:

kernel extractRedARMarker
<   namespace : "net.onthewings.filters";
    vendor : "Andy Li";
    version : 1;
    description : "used for FLARtoolkit, pre-proccess for red marker.";
>
{
    input image4 src;
    output pixel4 dst;
 
    parameter float threshold<
        minValue: 0.0;
        maxValue: 1.0;
        defaultValue: 0.4;
        description:"decrease to increase likelihood of marker detection.";
    >;
 
    void
    evaluatePixel()
    {
        float4 p = sampleNearest(src,outCoord());
        float sum = p.r+p.g+p.b;
        float val = p.r - (p.g + p.b)*0.5;
 
        if (val+(1.0-(sum/3.0))*0.1 <= threshold) {
            val = 0.0;
        } else {
            val = 1.0;
        }
        dst = float4 (val,val,val,1);
    }
}

Note that I let the dark red pass the threshold easier since I find it is common the web cam image is too dark.

To use the finished detector class:

tempFLARRgbRaster_BitmapData.bitmapData.draw(videoDisplay);
if (detector.detectMarkerLite(tempFLARRgbRaster_BitmapData,170)){
	detector.getTransformMatrix(lastDetectionResult);
	//FLARPVGeomUtils is from FLARmanager
	overlay3dObj.transform = FLARPVGeomUtils.convertFLARMatrixToPVMatrix(lastDetectionResult);
	overlay3dObj.visible = true;
} else {
	overlay3dObj.visible = false;
}
render();

Now you get all the concepts. So let’s skip all the boring stuff and see it in action. Or get the finished codes.

Happy AR!


Update: Here I have a PB kernel that works with any color. Try it!

Tags: , , ,
  • http://transmote.com/flar ericsoco

    hi andy. one thing worth mentioning is that a red marker does not show up as one color in a camera — it’s highly dependent on the lighting in the area the camera is viewing. depending on the lighting (which cannot be controlled for web applications), red can appear as anywhere from red to shades of green or blue or black.

    the most consistent thing for a computer vision application operating under variable lighting conditions to detect is brightness. that’s why FLARToolkit uses black.

    you do have a good point though, that since FLARToolkit first looks for black areas in the image, images with a lot of dark areas will cause slowdown in the marker detection. (this is especially true for someone with as much black hair as you! ;) and the PixelBender implementation is nice! have you benchmarked it to see how it compares with a ColorMatrixFilter?

  • http://www.onthewings.net/ Andy

    I see your point. That’s why we need to adjust the threshold carefully.
    Maybe we even need a better threshold mechanism or some auto adjustment like color balance before marker detection. I think that will be a interesting research topic :)

    I don’t have any benchmark yet since I think I will make a plug-in library for color marker detection some time in the future (may be, can’t promise). Don’t expect a dramatic difference between PB and ColorMatrixFilter for this case though.

  • http://www.pierbover.com Pier

    Hi Andy!

    I’m not such a great developper… but seeing your code it seems you try to detect red from and RGB data.

    Wouldn’t it be better to try to detect brighntess only in the R channel?

    Red in the R channel is white… that would mean processing only 8bit pixels instead of 24 bit pixels.

    Or maybe I’m just wrong and you’re already doing that… :P

    Anyway… good work!

  • http://andyli.pip.verisignlabs.com/ Andy

    White in red channel surely can be red(#FF0000), but it can be yellow(#FFFF00), magenta(#FF00FF) or even white(#FFFFFF)!
    In that case, the filtered area will be much larger, meaning the save in processing 8bit instead of 24bit wouldn’t result in net performance gain.

  • http://www.pierbover.com Pier

    You’re so right…

  • http://makc3d.wordpress.com/ makc

    just three points:

    1 usually when people think of “color support” in flart, they mean marker made of different colors, or markers that differ from each other by color combinations. this is not totally covered by swapping the filter.

    2 using svn version you can swap filters without changing flart code.

    3 your flickr humanizer displays unavailable images :(

  • http://andyli.pip.verisignlabs.com/ Andy

    @makc

    Yup, you’re right. I just pushed a little bit on the flexibility and it’s still pretty limited at this point. I think more complicated and colorful markers can be done but the FLARtoolkit is still in rapid development, like what you’ve told me, that now we can set custom filter which is not possible at the time I write the post.
    Anyway, I will update the codes soon. Thanks for the info :)

    I’ve just swapped the captcha plug-in and reported the bug to the author. Hope that will be fixed soon :P

  • http://www.vamoss.com.br Vamoss

    Hi Andy,

    I readed you text about the red market.
    I dont know nothing about PixelBender, e a few things about Flartoolkit.

    Could you help me in a addaption of your code?
    In your text, is not clear to me where I import your class “FLARSingleRedMarkerDetector.as”, in the Main.as?
    And how can I create a Yellow Marker?

    So much thanks for your time!
    @Vamoss

    —————————————————————

    I runned the FLARSingleRedMarkerDetector.as with my Black Marker normally.
    How can I run my Yellow Marker version, I have to generate it before? Where?
    And how can I addapt your Red class to detect Yellow colors?

    Thanks Andy!!
    @Vamoss

  • http://andyli.pip.verisignlabs.com/ Andy

    @Vamoss

    For the file location, just unzip the whole thing and put it under the source folder.

    It’s a bit hard to detect yellow marker since it is not one of the color channel. The get it done, you only need to change the pbk, recompile it into pbj. Sure, you can rename everything red to yellow.
    The change in pbk, it can be from “float val = p.r – (p.g + p.b)*0.5;” to “float val = (p.r + p.g)*0.5 – p.b;”. What this line of code do is letting val be 1 if the color is matched, less if it is not. FYI, red is #FF0000 and yellow is #FFFF00.
    I have not tested the yellow code, and it is not that accurate I can say. Feel free to make some change to improve it.

    Actually I have tried to write a class for detecting marker with any color, but I have been busying these week and it has been put aside. I think I can finish it in a week but I don’t know can you wait that long :P

  • kaka

    Hi Andy,
    thanx for this great work,
    I’ve to do a school work based on Flartoolkit and I’m facing some difficulties.
    Can Flartoolkit can recognize a circle or a specific picture in red color?
    Thanx in advance
    kaka

  • http://andyli.pip.verisignlabs.com/ Andy

    @kaka No, FLARtoolkit only works with square markers. You may put the picture inside the marker, but the picture cannot be too complex. Try the Marker Generator at http://flash.tarotaro.org/blog/2008/12/14/artoolkit-marker-generator-online-released/

  • kaka

    thanx Andy

  • Pingback: Playing with chroma key and thresholding in Flash (with Pixel Bender) | Andy Li's Blog

  • Jeff Lung

    Hi Andy, I am also from Hong Kong and really appreciate you share your flartoolkit knowledge with us. Can you tell us how to combine your color marker code with flash cs4? Wether for flarmanager or Saqoosha’s flartoolkit starter kit is ok, thanks in advance ;)

  • http://andyli.pip.verisignlabs.com/ Andy

    Sorry that I haven’t used the Flash IDE for quite a long time. But I don’t think there is much different from using it in pure AS3 project. Once you manage how to use FLARtoolkit(and the helper libraries) in the Flash IDE, integration of color marker should be easy and just the same as stated in the post.

  • kawayiboy

    Hi, Andy, I’m a Chinese who have just read your blog.However, I encounter a problem, when i use the class FLARSingleRedMarkerDetector, I always cannot import the flash.display.Shader and the flash.filters.ShaderFilter, and I know I should use the Pixel Blender, I have learned a little about the new feature of the Adobe CS4, and in some small cases, it work. but the problems appear sometimes. as you use the [Embed("extractRedARMarker.pbj", mimeType="application/octet-stream")] structure, the flash CS4 let me browse the flex sdk. I also encounter the problem in Flex. I use the Flex builder 3 and Flex sdk 3.5, I cannot find the solutions online, Please help me!Thank you

  • http://andyli.pip.verisignlabs.com/ Andy

    @kawayiboy Are you targeting Flash Player 10? It can be set in the publish setting in the Flash IDE and project setting in Flex Builder.

  • kawayiboy

    I’m very grateful for your help, and I finally solve the original problem. But I get a bad result in detecting the red marker. Can you send me a complete example flash project that I can detect a red marker? my email:tenglu2@qq.com, Thank you!

  • http://andyli.pip.verisignlabs.com/ Andy

    @kawayiboy

    Sorry that I’ve not worked on FLARtoolkit for quite a long time and don’t have time to code an example for you.

    But if you find the detecting result bad, try playing with the threshold value.

    And you may try replacing the filter with the chroma key PB kernel:
    http://blog.onthewings.net/2009/12/10/chroma-key-and-thresholding-in-flash-pixel-bender-revised/

  • kawayiboy

    ok,3q

  • Aamna

    hey great work. I am trying to handel Multi Markers using multimarkerdetector class. Please help me in that.

    thanks in advance

  • Hi Andy

    Hi Andy.
    I like your work for Red marker. I am currently making e-learning project for kids using AR. However, I am requiring 26 different markers for each of the albhabet of English language. If I use too many markers, flar toolkit may sometimes wrongly recognize the markers. Can you plz provide me some suggestion so that I can use 26 markers in my project. I hope that you will help me out.
    Thank you

  • tana

    can a marker perform task if let say a video file loaded in the red marker you holding and then if you show another black n white marker to indicate to play the video? and it plays the video

  • Pingback: Technology and tools « AReality_final_project_2011

  • Elamiz

    hi andy,i am a HK student who start using Flash Builder for trying to do multi marker of AR,(i don’t know programming too much)….can u teach me how to perform A marker can show A picture,B marker can show B picture by using FLARMultiMarkerDetector?
    {i hv try to make the demo…i hope u may understand wt i am talking…(http://www.4shared.com/video/BTzKcMTT/Screen_Record_2013-03-10_at_21.html)……as i can see many people doing multi-mark in many way, most of it only changing colour of a cube….i understand how to change the pattern of the marker only …i hv try to follow some foreigner youtube tutorials….quite difficult for understanding…that why i don’t know wt should do…can u teach me to do this in a simple way?}

  • Pingback: FLARtoolkit技巧:使用彩色marker | fingergame