Andy Li’s Blog - Pixel Bender

Colorblink - An AIR app that simulates how color blind people see Flash and images

2010-02-19T20:55:00+08:00

Accessibility is a an important part of both web and game design. And for web and game design, one of the popular tool is Flash. That means very often you need to ensure accessibility in Flash. However there are still not many tools available even for simple things like color blind simulation… so I write my own:)

>> download installer (.air)

It’s a very simple AIR app. You open it, drop a swf file on its window, select the color blind type and that’s it.

The inner of Colorblink is using a Pixel Bender filter, applying to the whole application. The algorithm is just a color transform matrix, found in a Java Color-Blindness Simulators. That simulator have more simulation config, which I used only the simplest one.

This is also my first time using a git repo. So go to have a look, see if you can fork it for more features.

One thing is, there is problem loading Flex applications into Colorblink… I don’t know how to read the loaded app’s default width and height and then resize it… So, if you want to test your Flex app, get the filter and apply it to your app manually (can’t be easier).
Now you can load Flex swf or even html file! But the app wouldn’t resize automatically since the size cannot be determined. If Colorblink does not work for you, you can still always get the filter and apply it to your app manually (can’t be easier).

Oh, yes, there is a simulation of what a dog sees… So, design some game for your dogs in your free time…

Chroma key and thresholding in Flash (Pixel Bender), revised

2009-12-10T18:12:00+08:00

Demo (Click to toggle filter)

source (Flex project including pbk)

I have been very busy since the last few weeks… so the chroma key filter was put aside for a long time until today :)

I have used conditional compile in the filter to avoid having if-else in the runtime, hopefully can increase performance… Anyone volunteer to measure it and post the different?

Pixel Bender port of Scale2x and Scale3x

2009-09-28T03:35:00+08:00

When I first saw the Scale2x, scale3x AS3 I am impressed. The algorithm is from scale2x sourceforge project and the AS3 port is coded as a class that is very easy to reuse. Surely the AS3 port author nicoptere can make a PB port too as you may find out many PB cool stuff over his blog, but my pixel bending desire force me to do it myself :P

Here below the video shows my demo. I get a camera capture stream and used Posterizer which is downloaded from Pixel Bender Exchange to convert the image to 4-color-only, and than use the scale2x filter to enlarge the low resolution capture. The top-left corner thumbnail-like image is actually the original 1:1 input(160*120).

It is quite interesting to see for scale2x, PB’s version nearly doubles the performance of AS3’s. But for scale3x, there is not much different between PB and AS3. I think it is because scale3x has too many if-else which PB is not strong at.

BTW, I discovered a bug in PB while coding this. The bug make my code much longer as I work-a-round it… I reported it to the PB forum.

Demo app

Demo app’s Flex project with Pixel Bender source (The shorter but buggy versions are included too.)

Bicubic resampling by Pixel Bender

2009-08-25T07:51:00+08:00

There is bilinear resampling by Pixel Bender already, why not bicubic too?

With the very helpful Java implementation, I can get bicubic resampling running in Pixel Bender in hours.

Here is my result, along with the resampling results in PhotoShop as a comparison:

Source image (4x4), scaled by your browser):

Source image enlarged in PS (nearest neighbor):

Bicubic resampling by Pixel Bender:

There is some difference with PhotoShop’s versions (below), hope it’s not my mistake…

Bicubic resampling by PhotoShop:

Bicubic smoother by PhotoShop:

Bicubic sharper by PhotoShop:

My codes can be downloaded below:
Bicubic resampling sample program source (with PB source)

Update:
Notified by author of the Java version, the Java code was wrong and has been updated. Here below is my updated code:
Bicubic resampling sample program source (with PB source)

updated output of PB:

Update (2010-11-17):
Once again notified by Paul, the author of the Java version, there is something need to be updated. Here below is my updated code:
Bicubic resampling sample program source (with PB source)

updated output of PB:

Playing with chroma key and thresholding in Flash (with Pixel Bender)

2009-08-18T06:22:00+08:00

While traveling with my gf in Europe, I’m planning to develop a better technique to use color marker in FLARtoolkit (which I’ve tried before).

For the first step of doing so is to find a algorithm of matching colors. Using blend mode “difference” on the image is not flexible and may not fit all situations. I decided to calculate the difference by comparing hue, saturation and brightness of the colors and then threshold the image.

This is very similar to (if not exactly) doing chroma key. Maybe there is some awesome beautiful chroma key algorithm but I can’t find a fast and flexible enough to do so. If you know one, please tell me. :)

One problem I faced is choosing between color space models. RGB surely wouldn’t be the choice since it do not give hue value directly. There are HSV and HSL I found to be quite suitable. The two models share the same algorithm to calculate hue but have different ones for saturation and brightness. After reading the algorithms, my decision is… try both. :P

Another problem is how to preform thresholding. First method is after calculating the pixel difference, add the difference in hue, saturation and brightness altogether with weightings and do a one-time threshold to the grayscale image. Second method is calculate the difference and give threshold limits to the three channels (H,S,V or H,S,L), only the pixels pass all the three thresholds will be white. First method should be more easily to incorporate with other filters developed for AR (like the very hot adaptive thresholding), but the second method can be more precise. Finally I tried both too…

Here is the result. While I have set some default values to the filters, be sure to play around with the values when you change the target color. There wouldn’t be a set of values that fits all methods and situations. :)

I’ll try integrate it with FLARtoolkit and release the source when it is ready, be sure to check back soon. :)

Update:
Source and new demo is here!

FLARtoolkit trick: use a colored marker

2009-05-23T10:40:00+08:00

Reasons why should we use colored marker instead of standard black and white only marker

Using a black and white only marker is boring

There are more and more AR applications in the world, but seems everybody use only the black square marker. May be the public is not yet bored by the black square yet, but if can use other color for the square, we can integrate the marker into real-life objects more beautifully and more flexible.

Increase marker detection performance

The internal algorithm of AR marker detection has a characteristic that, the less pixels’ color is the same of the marker, faster the detection can be done. When we use black square marker, the detector will waste a lot of time to process non-marker pixels as you can imagine that it is common to have black objects in real-life (especially we the Chinese people have black hair). So if we target less common color, the detector will be able to ignore most of the non-marker area.

How to achieve this in FLARtoolkit

We will create a custom detector class since we need to work at lower level of the FLARtoolkit. If you have not look into the internal classes of FLARtoolkit (because the FLARmanager is so much easier to work with ;-) ), there are two detectors: FLARSingleMarkerDetector and FLARMultiMarkerDetector. As their names suggest, one is for single marker detection and the other one can detect multiple markers. For simplicity, I will go through only the single detection for a red marker (if you want me to cover the other ones too, leave comment).

First we just copy all the codes inside FLARSingleMarkerDetector to a new class, I named it FLARSingleRedMarkerDetector.

To detect markers, we use the detectMarkerLite method, which is:

public function detectMarkerLite(i_raster:IFLARRgbRaster, i_threshold:int):Boolean

IFLARRgbRaster wraps a BitmapData, we can get it by:

var srcImg:BitmapData = (i_raster as FLARRgbRaster_BitmapData).bitmapData;

Originally, the method uses a threshold filter to extract black areas and give the result to other classes to process. As you might think of, we can simply swap the filter code. Originally the black areas will be turned to white pixels and others will be turned black and be ignored. We now want to detect red markers, so just try to turn everything black except the red regions to white. You may use ColorTransform if you want, but I am now obsessed with PixelBender. My version of kernel is:

kernel extractRedARMarker
<   namespace : "net.onthewings.filters";
    vendor : "Andy Li";
    version : 1;
    description : "used for FLARtoolkit, pre-proccess for red marker.";
>
{
    input image4 src;
    output pixel4 dst;

    parameter float threshold<
        minValue: 0.0;
        maxValue: 1.0;
        defaultValue: 0.4;
        description:"decrease to increase likelihood of marker detection.";
    >;

    void
    evaluatePixel()
    {
        float4 p = sampleNearest(src,outCoord());
        float sum = p.r+p.g+p.b;
        float val = p.r - (p.g + p.b)*0.5;

        if (val+(1.0-(sum/3.0))*0.1 <= threshold) {
            val = 0.0;
        } else {
            val = 1.0;
        }
        dst = float4 (val,val,val,1);
    }
}

Note that I let the dark red pass the threshold easier since I find it is common the web cam image is too dark.

To use the finished detector class:

tempFLARRgbRaster_BitmapData.bitmapData.draw(videoDisplay);
if (detector.detectMarkerLite(tempFLARRgbRaster_BitmapData,170)){
    detector.getTransformMatrix(lastDetectionResult);
    //FLARPVGeomUtils is from FLARmanager
    overlay3dObj.transform = FLARPVGeomUtils.convertFLARMatrixToPVMatrix(lastDetectionResult);
    overlay3dObj.visible = true;
} else {
    overlay3dObj.visible = false;
}
render();

Now you get all the concepts. So let’s skip all the boring stuff and see it in action. Or get the finished codes.

Happy AR!

Update: Here I have a PB kernel that works with any color. Try it!

The Bilateral Blur Filter for Flash is now on Pixel Bender Exchange (in staff picks!)

2009-04-24T00:19:00+08:00

I’ve put my bilateral blur filter on Pixel Bender Exchange and it is selected as staff picks!

Go and download it from the page: Bilateral Blur Filter for Flash on Pixel Bender Exchange

P.S.. That one is optimized, processing time originally is 2 - 3s , it’s now 0.6 - 0.7s (for a 320*240 image).

Bilateral Blur by Pixel Bender

2009-04-14T22:24:00+08:00

I’ve been working on implementing a bilateral blur for Flash. Currently the functionality is ok but the performance is quite poor (on my Intel MacBook, processing a 320x240 image takes ~2s). I’ve even tried to replace the Gaussian weighting functions with linear functions but it does not help a lot but leads to worse quality. The problem should be the sampling size of each pixel which I hard coded to be 9x9 square. Reducing the sampling area would help but again reducing quality.

It is frustrating that Pixel Bender in Flash do not have loops. Even accessing vector/matrix elements with variable index is not possible.

Anyway, here below is a demo.

Before applying filter:

After applying filter:

Hope I can further optimize it later. (or better Adobe gives us GPU mode :lol: )

UPDATED: See this blog post