Andy Li’s Blog - Flash

Head-tracking motion-parallax 3D in haXe/Flash

2012-03-04T22:19:00+08:00

Here above is my haXe implementation of head-tracking 3D, creating motion-parallax effect using Away3D 4. You can play with the live demo or grab the source.

This type of head tracking VR has been around for quite a long time. The first popular one should be Johnny Lee’s Wiimote hack, back in 2007, built in C# running as a desktop app. Two years later it appeared in the browser, a Flash version made by Mr. doob. But it was merely a hack and far from accurate comparing to Johnny Lee’s. Boffswana also created one in the same year (2009), and then an improved one with better head tracking algorithm. Sadly Boffswana hasn’t release the source.

Head tracking and camera movement is relatively simple, since there are so many libs for these. The tricky part is the perspective projection matrix. The projection point (the head), unlike most of the implementation assumption, isn’t always perpendicular to the projection surface (screen).

Simplest implementation would ignore the problem. It places a camera at the position of viewer’s head and render it to the screen, like what Mr. doob did. The problem is illustrated below.

When viewer’s head is perpendicular to the screen, everything is perfectly aligned:

However, when the viewer moves, for example to the left, there is misalignment of the rendering on screen and the “actual” position of the virtual object:

Knowing little about C#, it is hard for me to dig out Johnny Lee’s matrix code. I found the suitable code form the paper “Generalized Perspective Projection” from Robert Kooima. It is written in C++, but translating it to haXe isn’t hard, just remember matrix in OpenGL is column-major order but Flash’s is row-major order.

Head-detection is done using my fork of hxmarilena. My fork is simply some API changes and switching the XML parsing from flash.xml.XML to Xml so it may be used in C++ target in the future. In order to reduce jittering, I’ve included a simple optical flow tracking on the head. It is a block-matching process applied to four points on previous head detection result. The optical flow tracking result is merged to head-detection result by a ratio. It improved a bit, but I guess to address the problem it is better to port some better algorithms, like FaceTracker, in the future.

Finally, remember you can play with the live demo and it is open source.

How to know objects are really GC’ed in Flash(AS3)

2010-10-18T01:48:00+08:00

Memory leak is a bug that pretty hard to deal with. Usually people use some profilers to observer the memory usage while stressing the program, if there is no increase of memory usage after repeating calls of a function, we can conclude there is no memory leak in that function. It is of course fine to detect memory leak in this way, but it is always good to have a way that asserts your objects are really GC‘ed at some point.

The trick is to use a Dictionary with weak-reference keys and check its element count. It is already quite popular in the Java world but look like there aren’t many Flash devs talking about it.

A simple demo on how to write such test, source is available in wonderfl:

CanvasBlock: a GM script to block canvas tag (should we…or can we?)

2010-04-13T02:41:00+08:00

HTML5 is believed to be the Flash killer and will replace Flash in the future. Eventually those Flash-haters will become canvas-haters, so a canvas-blocking GreaseMonkey script will be useful (at least they think it will be).

I myself was making it for fun. I don’t think we should block any HTML5 tags anytime since it is a part of the web, just like I do not agree people blocking Flash. What is interesting is, seems like we cannot effectively block the canvas tag, which will be discussed later in this post.

Features

I made the script blocks not only canvas tag but also video tag and audio tag. It is because Flash is going to be replaced by all the three tags but not only canvas. Video and audio can be as annoying as canvas.

Like those Flash-blockers, the blocked contents are replaced by a line of text, “Click to show xxxx”. You may click on it to get back the blocked content.

CanvasBlock is tested on Firefox 3.6 on Mac, Firefox 3.5 on Ubuntu and Chrome on Ubuntu. Other platforms should works too.

Implementing the script

I used haXe instead of writing pure JS. No big reason here, just to see if it is possible. Turn out there is no big difference from using JS, you just need to copy all the meta data to the compiled JS file. And the events listener need to be assigned by addEventListener instead of something like onclick = function ....

Detecting HTML5 tags is easier than detecting Flash objects. Firstly, the tags are standardized meaning you can simply use getElementsByTagName. For Flash objects you need to check the params of the object and the embed tags, which is a bit troublesome. Secondly, currently no one (at least not much ppl) is creating canvas tags on the fly, unlike for Flash the standard way is to use swfobject. So I can detect the tags once the DOM is ready, no need to wait for the execution of the page’s JS.

Replacing the tags and saving them for later retrieval is easy too. Using normal JS method will work.

The tags are blocked but…

The script is functional and I went to do some more testing. Blocking video and audio seems ok, but there is some problem of blocking canvas.

Possibility of Freezing/Crashing

I went to ChromeExperiments and randomly picked one. You know what? The browser froze.

So, look like after I blocked the canvas, the JS on the page which is used to draw thing on the canvas originally, is still there trying to work on the canvas. And this become a big problem since I don’t know exactly which JS function is going to work on canvas, because different web page have different functions, so no way I can work on that. You can completely disable JS, but this is not a CanvasBlock should do.

Browser froze and an timeout message was shown after blocking canvas:

However, I have tested some more web page and they are ok with the blocking script. So maybe only some complex JS will have this problem. But anyway the possibility of freezing a browser is not a good thing.

No significant reduction on CPU usage

The remaining JS script problem reminded me to test on the CPU usage. And the result is, blocking canvas does not bring significant reduction on CPU usage. See the following screenshots, which is using another Chrome experiment:

Page opened, canvas blocked. However CPU usage still rise to a very high level:

Clicked to show back the canvas. CPU usage rose even higher, but not much difference:

So what’s that mean? Again if I want to block the real CPU hog, I should block the JS with the canvas. But again, stated above, it is not possible to block the right function without blocking all JS.

Conclusion

Blocking canvas can only introduce little benefit on CPU usage but give you possibility of freezing/crashing!

Let’s look at Flash again. Flash can be blocked easily because ActionScript is included in the swf. Removing the swf from the page automatically removes the associated scripts. Also, usually JS does not control a Flash object so it is pretty safe to kick Flash out without dealing with the remained JS. From this point of view, seems that Flash is more user friendly than canvas, and HTML5 ads will be more annoying than Flash ads…

If you want to try, you may install the script from CanvasBlock’s page on userscripts.org.

You’re also welcome to get the haXe source from CanvasBlock’s Github repo.

Colorblink - An AIR app that simulates how color blind people see Flash and images

2010-02-19T20:55:00+08:00

Accessibility is a an important part of both web and game design. And for web and game design, one of the popular tool is Flash. That means very often you need to ensure accessibility in Flash. However there are still not many tools available even for simple things like color blind simulation… so I write my own:)

>> download installer (.air)

It’s a very simple AIR app. You open it, drop a swf file on its window, select the color blind type and that’s it.

The inner of Colorblink is using a Pixel Bender filter, applying to the whole application. The algorithm is just a color transform matrix, found in a Java Color-Blindness Simulators. That simulator have more simulation config, which I used only the simplest one.

This is also my first time using a git repo. So go to have a look, see if you can fork it for more features.

One thing is, there is problem loading Flex applications into Colorblink… I don’t know how to read the loaded app’s default width and height and then resize it… So, if you want to test your Flex app, get the filter and apply it to your app manually (can’t be easier).
Now you can load Flex swf or even html file! But the app wouldn’t resize automatically since the size cannot be determined. If Colorblink does not work for you, you can still always get the filter and apply it to your app manually (can’t be easier).

Oh, yes, there is a simulation of what a dog sees… So, design some game for your dogs in your free time…

Chroma key and thresholding in Flash (Pixel Bender), revised

2009-12-10T18:12:00+08:00

Demo (Click to toggle filter)

source (Flex project including pbk)

I have been very busy since the last few weeks… so the chroma key filter was put aside for a long time until today :)

I have used conditional compile in the filter to avoid having if-else in the runtime, hopefully can increase performance… Anyone volunteer to measure it and post the different?

AS3/JS string concatenation methods performance test

2009-10-09T23:25:00+08:00

When writing BrainFlash, I was thinking if the string concatenations for program output will slow down the whole interpreter. That is because I was once hit that a year ago when dealing with XML.

I asked for better handling methods over StackOverflow. It is interesting that some of the answers shown that using “+=” is already the fastest. But I still want to know more about that, so I wrote my own simple testing program, which is below:

ActionScript version

source code

Methods used are:

str += concateString;
str = str.concat(concateString);
array.push(concateString); … str = array.join(“”);
vector.push(concateString); … str = vector.join(“”);
byteArray.writeUTFBytes(concateString); … str = byteArray.readUTFBytes(byteArray.length);
byteArray.writeMultiByte(concateString,”us-ascii”); … str = byteArray.readMultiByte(str.length,”us-ascii”);

The result shows that fastest method is using +=, but using Array/Vector is still very close to it. Using ByteArray is slow and with ASCII instead of UTF-8 is even slower…

And all the methods are performed reasonably fast, what is slow is when showing the resulting string on TextArea… It takes a few seconds! But when it is drawn, run it again and it will become normal speed, maybe there is some caching?

JavaScript version

I coded a JavaScript port too. See it here.

Methods used are:

str += concateString;
str = str.concat(concateString);
array.push(concateString); … str = array.join(“”);

Interesting enough, the result is very similar to AS3. It is the opposite of what we believe using the Array trick will let it performs faster. += is the fastest in most cases, if not, that’s not much difference.

I’ve only tested in IE8(Win), Firefox 3 (Win/Mac), Safari 4 (Win/Mac), Chrome 3 (Win). The really really really interesting part is in Chrome 3, using the concat method of String is x7000 SLOWER than the other two!! What are your results?

BrainFlash, the AS3 BrainFuck interpreter

2009-10-08T03:11:00+08:00

If you haven’t heard about Brainfuck yet, it is a famous esoteric programming language. If you want to know why it has such a name take a look at its hello-world program source code (from Wikipedia):

+++ +++ +++ +           initialize counter (cell #0) to 10
[                       use loop to set the next four cells to 70/100/30/10
    > +++ +++ +             add  7 to cell #1
    > +++ +++ +++ +         add 10 to cell #2
    > +++                   add  3 to cell #3
    > +                     add  1 to cell #4
    <<< < -                 decrement counter (cell #0)
]
>++ .                   print 'H'
>+.                     print 'e'
+++ +++ +.              print 'l'
.                       print 'l'
+++ .                   print 'o'
>++ .                   print ' '
<<+ +++ +++ +++ +++ ++. print 'W'
>.                      print 'o'
+++ .                   print 'r'
--- --- .               print 'l'
--- --- --.             print 'd'
>+.                     print '!'
>.                      print '\n'

the above program can be written as:

++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.>.

So now you should know how lovely the language is and why I need to write a interpreter in Flash: now you can manipulate a ByteArray with BrainFuck code!!! And ya, the name of my interpreter is even more friendly than the language’s name! :)

Here is the result demo (source code here), type in a program and its input then click run. The default program is to add up two single-digit integers. And the output is correct only if the sum is single-digit too. And again the code is from Wikipedia. You may find other BrainFuck source code to test from Panu Kalliokoski’s Brainfuck Archive.

The interpreter is written by only looking at its command description. There are some bugs as I know. For example when inputting multiple lines, the output may have wrong formatting. And it can not have dynamic input like having inputs in a infinite loop. Actually any infinite loop will freeze your browser.

I don’t think I will go to improve it anytime soon. So please, don’t use it in production (will anyone??).

Pixel Bender port of Scale2x and Scale3x

2009-09-28T03:35:00+08:00

When I first saw the Scale2x, scale3x AS3 I am impressed. The algorithm is from scale2x sourceforge project and the AS3 port is coded as a class that is very easy to reuse. Surely the AS3 port author nicoptere can make a PB port too as you may find out many PB cool stuff over his blog, but my pixel bending desire force me to do it myself :P

Here below the video shows my demo. I get a camera capture stream and used Posterizer which is downloaded from Pixel Bender Exchange to convert the image to 4-color-only, and than use the scale2x filter to enlarge the low resolution capture. The top-left corner thumbnail-like image is actually the original 1:1 input(160*120).

It is quite interesting to see for scale2x, PB’s version nearly doubles the performance of AS3’s. But for scale3x, there is not much different between PB and AS3. I think it is because scale3x has too many if-else which PB is not strong at.

BTW, I discovered a bug in PB while coding this. The bug make my code much longer as I work-a-round it… I reported it to the PB forum.

Demo app

Demo app’s Flex project with Pixel Bender source (The shorter but buggy versions are included too.)

Bicubic resampling by Pixel Bender

2009-08-25T07:51:00+08:00

There is bilinear resampling by Pixel Bender already, why not bicubic too?

With the very helpful Java implementation, I can get bicubic resampling running in Pixel Bender in hours.

Here is my result, along with the resampling results in PhotoShop as a comparison:

Source image (4x4), scaled by your browser):

Source image enlarged in PS (nearest neighbor):

Bicubic resampling by Pixel Bender:

There is some difference with PhotoShop’s versions (below), hope it’s not my mistake…

Bicubic resampling by PhotoShop:

Bicubic smoother by PhotoShop:

Bicubic sharper by PhotoShop:

My codes can be downloaded below:
Bicubic resampling sample program source (with PB source)

Update:
Notified by author of the Java version, the Java code was wrong and has been updated. Here below is my updated code:
Bicubic resampling sample program source (with PB source)

updated output of PB:

Update (2010-11-17):
Once again notified by Paul, the author of the Java version, there is something need to be updated. Here below is my updated code:
Bicubic resampling sample program source (with PB source)

updated output of PB:

Playing with chroma key and thresholding in Flash (with Pixel Bender)

2009-08-18T06:22:00+08:00

While traveling with my gf in Europe, I’m planning to develop a better technique to use color marker in FLARtoolkit (which I’ve tried before).

For the first step of doing so is to find a algorithm of matching colors. Using blend mode “difference” on the image is not flexible and may not fit all situations. I decided to calculate the difference by comparing hue, saturation and brightness of the colors and then threshold the image.

This is very similar to (if not exactly) doing chroma key. Maybe there is some awesome beautiful chroma key algorithm but I can’t find a fast and flexible enough to do so. If you know one, please tell me. :)

One problem I faced is choosing between color space models. RGB surely wouldn’t be the choice since it do not give hue value directly. There are HSV and HSL I found to be quite suitable. The two models share the same algorithm to calculate hue but have different ones for saturation and brightness. After reading the algorithms, my decision is… try both. :P

Another problem is how to preform thresholding. First method is after calculating the pixel difference, add the difference in hue, saturation and brightness altogether with weightings and do a one-time threshold to the grayscale image. Second method is calculate the difference and give threshold limits to the three channels (H,S,V or H,S,L), only the pixels pass all the three thresholds will be white. First method should be more easily to incorporate with other filters developed for AR (like the very hot adaptive thresholding), but the second method can be more precise. Finally I tried both too…

Here is the result. While I have set some default values to the filters, be sure to play around with the values when you change the target color. There wouldn’t be a set of values that fits all methods and situations. :)

I’ll try integrate it with FLARtoolkit and release the source when it is ready, be sure to check back soon. :)

Update:
Source and new demo is here!