Andy Li’s Blog - experiment

Head-tracking motion-parallax 3D in haXe/Flash

2012-03-04T22:19:00+08:00

Here above is my haXe implementation of head-tracking 3D, creating motion-parallax effect using Away3D 4. You can play with the live demo or grab the source.

This type of head tracking VR has been around for quite a long time. The first popular one should be Johnny Lee’s Wiimote hack, back in 2007, built in C# running as a desktop app. Two years later it appeared in the browser, a Flash version made by Mr. doob. But it was merely a hack and far from accurate comparing to Johnny Lee’s. Boffswana also created one in the same year (2009), and then an improved one with better head tracking algorithm. Sadly Boffswana hasn’t release the source.

Head tracking and camera movement is relatively simple, since there are so many libs for these. The tricky part is the perspective projection matrix. The projection point (the head), unlike most of the implementation assumption, isn’t always perpendicular to the projection surface (screen).

Simplest implementation would ignore the problem. It places a camera at the position of viewer’s head and render it to the screen, like what Mr. doob did. The problem is illustrated below.

When viewer’s head is perpendicular to the screen, everything is perfectly aligned:

However, when the viewer moves, for example to the left, there is misalignment of the rendering on screen and the “actual” position of the virtual object:

Knowing little about C#, it is hard for me to dig out Johnny Lee’s matrix code. I found the suitable code form the paper “Generalized Perspective Projection” from Robert Kooima. It is written in C++, but translating it to haXe isn’t hard, just remember matrix in OpenGL is column-major order but Flash’s is row-major order.

Head-detection is done using my fork of hxmarilena. My fork is simply some API changes and switching the XML parsing from flash.xml.XML to Xml so it may be used in C++ target in the future. In order to reduce jittering, I’ve included a simple optical flow tracking on the head. It is a block-matching process applied to four points on previous head detection result. The optical flow tracking result is merged to head-detection result by a ratio. It improved a bit, but I guess to address the problem it is better to port some better algorithms, like FaceTracker, in the future.

Finally, remember you can play with the live demo and it is open source.

Chroma key and thresholding in Flash (Pixel Bender), revised

2009-12-10T18:12:00+08:00

Demo (Click to toggle filter)

source (Flex project including pbk)

I have been very busy since the last few weeks… so the chroma key filter was put aside for a long time until today :)

I have used conditional compile in the filter to avoid having if-else in the runtime, hopefully can increase performance… Anyone volunteer to measure it and post the different?

CSS-only experiments

2009-11-24T16:35:00+08:00

In the last week I have tried to make some CSS-only experiments. Originally it’s just an idea to play some effects with CSS animation which now available in Webkit. But at the end I am obsessed and making games with CSS… Yup, that’s embedding game logic in the place which designed to be presentation layer only… :P

Video first, description follows:

Mouse shadow

Open experiment

It works on Webkit browsers only (Chrome, Safari).

What I want to create is just some old-school shadow, fading etc. Turn out using a deeply-nested div for the effect is quite stylish. The divs are assigned to different classes based on their nested level to improve performance because div > div > div > ... is just too slow…

Gear

Open experiment

Again, works on Webkit browsers only (Chrome, Safari).

I find the deeply-nested div has quite a potential to make some more things, so here is another product of it. Swapping the CSS transition with CSS transform achieved something very similar to fractal.

One little defect is I used percentage for the size of divs and the rounding number cause some gaps between divs… It is more noticeable when using resolution 3.

Pixel art

Open experiment

This one works on all major browsers (IE 6/7/8, Firefox, Safari, Chrome) :)

This is a drawing board that you can click to draw pixel.

I remembered there are hackers tried to get users’ browsing history by using color-change of visited link. It impressed me to store valued in the browsing history. Originally I used links with hash values (i.e.. <a href="#123"></a>) and it works in all the browsers… except IE. IE will turn the color when you click on it but it does not store it in the history. So I switched to use iframe, that is setting the links to open in the iframe and serve a dummy web page with random URL…

Click Challenge

Open experiment

This one is Webkit browsers only (Chrome, Safari).

This is the latest experiment I made, which is really a playable game. CSS keyframe animation is used to simulate a timer. Lots of identical checkboxes to act as an area to let user clicks on. They are removed when clicked on it, using :checked in CSS. Finally a CSS counter is used to label the checkboxes (actually a ordered list will be fine).

Update

2009-11-26: Gear is now on Chrome Experiments! Being the first CSS-only experiment over there :)

AS3/JS string concatenation methods performance test

2009-10-09T23:25:00+08:00

When writing BrainFlash, I was thinking if the string concatenations for program output will slow down the whole interpreter. That is because I was once hit that a year ago when dealing with XML.

I asked for better handling methods over StackOverflow. It is interesting that some of the answers shown that using “+=” is already the fastest. But I still want to know more about that, so I wrote my own simple testing program, which is below:

ActionScript version

source code

Methods used are:

str += concateString;
str = str.concat(concateString);
array.push(concateString); … str = array.join(“”);
vector.push(concateString); … str = vector.join(“”);
byteArray.writeUTFBytes(concateString); … str = byteArray.readUTFBytes(byteArray.length);
byteArray.writeMultiByte(concateString,”us-ascii”); … str = byteArray.readMultiByte(str.length,”us-ascii”);

The result shows that fastest method is using +=, but using Array/Vector is still very close to it. Using ByteArray is slow and with ASCII instead of UTF-8 is even slower…

And all the methods are performed reasonably fast, what is slow is when showing the resulting string on TextArea… It takes a few seconds! But when it is drawn, run it again and it will become normal speed, maybe there is some caching?

JavaScript version

I coded a JavaScript port too. See it here.

Methods used are:

str += concateString;
str = str.concat(concateString);
array.push(concateString); … str = array.join(“”);

Interesting enough, the result is very similar to AS3. It is the opposite of what we believe using the Array trick will let it performs faster. += is the fastest in most cases, if not, that’s not much difference.

I’ve only tested in IE8(Win), Firefox 3 (Win/Mac), Safari 4 (Win/Mac), Chrome 3 (Win). The really really really interesting part is in Chrome 3, using the concat method of String is x7000 SLOWER than the other two!! What are your results?

BrainFlash, the AS3 BrainFuck interpreter

2009-10-08T03:11:00+08:00

If you haven’t heard about Brainfuck yet, it is a famous esoteric programming language. If you want to know why it has such a name take a look at its hello-world program source code (from Wikipedia):

+++ +++ +++ +           initialize counter (cell #0) to 10
[                       use loop to set the next four cells to 70/100/30/10
    > +++ +++ +             add  7 to cell #1
    > +++ +++ +++ +         add 10 to cell #2
    > +++                   add  3 to cell #3
    > +                     add  1 to cell #4
    <<< < -                 decrement counter (cell #0)
]
>++ .                   print 'H'
>+.                     print 'e'
+++ +++ +.              print 'l'
.                       print 'l'
+++ .                   print 'o'
>++ .                   print ' '
<<+ +++ +++ +++ +++ ++. print 'W'
>.                      print 'o'
+++ .                   print 'r'
--- --- .               print 'l'
--- --- --.             print 'd'
>+.                     print '!'
>.                      print '\n'

the above program can be written as:

++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.>.

So now you should know how lovely the language is and why I need to write a interpreter in Flash: now you can manipulate a ByteArray with BrainFuck code!!! And ya, the name of my interpreter is even more friendly than the language’s name! :)

Here is the result demo (source code here), type in a program and its input then click run. The default program is to add up two single-digit integers. And the output is correct only if the sum is single-digit too. And again the code is from Wikipedia. You may find other BrainFuck source code to test from Panu Kalliokoski’s Brainfuck Archive.

The interpreter is written by only looking at its command description. There are some bugs as I know. For example when inputting multiple lines, the output may have wrong formatting. And it can not have dynamic input like having inputs in a infinite loop. Actually any infinite loop will freeze your browser.

I don’t think I will go to improve it anytime soon. So please, don’t use it in production (will anyone??).

Playing with chroma key and thresholding in Flash (with Pixel Bender)

2009-08-18T06:22:00+08:00

While traveling with my gf in Europe, I’m planning to develop a better technique to use color marker in FLARtoolkit (which I’ve tried before).

For the first step of doing so is to find a algorithm of matching colors. Using blend mode “difference” on the image is not flexible and may not fit all situations. I decided to calculate the difference by comparing hue, saturation and brightness of the colors and then threshold the image.

This is very similar to (if not exactly) doing chroma key. Maybe there is some awesome beautiful chroma key algorithm but I can’t find a fast and flexible enough to do so. If you know one, please tell me. :)

One problem I faced is choosing between color space models. RGB surely wouldn’t be the choice since it do not give hue value directly. There are HSV and HSL I found to be quite suitable. The two models share the same algorithm to calculate hue but have different ones for saturation and brightness. After reading the algorithms, my decision is… try both. :P

Another problem is how to preform thresholding. First method is after calculating the pixel difference, add the difference in hue, saturation and brightness altogether with weightings and do a one-time threshold to the grayscale image. Second method is calculate the difference and give threshold limits to the three channels (H,S,V or H,S,L), only the pixels pass all the three thresholds will be white. First method should be more easily to incorporate with other filters developed for AR (like the very hot adaptive thresholding), but the second method can be more precise. Finally I tried both too…

Here is the result. While I have set some default values to the filters, be sure to play around with the values when you change the target color. There wouldn’t be a set of values that fits all methods and situations. :)

I’ll try integrate it with FLARtoolkit and release the source when it is ready, be sure to check back soon. :)

Update:
Source and new demo is here!

Simulating atomic model of metal by Box2DFlash

2009-07-14T01:49:00+08:00

Impressed by the video here, I tried to simulate the atomic model of metal in Flash using Box2DFlash.

What is done in the program is just put a lot of spheres in the area randomly, and then let the physics engine to solve all the collisions. When all the atoms stop moving, a single still of the metal structure will be formed.

The atoms are only drawn on the screen when it is sleeping (i.e.. no movement). It can save some time from not rendering them in each frame.

caution: The browser will freeze for a few seconds after “start” is pressed. And it takes several minutes to an hour to finish (when sleep count == atom count).

source

PV3D + Async-threading = Async Render Engine

2009-06-07T03:28:00+08:00

Recently I need to write a 3D tool to help making a installation artwork. I know I should use Maya, Processing or anything like that, but in order to get the least risk, I used my most experienced language, AS3.

Since I need to render A LOT of stuff (\~400,000 objects), instantly I faced the performance issue. As you can imagine, even if I could generate all the objects in 15 seconds, the rendering process must excess the Flash Player’s execution time limit.

So I try to make my own async render engine with the help of async-threading. I extended BasicRenderEngine to AsyncBasicRenderEngine. The class even has a showBusyCursor property that let you optionally show the busy cursor on rendering.

But note that this is only a proof-of-concept and actually has little help of preventing freeze-screen. It is because I only distribute the individual 3D object’s rendering to frames but leaving the sorting, filtering, post-process code unchanged. Also, the RenderStatistics returned by renderScene/renderLayers will contain only wrong data until the RENDER_DONE event is dispatched.

One last thing, Flash’s drawing algorithm seems affecting the async rendering performance too. So I do not add the viewport to stage. Instead, once RENDER_DONE, I draw the viewport on a Bitmap on stage.

source

Vector Sound Wave Morphing

2009-05-20T06:53:00+08:00

This my my final project for the course “Interactivity I”. For this final project, my aim is to explore the vector nature of sound wave while making a tool to let people create their own waveform which can then be saved for other use.

The waveform in the program consists of 10 control points. The first point and the last point are fixed at zero so that the wave can be looped smoothly to form a sound wave. The controls points can be adjusted by dragging or by using keyboard. The points are assigned with keys labelled near them, which are number keys (+shift to move points up instead of down). I recommend using the keyboard as you may adjust multiple points at the same time (which feels like using multitouch surface 8-) ).

Degrafa is used to construct the wave form. I choose to use cubic bezier curve as it is easier for me to implement the algorithm (with the still-in-beta AdvancedCubicBezier which available via SVN).

The 3D cylinder is made by Flash Player 10’s 3D API. It looks quite nice as similar to landscapes.

Sound is synthesized dynamically in real-time (with some delay… okay? ;-) ). If you hear some “clicks” or “pops”, try to export the sound first by clicking on the export button at the bottom-right corner (it will be in wav format, encoded by the WavEncoder from popforge) .

If you hear nothing, you may try to increase the buffer size or use the export function.

source

Chocolate tree…

2008-07-16T03:14:00+08:00

Click below to draw (and redraw).

Originally, I attempted to make a normal tree trunk, but the result make me think of chocolate…

And the performance seems hard to be improve since I’m animating bitmaps…