Andy Li’s Blog - 3D

Head-tracking motion-parallax 3D in haXe/Flash

2012-03-04T22:19:00+08:00

Here above is my haXe implementation of head-tracking 3D, creating motion-parallax effect using Away3D 4. You can play with the live demo or grab the source.

This type of head tracking VR has been around for quite a long time. The first popular one should be Johnny Lee’s Wiimote hack, back in 2007, built in C# running as a desktop app. Two years later it appeared in the browser, a Flash version made by Mr. doob. But it was merely a hack and far from accurate comparing to Johnny Lee’s. Boffswana also created one in the same year (2009), and then an improved one with better head tracking algorithm. Sadly Boffswana hasn’t release the source.

Head tracking and camera movement is relatively simple, since there are so many libs for these. The tricky part is the perspective projection matrix. The projection point (the head), unlike most of the implementation assumption, isn’t always perpendicular to the projection surface (screen).

Simplest implementation would ignore the problem. It places a camera at the position of viewer’s head and render it to the screen, like what Mr. doob did. The problem is illustrated below.

When viewer’s head is perpendicular to the screen, everything is perfectly aligned:

However, when the viewer moves, for example to the left, there is misalignment of the rendering on screen and the “actual” position of the virtual object:

Knowing little about C#, it is hard for me to dig out Johnny Lee’s matrix code. I found the suitable code form the paper “Generalized Perspective Projection” from Robert Kooima. It is written in C++, but translating it to haXe isn’t hard, just remember matrix in OpenGL is column-major order but Flash’s is row-major order.

Head-detection is done using my fork of hxmarilena. My fork is simply some API changes and switching the XML parsing from flash.xml.XML to Xml so it may be used in C++ target in the future. In order to reduce jittering, I’ve included a simple optical flow tracking on the head. It is a block-matching process applied to four points on previous head detection result. The optical flow tracking result is merged to head-detection result by a ratio. It improved a bit, but I guess to address the problem it is better to port some better algorithms, like FaceTracker, in the future.

Finally, remember you can play with the live demo and it is open source.

Sandy3D, C++, Haxe

2010-03-18T07:39:00+08:00

I’ve been learning Haxe since the last few months. Beside the reason of many Flash guys moved/moving to Haxe because of better performance of compiled swf, I was tempted to try using its C++ target, which runs even faster(at least in theory, in most parts). Its language feature is also very nice, which is quite similar to AS3 but more powerful in many aspects. Check out its feature page if you want to know more.

And because of I need to write some 3D stuff in a project, I tried finding a Haxe 3D engine that can compiled to C++, but who knows, there wasn’t one. Actually the 3D engines in Haxe are all quite young. There are 3 I found, all targeting only Flash: Sandy3D, originally written in AS, has switched its trunk to Haxe; Away3D, again originally written in AS, now has a Haxe branch inside its SVN in active development; Haxe3D, developed by Nicolas(the creator of Haxe), written in Haxe from the very beginning, seems to be very light-weight but I can’t found much info about it.

So I decide to work on one of them to enable a Haxe C++ 3D engine. Because Sandy3D should be the most complete and stable one(which actually stopped active development…), I chose it.

In the development of it, some issues are related to nme/neash, which are the re-invented cross-platform Haxe version of the Flash API. With the help from Hugh and Niel, project owner of neash and developer of Sandy3D, most of the problems are solved. However as I can see, re-inventing the Flash API is a hard job as consideration of other target is also needed, e.g.. JS/Canvas. I really hope Adobe can help this.

Currently some very basic demos of Sandy can be compiled to C++ and it shows that, in OpenGL rendering mode, the performance can goes over 300% of the Flash(Haxe, not AS3) version. However in normal rendering mode, the frame rate may actually be reduced if the window size is large. Anyway, I think the performance is quite depend on hardware/OS, also the 3D scene you’re showing.

Thanks for the Sandy3D team, they added me as a committer in their official Google project. And I have just committed the changes to the trunk :) There should be a new release soon if everything is fine.

A note to the ones can’t wait for a release:

Install hxcpp, nme, neash from haxelib.
Replace contents in neash’s neash/display/IBitmapDrawable.hx to a simple typedef IBitmapDrawable = nme.display.IBitmapDrawable;.

To make mouse event work, in neash’s neash/Lib.hx, the code:

// Process pending timers ...

neash.Timer.CheckTimers();

// Send frame-enter event

var event = new Event( Event.ENTER_FRAME );

mStage.Broadcast(event);

should be moved to the end of the do-while loop, just before mStage.RenderAll();

For mouse event in OpenGL mode to work, additional fix which required a recompile of nme is needed. I haven’t tried yet. Now just ignore this…
You should be able to use Sandy3D in C++ without the above patches for the next release of nme/neash. But for now, please do it.

BTW, I am also ~~porting~~ ported casalib, which I often use in AS3 development, to Haxe, see casahx.

Vector Sound Wave Morphing

2009-05-20T06:53:00+08:00

This my my final project for the course “Interactivity I”. For this final project, my aim is to explore the vector nature of sound wave while making a tool to let people create their own waveform which can then be saved for other use.

The waveform in the program consists of 10 control points. The first point and the last point are fixed at zero so that the wave can be looped smoothly to form a sound wave. The controls points can be adjusted by dragging or by using keyboard. The points are assigned with keys labelled near them, which are number keys (+shift to move points up instead of down). I recommend using the keyboard as you may adjust multiple points at the same time (which feels like using multitouch surface 8-) ).

Degrafa is used to construct the wave form. I choose to use cubic bezier curve as it is easier for me to implement the algorithm (with the still-in-beta AdvancedCubicBezier which available via SVN).

The 3D cylinder is made by Flash Player 10’s 3D API. It looks quite nice as similar to landscapes.

Sound is synthesized dynamically in real-time (with some delay… okay? ;-) ). If you hear some “clicks” or “pops”, try to export the sound first by clicking on the export button at the bottom-right corner (it will be in wav format, encoded by the WavEncoder from popforge) .

If you hear nothing, you may try to increase the buffer size or use the export function.

source

Data structure assignment - Coin Collecting Game

2008-11-03T04:56:00+08:00

It is just a simple game (If it is really a game), but I was having fun with Flex and PaperVision3D.

Crazy 3D modelling technology … Video Trace

2008-06-22T21:40:00+08:00

VideoTrace: Rapid interactive scene modelling from video

This is a really crazy modeling technology…
Use a video clip that has some view points (in different angles) to an object, trace the object with a few clicks in several frames, and you get the 3D model (mapped with auto-generated bitmaps from the clip).