Andy Li's Blog

Here above is my haXe implementation of head-tracking 3D, creating motion-parallax effect using Away3D 4. You can play with the live demo or grab the source.

This type of head tracking VR has been around for quite a long time. The first popular one should be Johnny Lee’s Wiimote hack, back in 2007,  built in C# running as a desktop app. Two years later it appeared in the browser, a Flash version made by Mr. doob. But it was merely a hack and far from accurate comparing to Johnny Lee’s. Boffswana also created one in the same year (2009), and then an improved one with better head tracking algorithm. Sadly Boffswana hasn’t release the source.

Head tracking and camera movement is relatively simple, since there are so many libs for these. The tricky part is the perspective projection matrix. The projection point (the head), unlike most of the implementation assumption, isn’t always perpendicular to the projection surface (screen).

Simplest implementation would ignore the problem. It places a camera at the position of viewer’s head and render it to the screen, like what Mr. doob did. The problem is illustrated below.

When viewer’s head is perpendicular to the screen, everything is perfectly aligned:

However, when the viewer moves, for example to the left, there is misalignment of the rendering on screen and the “actual” position of the virtual object:

Knowing little about C#, it is hard for me to dig out Johnny Lee’s matrix code. I found the suitable code form the paper “Generalized Perspective Projection” from Robert Kooima. It is written in C++, but translating it to haXe isn’t hard, just remember matrix in OpenGL is column-major order but Flash’s is row-major order.

Head-detection is done using my fork of hxmarilena. My fork is simply some API changes and switching the XML parsing from flash.xml.XML to Xml so it may be used in C++ target in the future. In order to reduce jittering, I’ve included a simple optical flow tracking on the head. It is a block-matching process applied to four points on previous head detection result. The optical flow tracking result is merged to head-detection result by a ratio. It improved a bit, but I guess to address the problem it is better to port some better algorithms, like FaceTracker, in the future.

Finally, remember you can play with the live demo and it is open source.

Memory leak is a bug that pretty hard to deal with. Usually people use some profilers to observer the memory usage while stressing the program, if there is no increase of memory usage after repeating calls of a function, we can conclude there is no memory leak in that function. It is of course fine to detect memory leak in this way, but it is always good to have a way that asserts your objects are really GC‘ed at some point.

The trick is to use a Dictionary with weak-reference keys and check its element count. It is already quite popular in the Java world but look like there aren’t many Flash devs talking about it.

A simple demo on how to write such test, source is available in wonderfl:

Memory leak unit test - wonderfl build flash online

HTML5 is believed to be the Flash killer and will replace Flash in the future. Eventually those Flash-haters will become canvas-haters, so a canvas-blocking GreaseMonkey script will be useful (at least they think it will be).

I myself was making it for fun. I don’t think we should block any HTML5 tags anytime since it is a part of the web, just like I do not agree people blocking Flash. What is interesting is, seems like we cannot effectively block the canvas tag, which will be discussed later in this post.

Features

I made the script blocks not only canvas tag but also video tag and audio tag. It is because Flash is going to be replaced by all the three tags but not only canvas. Video and audio can be as annoying as canvas.

Like those Flash-blockers, the blocked contents are replaced by a line of text, “Click to show xxxx”. You may click on it to get back the blocked content.

CanvasBlock is tested on FireFox 3.6 on Mac, FireFox 3.5 on Ubuntu and Chrome on Ubuntu. Other platforms should works too.

Implementing the script

I used haXe instead of writing pure JS. No big reason here, just to see if it is possible. Turn out there is no big difference from using JS, you just need to copy all the meta data to the compiled JS file. And the events listener need to be assigned by “addEventListener” instead of something like “onclick = function …”.

Detecting HTML5 tags is easier than detecting Flash objects. Firstly, the tags are standardized meaning you can simply use “getElementsByTagName”. For Flash objects you need to check the params of the “object” and the “embed” tags, which is a bit troublesome. Secondly, currently no one (at least not much ppl) is creating canvas tags on the fly, unlike for Flash the standard way is to use swfobject. So I can detect the tags once the DOM is ready, no need to wait for the execution of the page’s JS.

Replacing the tags and saving them for later retrieval is easy too. Using normal JS method will work.

The tags are blocked but…

The script is functional and I went to do some more testing. Blocking video and audio seems ok, but there is some problem of blocking canvas.

Possibility of Freezing/Crashing

I went to ChromeExperiments and randomly picked one. You know what? The browser freezed.

So, look like after I blocked the canvas, the JS on the page which is used to draw thing on the canvas originally, is still there trying to work on the canvas. And this become a big problem since I don’t know exactly which JS function is going to work on canvas, because different web page have different functions, so no way I can work on that. You can completely disable JS, but this is not a CanvasBlock should do.

Browser freezed and an timeout message was shown after blocking canvas.

However, I have tested some more web page and they are ok with the blocking script. So maybe only some complex JS will have this problem. But anyway the possibility of freezing a browser is not a good thing.

No significant reduction on CPU usage

The remaining JS script  problem reminded me to test on the CPU usage. And the result is, blocking canvas does not bring significant reduction on CPU usage. See the following screenshots, which is using another Chrome experiment:

Page opened, canvas blocked. However CPU usage still rise to a very high level.

Clicked to show back the canvas. CPU usage rose even higher, but not much difference.

So what’s that mean? Again if I want to block the real CPU hog, I should block the JS with the canvas. But again, stated above, it is not possible to block the right function without blocking all JS.

Conclusion

Blocking canvas can only introduce little benefit on CPU usage but give you possibility of freezing/crashing!

Let’s look at Flash again. Flash can be blocked easily because ActionScript is included in the swf. Removing the swf from the page automatically removes the associated scripts. Also, usually JS does not control a Flash object so it is pretty safe to kick Flash out without dealing with the remained JS. From this point of view, seems that Flash is more user friendly than canvas, and HTML5 ads will be more annoying than Flash ads…

If you want to try, you may install the script from CanvasBlock’s page on userscripts.org.

You’re also welcome to get the haXe source from CanvasBlock’s github repo.