The above text cloud is created using Fotowall.
Starting from the recently released PhantomJS version 1.5, also known as Ghost Flower, a minimalistic version of Qt library is bundled into its source tree. This blog post shows some history and reasoning behind it. If you just want to use PhantomJS, nothing changes for you. However, if you are curious about the machinery behind its build, then continue reading.
One thing I have wanted to experiment for a while is the rapid release cycle. For PhantomJS, it means four releases per year, each is aligned with the season and the release date is matched to the equinox or solstice. PhantomJS uses Qt (with its excellent WebKit integration) as the basis. Since Qt releases are not as often as PhantomJS, this caused some complications.
The first and foremost problem was the long wait until a new Qt release is available. In particular, every Qt release carries a fresh and updated WebKit module which usually solves a lot of compatibilities issues. However, longer release period means that a Qt bug will stay in few PhantomJS versions until a fix is available. In addition, an extremely long period of Qt 4.8 development did not help PhantomJS marched forward as fast I would like it to be. Qt 4.8.0 was out mid December last year, 15 months after Qt 4.7.0. As a comparison, the time lapse between Qt 4.6.0 to 4.7.0 was just over 9 months.
Another problem is the build issue. As much as I’d like Qt to be ubiquitious, unfortunately that’s not the case. On Windows, the situation is not that bad because it’s not very often someone is compiling from source on this platform. On Linux, there were often crash reports because people are using PhantomJS against outdated Qt 4.6 and no official fresh Qt package available on that distribution. That’s discouraging but not much we can do about it.
On Mac OS X, it was pretty hopeless since everyone tends to use MacPorts or Homebrew to install PhantomJS. They think PhantomJS is just a fun little tool. Unless Qt is already built by MacPorts/Homebrew, suddenly that innocent one line command keeps the machine busy for hours. Everyone complained about this all the time that I decided to put a notice about this situation everywhere: in the README, build instructions wiki, and even a blog post about it. Amazingly some people still prefer to “shoot first, ask later”, once a while the predictable ‘PhantomJS takes ages to build’-type of rant shows up somewhere.
This build issue is temporarily solved by having a special build script. The script automatically downloads latest Qt source code from Nokia web site, builds Qt locally, and compiles PhantomJS with it. As a bonus, the entire build on Mac OS X is faster than Homebrew/MacPorts because only the necessary parts of Qt were compiled (no need for demos, samples, documentation, various tools). Another advantage is that we can apply patches which improve various things. Here is an incomplete list of shortcomings of using plain vanilla or outdated Qt which other QtWebKit-based tools may suffer but we don’t (thanks to the custom build and patching):
- no cross-domain XHR (issue 28)
- console.log does not work with multiple arguments (issue 36)
- no file upload (issue 307)
- lack of error reporting and stack trace (issue 166)
- potential memory corruption (issue 274)
- no functioning remote debugging (issue 6)
- crash or unexpected behavior with empty URL scheme (issue 365)
On Mac OS X, the build script even creates a static version of PhantomJS which is quite compact, around 10 MB (after extra compression). This Mac OS X static build is rather popular because (1) you can freely move the executable to any machine, any place you want (2) you don’t need to install Qt or other software, it is self-contained and guaranteed to work on a fresh install of Snow Leopard or later version. The size also helps. Guess which one is more convincing: (a) tell someone to download and unzip 10 MB file or (b) ask him/her to grab 178 MB binary install of Qt? Only in a very rare case choice (b) will win.
The final issue, most probably the most important one, is regarding the bridging mechanism between the native and the JavaScript world. The existing bridge provided by Qt is very good for the use cases of hybrid apps, e.g. web-based UI running a C++ logic. However, in the case of PhantomJS the flow actually goes a little bit different: JavaScript world, native (Qt) space, then goes back to JavaScript. This makes the situation very delicate to handle. It causes all sorts of “unexpected” behavior, from lack of argument binding to lossy null. In fact, we are already exhausting and abusing the bridging to a certain extent where we can’t go further.
I secretly wished that the bridging will be different in the upcoming Qt 5. However, seems that as of now such a bridging is not high in the priority list. Beside, Qt 5 is still months away from the release, maybe even more if you want to wait for all initial bugs to be squashed (this is a 5.0.0 release after all). And speaking about Qt 5, it seems that for all intents and purposes, Qt 4.8 will be the last in the 4.x series and we’re stucked with it for at least a few months in the future.
Combining the fact that we already custom-build Qt 4.8 and no chance to jumpstart to Qt 5 any time soon, finally a wild but pragmatic decision was made: we import Qt code into the source tree. What is being imported is only the necessary portion needed for PhantomJS (we do not care for e.g. declarative module). The obvious drawback is that the repository is getting larger although it is not actually that bad thanks to the efficient Git storage mechanism.
Few more doors are available to us after this import, among others making pure headless (no Xlib, no Xvfb) as the canonical build for Linux, getting bleeding edge WebKit, immediate fixes to major deficiencies (e.g. error handling with stack trace), as well improvements to the bridging situation. We’ll find out if this experiment is worth the effort in the long run.
Enjoy Ghost Flower!