Autodetect JavaScript TDD/BDD Library

A unique feature of Venus.js, a JavaScript test runner from LinkedIn, is that the test configuration can be in the form of source annotation. This is useful, e.g. to choose which test library (Mocha, Jasmine, QUnit) should be used to execute the tests. Now, wouldn’t it be fantastic if the test runner can deduce the said library automatically?

During a chat over coffee, I proposed that kind of best-effort detection idea to my fellow Shape Security engineer, Seth (who is involved with Venus.js). Obviously, this type of detection might not be 100% accurate. However, I postulate that it should be good enough in most cases, particularly in the case where the annotation is not present.

As a proof of concept, I have implemented detect-testlib.js (see the Git repository at gitlab.com/ariya/detect-testlib). It uses Esprima to parse the test code, collect the important verbs (more about this later), and then use the gathered information to decide whether the test is written in Jasmine or qUnit, or if it is completely unknown. To follow along, clone the repo, run npm install first, and the try the following:

node detect-testlib.js test/jquery-attributes.js

That jquery-attributes.js is taken from the actual jQuery unit tests. As expected, detect-testlib will confidently say that those tests are using qUnit. For another attempt, check test/yeoman-env.js (again, taken from Jasmine-based Yeoman unit tests).

How does the detection work? There are many different ways to implement it. For this proof-of-concept, I opt for something simple. The tool will scan the names of the function (aka, the verbs) used in the top-level and the secondary level. In other words, given the following code:

describe("sqrt", function() {
    it('computes the square root of 9 as 3', function() {
        expect(My.sqrt(9)).toBeCloseTo(3, 10);
    });
}

then it will collect describe in the top-level group and it in the test-level group. After a while, we will have these two arrays populated (without any duplicates). For example, running it on test/jquery-attributes.js will give the arrays as:

{ topLevel:
   ['module', 'test' ],
  testLevel: 
   [ 'expect',
     'deepEqual',
     'equal',
     'ok',
     'strictEqual',
     'testVal',
     'testAddClass',
     'testRemoveClass',
     'testToggleClass' ] }

Once the array is obtained, the special decide() function will use a simple heuristic to figure out the library being used in the test code. As an illustration, for the above example, it will conclude that the test is using Jasmine (based on the existence of describe and it). For a different test code like the following:

test("sqrt", function() {
  equal(My.sqrt(4), "2", "Check for square root of 4" );
});

then decode() will go for QUnit as its solution.

Obviously, in some corner cases this simple decision rule will fail. However, that is expected since it is rather minimalistic. You are encouraged to develop a more sophisticated classification implementation in case a more faithful decision is needed for your application.

How about Mocha? Since Mocha can support both TDD and BDD style, the decision factor is more complicated. A possible solution is to detect the assertion mechanism and cross-correlate it with the typical pattern of assertions (expect.js, Chai, etc) used with Mocha.

Have fun with autodetection!