Code Coverage of Jasmine Tests using Istanbul and Karma

For modern web application development, having dozens of unit tests is not enough anymore. The actual code coverage of those tests would reveal if the application is thoroughly stressed or not. For tests written using the famous Jasmine test library, an easy way to have the coverage report is via Istanbul and Karma.

For this example, let’s assume that we have a simple library sqrt.js which contains an alternative implementation of Math.sqrt. Note also how it will throw an exception instead of returning NaN for an invalid input.

var My = {
  sqrt: function(x) {
    if (x < ) throw new Error("sqrt can't work on negative number");
      return Math.exp(Math.log(x)/2);
  }
};

Using Jasmine placed under test/lib/jasmine-1.3.1, we can craft a test runner that includes the following spec:

describe("sqrt", function() {
  it("should compute the square root of 4 as 2", function() {
    expect(My.sqrt(4)).toEqual(2);
  });
});

Opening the spec runner in a web browser will give the expected outcome:

So far so good. Now let’s see how the code coverage of our test setup can be measured.

The first order of business is to install Karma. If you are not familiar with Karma, it is basically a test runner which can launch and connect to a specific set of web browsers, run your tests, and then gather the report. Using Node.js, what we need to do is:

npm install karma karma-coverage

Before launching Karma, we need to specify its configuration. It could be as simple as the following my.conf.js (most entries are self-explained). Note that the tests are executed using PhantomJS for simplicity, it is however quite trivial to add other web browsers such as Chrome and Firefox.

module.exports = function(config) {
  config.set({
    basePath: '',
    frameworks: ['jasmine'],
    files: [
      '*.js',
      'test/spec/*.js'
    ],
    browsers: ['PhantomJS'],
    singleRun: true,
    reporters: ['progress', 'coverage'],
    preprocessors: { '*.js': ['coverage'] }
  });
};

Running the tests, as well as performing code coverage at the same time, can be triggered via:

node_modules/.bin/karma start my.conf.js

which will dump the output like:

INFO [karma]: Karma v0.10.2 server started at http://localhost:9876/
INFO [launcher]: Starting browser PhantomJS
INFO [PhantomJS 1.9.2 (Linux)]: Connected on socket N9nDnhJ0Np92NTSPGx-X
PhantomJS 1.9.2 (Linux): Executed 1 of 1 SUCCESS (0.029 secs / 0.003 secs)

As expected (from the previous manual invocation of the spec runner), the test passed just fine. However, the most particular interesting piece here is the code coverage report, it is stored (in the default location) under the subdirectory coverage. Open the report in your favorite browser and there you’ll find the coverage analysis report.

Behind the scene, Karma is using Istanbul, a comprehensive JavaScript code coverage tool (read also my previous blog post on JavaScript Code Coverage with Istanbul). Istanbul parses the source file, in this example sqrt.js, using Esprima and then adds some extra instrumentation which will be used to gather the execution statistics. The above report that you see is one of the possible outputs, Istanbul can also generate LCOV report which is suitable for many continuous integration systems (Jenkins, TeamCity, etc). An extensive analysis of the coverage data should also prevent any future coverage regression, check out my other post Hard Thresholds on JavaScript Code Coverage.

One important thing about code coverage is branch coverage. If you pay attention carefully, our test above is still not exercising the situation where the input to My.sqrt is negative. There is a big “I” marking in the third-line of the code, this is Istanbul telling us that the if branch is not taken at all (for the else branch, it will be an “E” marker). Once this missing branch is noticed, improving the situation is as easy as adding one more test to the spec:

it("should throw an exception if given a negative number", function() {
  expect(function(){ My.sqrt(-1); }).
    toThrow(new Error("sqrt can't work on negative number"));
});

Once the test is executed again, the code coverage report looks way better and everyone is happy.

If you have some difficulties following the above step-by-step instructions, take a look at a Git repository I have prepared: github.com/ariya/coverage-jasmine-istanbul-karma. Feel free to play with it and customize it to suit your workflow!