It is a truth universally acknowledged, that a single function critical to the success of the application, must be in want of a unit test. A practical way to prevent the lack of a unit test is to ensure that the overall code coverage does not regress. Fortunately, for applications written in JavaScript, there are a few code coverage services which can help with the task.
Thanks to a variety of language tooling available these days, it is not hard to measure and track code coverage of a JavaScript application. My go-to solution is involving Istanbul as the coverage tool, combined with either Karma or Venus.js as the test runner. This setup works with various popular unit test libraries out there. If you are new to this, I recommend checking out my past blog posts on this subject:
- Karma and Istanbul with Mocha
- Karma and Istanbul with Jasmine
- Karma and Istanbul with QUnit
- Istanbul and Venus.js (Mocha, Jasmine, QUnit)
And yet, the work does not stop there. Would it be fantastic if the code coverage report becomes another feedback information for a contributor? Is it possible to track down every single pull request and check if the changes associated with that pull request would regress the coverage? The answer is yes. The key to that is utilizing a hosted code coverage service. There are many out there and in this post I will cover (pun intended) my current favorite, Codecov.io.
Thank for a set of its rich features, integrating Codecov.io to your open-source project is very easy. For a start, you do not need to create a dedicated account as you can just authenticate using Github. Furthermore, Codecov.io has a built-in support for Github (as well as other hosted Git such as Bitbucket), choosing a project to be added to your dashboard is trivial.
Keep in mind that Codecov.io displays the coverage information of your project. Your build process still need to produces that coverage information. Also, it is assumed that you have a continuous integration system that runs the build process every time there is a new check-in or when someone has a feature branch in a pull request. For many FOSS project, Travis CI is the most common solution although there are a few other hosted CI services out there.
To following a long, check out this simple repository that I have created: github.com/ariya/coverage-mocha-istanbul-karma. This repo contains a simple JavaScript project along with its equally simple test suite designed for Mocha. The tests will be executed by Karma.
To start using Codecov.io, first we need to enable the coverage information in Cobertura format. I have played with different coverage formats and I discovered that Cobertura is the most suitable (your mileage may vary and things can change from time to time). If you use Istanbul directly, you can use its report command to generate the coverage information in the right format (refer to the documentation for more details). With our setup, I modified a section in the Karma configuration file, karma.conf.js, from:
coverageReporter: {
dir : 'coverage/',
reporters: [
{ type: 'html', subdir: 'html' },
{ type: 'lcov', subdir: 'lcov' },
]
}
to:
coverageReporter: {
dir : 'coverage/',
reporters: [
{ type: 'html', subdir: 'html' },
{ type: 'lcovonly', subdir: 'lcov' },
{ type: 'cobertura', subdir: 'cobertura' }
]
}
This ensures that Karma tells Istanbul to produce another coverage information, in addition to the default lcov, in the format that we want, Cobertura. You can test this, simply execute npm test and after a while, you will spot the file coverage/cobertura/cobertura-coverage.xml that contains the coverage information. This is what we need to send to Codecov.io. There are multiple ways to do that, the easiest is to use codecov.io package. You can use this package by running:
npm install --save-dev codecov.io.
In this example, package.json is modified to look like this:
"scripts": {
"test": "grunt karma:test",
"ci": "npm test && codecov < coverage/cobertura/cobertura-coverage.xml"
}
Thus, everytime you invoke npm run ci on your Travis CI job, the tests will be executed and the coverage information will be sent to Codecov.io.
To setup the dashboard, login to Codecov.io and add the repository as a new project. Codecov.io maintains a nice mapping of project URL. For example, the coverage dashboard for this example repo github.com/ariya/coverage-mocha-istanbul-karma is codecov.io/github/ariya/coverage-mocha-istanbul-karma. The next time you kick a build on the project, the dashboard will display the coverage information as sent from the build process.
If that works flawlessly, now you want to enable its pull request integration. Go to the project page and choose Integration and Setup, Pull Request Comment. Now you can determine various ways Codecov.io will comment on every pull request. For a start, you may want to enable Header and Compare Diff.
In the example repo, I have created a pull request, github.com/ariya/coverage-mocha-istanbul-karma/pull/3, that demonstrated a coverage regression. In that pull request, there is a commit that aims to optimize the code but that optimization does not include an additional unit test. This triggers the following response from Codecov.io, a feedback that is rather obvious:
With the build process that produces the coverage information, combined with a service such as Codecov.io, it is easy to keep untested code away from your project!