If your library is being used by many other applications, it is often a good idea to run their test suites. Since those downstream applications may break because of your library, this helps catching those cases before you officially release a new public version of the library.
This is the extra cautious step that we have been practicing for the Esprima project. Since Esprima is the dependency of a few dozens projects out there and its npm package is downloaded at a rate of 10 millions/month, we want to be aware if anything will fall apart before we update it. Thus, those projects become a sort of animal sentinel (“canary in a coal mine”).
Every time there is a new pull request, there is a number of downstream projects (e.g. Istanbul, JSCS, jsfmt, etc) which will be tested against the modified Esprima in that pull request. The logic is rather simple and the implementation itself weighs just around 100 lines of code.
First, a new temporary directory is created (since we don’t want to clobber the current working directory). In that directory, the downstream project, e.g. Istanbul, will be check out using git clone. For Esprima, we choose to track the master branch of every downstream project and live with the risk of instability. However, for your own project, you may consider tracking a stable branch instead.
After that, the project dependencies are installed via npm i. Obviously, among many different packages, this will install the stable, published version of Esprima. This is where the fun begins. That Esprima module will be replaced by dropping in the current development version of Esprima (i.e. of the feature branch associated with the pull request). Following this step, the project tests are executed by running npm test. If any of the tests failed, there is a possibility that it is caused by the pull request. Whether it is a success or a failure, the information will be posted to the pull request to help making the informed decision.
Note that in some cases, the downstream project itself could fail its own test suite. The solution is to track its more stable branch, rather than its development branch. Of course, if the project is too fragile and it does not keep the good hygiene, then it is not terribly suitable as a canary and you have to pick another project.
As with many other FOSS projects, Esprima is relying on Travis CI to run its tests. For this particular downstream test however, we decided to run it on a different hosted CI system, Circle CI. The reason is simple, we want to have the downstream tests running in parallel to the other usual battery of unit tests and regression tests. For Esprima, the entire downstream test could take some time to finish, about 14 minutes give or take. Running both in parallel means that every pull request can be reviewed after the Travis CI job is completed, while waiting for the longer downstream tests.
Maintaining a library? Be responsible and test your downstream projects!