When developing a web app, it is important to verify that the app works with various web browers. A very comprehensive way to do this is to use a Selenium Grid with a wide selection of web browsers connected to the grid. A complementary approach is to ensure that the continuous integration includes running the tests with the latest version of the most popular desktop browsers.
A setup to run Chrome, Firefox, Edge, and Safari to run your app tests is easier than you imagine. Even better, a few populer hosted continuous integration system offers multi-platform support. It is necessary since there is no single operating system that cover all the above browsers. Edge will always need Windows and Safari can run only on macOS.
In this blog post, and its accompanying git repository at github.com/ariya/evergreen-browser-tests, we discuss the use of three hosted CI systems: Travis CI, AppVeyor, and Azure Pipelines. The sample project itself is very simple. To run it locally, clone the repository, run npm install and after than, npm test. Note that some web browsers in your system will be detected and launched to run the tests. While the tests for this sample project are written using Mocha and executed using Karma, the concept should apply to any other combinations of test framework and test runner.
Travis CI is very popular among free and open-source developers. While initially the tests can only be executed on Linux, with the addition of macOS and Windows to the supported platforms, it is now possible to cover all these popular desktop operating systems. The Travis CI dashboard for this particular test project looks like the following:
As you can witness, the build matrix specifies running the browser tests on macOS and Linux. Unfortunately, as of now, running the browser tests on Travis CI Windows environment is highly inconsistent, leading to spurious failures. If anyone discovers a trick to resolve it, please let me know! The complete build log for each row shows the test output in much more detail (see this example).
language: node_js
node_js:
- "10"
os:
- linux
- osx
script: "npm run cli-tests"
jobs:
include:
- stage: Browser Tests
node_js: "10"
os: osx
script: "npm run browser-tests"
- stage: Browser Tests
node_js: "10"
os: linux
script: "xvfb-run npm run browser-tests"
The YAML file to achieve the above build on Travis CI is simple. Note the use of xvfb-run to ensure that the browsers can run in an X environment, thanks to Xvfb (X virtual framebuffer). On Linux, the tested browsers are the latest version of Chrome, Chromium, and Firefox. Meanwhile, on macOS, it will be only Safari. Currently, it is Safari 11 running on macOS 10.13 (High Sierra) and I am not entirely sure why it is not Safari 12 instead.
AppVeyor is another good CI system to cover both Linux and Windows. When the tests are being executed on Windows, they are running on Chrome, Chromium, Firefox, and Internet Explorer 11.
Here is the YAML configuration file for AppVeyor:
version: "{build}"
image:
- Visual Studio 2015
- Ubuntu
environment:
nodejs_version: "11"
stack: node 10
matrix:
fast_finish: true
install:
- ps: |
if ($isWindows) {
Install-Product node $env:nodejs_version
}
- cmd: choco install firefox
- cmd: choco install chromium
before_test:
- node --version
- npm --version
- npm install
test_script:
- cmd: npm test
- sh: xvfb-run npm test
build: off
Unfortunately Microsoft Edge is not available (since AppVeyor utilizes Windows Server). However, with the upcoming change of Edge to be based on Chromium instead of its EdgeHTML, I do not think we need to worry much about Edge for the near future.
Last but not least, Azure Pipelines, which got rebranded sometime ago after being previously known as Visual Studio Team Service, is the most complete solution as it offers running on tests on Linux, macOS, and Windows. Just like the other two services above, its configuration is also in the YAML format, presented here for your pleasure.
jobs:
- job: 'BrowserTests_Linux'
pool:
vmImage: 'ubuntu-16.04'
steps:
- task: NodeTool@0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js v10'
- task: Npm@0
inputs:
command: install
- script: Xvfb :99 &
displayName: 'Start Xvfb'
- script: DISPLAY=:99 npm test
displayName: 'Run the tests on Firefox and Chrome'
- job: 'BrowserTests_macOS'
pool:
vmImage: 'macOS-10.13'
steps:
- task: NodeTool@0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js v10'
- task: Npm@0
inputs:
command: install
- script: npm test
displayName: 'Run the tests on Safari'
- job: 'BrowserTests_Windows'
pool:
vmImage: 'vs2017-win2016'
steps:
- script: choco install firefox
displayName: 'Install the latest Firefox'
- script: choco install chromium
displayName: 'Install the latest Chromium'
- task: NodeTool@0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js v10'
- task: Npm@0
inputs:
command: install
- script: npm test
displayName: 'Run browser tests on Internet Explorer, Firefox, and Chrome'
The above file should be self explanatory. When all the jobs are being executed to the completed, a nice dashboard is shown.
All the above services (Travis CI, AppVeyor, Azure Pipelines) can be used generously to run the tests of your free/open-source projects. Therefore, if you have the need to cover all three major desktop operating systems (Windows, Linux, macOS), do not skip the opportunity to use these services and to ship a better product!