Image source: https://unsplash.com/photos/Grg6bwZuBMs

Avid readers of my blog know that I’m really appreciating a proper CI / CD setup. Until a few weeks ago, my go-to solution for such setups has been Travis CI. Unfortunately, Travis slightly changed its business model which drastically limits its use for open-source projects. For me, this was a huge setback, since I’m relying on Travis CI to provide nut.js. And since I’m not making any money out of this project, it’s hard to reason paying at least 69$ per month for CI.

As I had already started using GitHub Actions in other repos, I decided to migrate nut.js to GitHub Actions as well.

The existing pipeline

So, what do we have to migrate?

A nut.js build consists of two to three stages:

  • sonar
  • test
  • deploy (only triggered on tags and on commits to the develop branch)

The sonar stage run on Linux with a fixed node version and executes all tests.

That is:

  • all platform independent tests
  • all platform dependent tests
  • all tests requiring a GUI
  • all tests running in a Docker container

Generated coverage reports are merged and reported to sonarcloud.

The following test stage is a build matrix, running tests on both macOS and Linux with varying node versions.

On Linux, the test stage runs the exact same set of tests as the sonar stage while on macOS tests running in Docker containers are left out.

The optional deploy stage will either publish a new @next snapshot release for build on the develop branch, or a new @latest release for tags.

In summary, the requirements for a successful migration to GitHub Actions are the following:

  • Build stages
  • Build matrices
  • Docker support
  • Running GUI tests
  • Sonar reports

Let’s migrate!

The final setup is split into three distinct files, one for general branch and pull request build, one for snapshot releases and one for stable releases.

Let’s look at branch build first.

Sonar stage

Or workflow should run on pushes to all branches except develop and any release branch under release/**, as well as pull requests.

name: Run CI
on:
  push:
    branches-ignore:
      - develop
      - release/**
  pull_request:

Travis stages are resembled by jobs on GitHub Actions.

jobs:
  sonar:
    runs-on: ubuntu-latest
    steps:
      - name: Set up Git repository
        uses: actions/checkout@v2
      - name: Set up node
        uses: actions/setup-node@v2
        with:
          node-version: 14

Our job runs on Linux (ubuntu-latest defaults Ubuntu 18.04, soon upgrading to Ubuntu 20.04), checks out our repo and installs node v14.x.x.

On Linux, Docker is available right out of the box, so we’re able to start a daemonized container:

- name: Setup Docker
  run: |
    docker pull s1hofmann/nut-ci:latest
    docker run -it -d --name nut-ci --shm-size 4gb --user $(id -u):$(id -g) -v ${PWD}:${PWD}:rw s1hofmann/nut-ci:latest bash

What follows are some steps to initialise the project:

- name: Install
  run: npm ci
- name: Compile
  run: npm run compile
- name: Init e2e test subpackage
  run: npm --prefix e2e/tests ci
- name: Clean coverage report
  run: npm run coverage:clean

Once set up, we now want to run tests which require a GUI.

A common way of running headless GUI tests on Linux is using a virtual framebuffer. Travis provides a simple to use service to enable a virtual framebuffer in a build, so let’s find a proper replacement for GitHub Actions.

GabrielBB/xvfb-action provides a nice wrapper which will start a virtual framebuffer where possible.

When running on Linux, a virtual framebuffer will be set up before running your action, while on other platforms it will be executed right away. No additional configuration required when running in a cross-platform setup (like we do in our matrix build)!

Following the GUI and unit tests, let’s execute E2E tests running in a Docker container and merge all coverage reports:

- name: Run Docker E2E tests
  run: docker exec nut-ci bash -c "bash $PWD/.build/build.sh ${PWD} 14"
- name: Merge coverage reports
  run: |
    npm run coverage:merge
    npm run coverage:merge-report

Travis provides an add-on for sonarcloud to easily run sonarcloud analyses.

Luckily, there’s an official SonarCloud Action to run scans on GitHub Actions.

Running sonar-scanner with an existing sonar-project.properties config becomes as easy as

- name: Send results to SonarCloud
  uses: SonarSource/sonarcloud-github-action@v1.4
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

Instead of storing the encrypted Sonar token in our workflow file like on Travis, we’re storing and referencing it as a repo secret and we’re good to go!

Test stage

Following our sonar stage we want to run a build matrix with multiple platforms and node versions.

By default, jobs run in parallel, but since we only want to run our test job in case the previous sonar job has been successful, we’re setting a requirement to the previous job:

test:
  needs:
    - sonar

Now let’s configure the build matrix:

strategy:
  matrix:
    os: [ ubuntu-latest, windows-latest, macos-latest ]
    node: [ 10, 12, 14 ]
    exclude:
      - os: ubuntu-latest
        node: 14
    runs-on: ${{matrix.os}}
    steps:
      - name: Set up Git repository
        uses: actions/checkout@v2
      - name: Set up node
        uses: actions/setup-node@v2
        with:
          node-version: ${{matrix.node}}

We want to run our tests on Linux, Windows and macOS using node version 10, 12 and 14.

But since our sonar stage already ran on Linux using node 14, we’re excluding it from our build matrix to save time and compute. In total, our build matrix will expand into 8 separate builds, all running in parallel.

We also want to limit our Docker tests to Linux, so we can limit the whole Docker setup to build running on ubuntu-latest:

- name: Setup Docker
  if: ${{matrix.os == 'ubuntu-latest'}}
  run: |
    docker pull s1hofmann/nut-ci:latest
    docker run -it -d --name nut-ci --shm-size 4gb --user $(id -u):$(id -g) -v ${PWD}:${PWD}:rw s1hofmann/nut-ci:latest bash

    ...

- name: Run Docker E2E tests
  if: ${{matrix.os == 'ubuntu-latest'}}
  run: docker exec nut-ci bash -c "bash $PWD/.build/build.sh ${PWD} ${{matrix.node}}"

Deploy stage

Snapshot releases should only happen on commits to develop:

name: Create snapshot release
on:
  push:
    branches:
      - develop

while latest releases are limited to semver tags:

name: Create tagged release
on:
  push:
    tags:
      - v*.*.*

When releasing new packages we do not require a sonar scan, so we just skip this stage completely and only run our test stage.

Depending on the outcome of this stage, a package release is triggered:

deploy:
  needs:
    - test
  runs-on: ubuntu-latest
  steps:
    - name: Set up Git repository
      uses: actions/checkout@v2
    - name: Set up node
      uses: actions/setup-node@v2
      with:
        node-version: 14
    - name: Publish tagged release
      uses: JS-DevTools/npm-publish@v1
      with:
        token: ${{ secrets.NPM_TOKEN }}

To be resolved?

There’s quite a lot of duplication in our workflows which I’d like to reduce. Something similar to how GitLab CI handles it would be nice, but it looks like it has yet to be supported by GitHub Actions.

Another thing which is not that great is missing support to skip builds via commit message.

It’s a common pattern to skip CI builds via commit message by prefixing the message with e.g. [skip_travis] or [skip_ci].

Unfortunately, GitHub Actions do not yet support such kind of shortcut and will still run your workflows.

Conclusion

Migrating the existing Travis CI pipeline to GitHub Actions turned out easier than initially expected.

Due to the wide range of community actions it was fairly easy to tick all the required boxes.

As a result we’re now able to run all our builds on a single CI system (Windows builds were carried out on AppVeyor since Windows build are in an early stage on Travis while the overall build runtime decreased!