CI/CD on Haseeb Majid

How to Get Code Coverage From Playwright Tests in a Sveltekit App

Me — Sat, 15 Apr 2023 00:00:00 +0000

In this post, I will show you how to get code coverage reports from your Playwright tests in your SvelteKit app. So let’s imagine we are starting with the basic SvelteKit template. First, we need to install:

npm i -D vite-plugin-istanbul

We need the vite plugin to instrument our code using Istanbul. Istanbul is a tool which allows us to instrument our code such that it can determine which lines were covered by our tests.

vite.config.ts

First, let’s update our vite.config.ts (or .js) file:

import { sveltekit } from "@sveltejs/kit/vite";
import { defineConfig } from "vitest/config";
import istanbul from "vite-plugin-istanbul";

export default defineConfig({
  build: {
    sourcemap: true,
  },
  plugins: [
    sveltekit(),
    istanbul({
      include: "src/*",
      exclude: ["node_modules", "test/"],
      extension: [".ts", ".svelte"],
      requireEnv: false,
      forceBuildInstrument: true,
    }),
  ],
  test: {
    include: ["src/**/*.{test,spec}.{js,ts}"],
  },
});

forceBuildInstrument - Optional boolean to enforce the plugin to add instrumentation in build mode. Defaults to false.

In theory, we should just be able to use requireEnv: true and then pass the VITE_COVERAGE=true environment variable. To our playwright tests job, so instead I just set the forceBuildInstrument. Now, this always instrument our builds with Istanbul. However, we can do something like process.env.NODE_ENV === "test". Which will only run when NODE_ENV is test. In this example, we will keep it simple and leave it as is.

tests

Now let’s go to our tests folder we need to add a new file called baseFixtures.ts [1], which looks like this:

import * as fs from 'fs';
import * as path from 'path';
import * as crypto from 'crypto';
import { test as baseTest } from '@playwright/test';

const istanbulCLIOutput = path.join(process.cwd(), '.nyc_output');

export function generateUUID(): string {
  return crypto.randomBytes(16).toString('hex');
}

export const test = baseTest.extend({
  context: async ({ context }, use) => {
    await context.addInitScript(() =>
      window.addEventListener('beforeunload', () =>
        (window as any).collectIstanbulCoverage(JSON.stringify((window as any).__coverage__))
      ),
    );
    await fs.promises.mkdir(istanbulCLIOutput, { recursive: true });
    await context.exposeFunction('collectIstanbulCoverage', (coverageJSON: string) => {
      if (coverageJSON)
        fs.writeFileSync(path.join(istanbulCLIOutput, `playwright_coverage_${generateUUID()}.json`), coverageJSON);
    });
    await use(context);
    for (const page of context.pages()) {
      await page.evaluate(() => (window as any).collectIstanbulCoverage(JSON.stringify((window as any).__coverage__)))
    }
  }
});

export const expect = test.expect;

We will use test and expect functions from this module instead of the playwright ones. As this module, will collect the corresponding coverage files into .nyc_output in a JSON file.

Now open our tests file and update them to use the base fixture module from this:

import { expect, test } from '@playwright/test';

to this:

import { expect, test } from "./baseFixtures.js";

Run the Tests

Now we can run the tests like npm run test, this should create a new file .nyc_output. To see the actual coverage we need to use nyc we can do that:

npx nyc report --report-dir ./coverage --temp-dir .nyc_output --reporter=text --exclude-after-remap false

# output
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |     100 |        0 |     100 |     100 |                   
 +page.svelte |     100 |        0 |     100 |     100 | 16                
 +page.ts     |     100 |      100 |     100 |     100 |                   
--------------|---------|----------|---------|---------|-------------------

C8

If we want to generate a C8 (Coberatura) report we can do:

npx nyc report --report-dir ./coverage --temp-dir .nyc_output --reporter=cobertura --exclude-after-remap false

This will save the code coverage data in the C8 format in the coverage folder.

JUnit

We can also generate a JUnit file, first we need to add the following to our playwright.config.ts file: reporter: [["junit", { outputFile: "results.xml" }]]. Then to generate the report we need to run PLAYWRIGHT_JUNIT_OUTPUT_NAME=results.xml npm run test, we pass an environment variable to tell playwright where to save the JUnit file.

(Optional) Run in GitLab CI

If we put all of this together we can run the tests in GitLab CI and get the code coverage, JUnit and Coberatura reports like so:

image: node

stages:
  - test

tests:e2e:
  stage: test
  script:
    - npx playwright install
    - npm run test -- --reporter=junit
    - npx nyc report --report-dir ./coverage --temp-dir .nyc_output --reporter=cobertura --exclude-after-remap false
    - npx nyc report --report-dir ./coverage --temp-dir .nyc_output --reporter=text --exclude-after-remap false
  coverage: /All files[^|]*\|[^|]*\s+([\d\.]+)/
  artifacts:
    reports:
      junit: results.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
    when: always
    paths:
      - test-results/
    expire_in: 1 week

GitLab will be able to show some useful information using this code coverage data, such as lines covered in an open MR. It can show if the code coverage has gone up or down within the MR as well.

That’s it! Hopefully, this post has helped you set up retrieving code coverage from your playwright tests!

Appendix

TIL: How to Deploy 'Multiple' Sites on One GitLab Page Site

Me — Thu, 06 Apr 2023 00:00:00 +0000

I have a repo which I use to store all of my conference and similar talks. This repo includes any code examples and most importantly the slides.

At the moment all the slides are RevealJS “sites”, which means it’s a presentation built in HTML. Now I would like to have all my talks deployed to single GitLab pages site. For example something like:

So how can we do this ? Simple! In our job to deploy the site we just need to add public/docker-as-a-dev-tool and public/an-intro-to-pocketbase respectively. Which ever path we provide is that path we will be able to access those slides on.

Here is an example .gitlab-ci.yml:

# .gitlab-ci.yml
stages:
  - pages

pages:
  stage: pages
  only:
    - main
  scripts:
    - mkdir -p public/docker-as-a-dev-tool
    - mv docker-as-a-dev-tool public/docker-as-a-dev-tool
  artifacts:
    paths:
      - public

Voila that’s it! Now we have multiple slides deployed to the same GitLab pages site. So I can keep all of my talks nice and neat in a single repo.

Appendix

My Talks Project

TIL: How to Run Parallel Jobs on Gitlab CI (Different Stages)

Me — Fri, 03 Mar 2023 00:00:00 +0000

TIL: How to Run Parallel Jobs on Gitlab CI (Different Stages)

If you are familiar with Gitlab CI you probably know that jobs in the same stages will run in parallel. However you can also run jobs in different stages at the same time. Let’s see our .gitlab-ci.yml looks like this:

stages:
  - test
  - deploy

format:
  stage: test
  script:
    - task format

lint:
  stage: test
  script:
    - task lint

deploy:preview:
  stage: deploy
  only:
    - merge_request
  image: docker
  script:
    - task deploy

So for my use case when a user creates a merge request, I want to run some tests against the code, such as linting and formatting. But also deploy my app to a preview environment. Whilst these jobs belong to different stages.

How Can We Update GitLab CI on the Status of the Netlify Deploy Part I

Me — Tue, 17 Jan 2023 00:00:00 +0000

In this post, I will show you how you can update your GitLab CI pipeline with the status of your Netlify deploys. As it is quite nice to have everything in one place if our CI pipeline passed or failed.

I have spoken about this in a previous article, how you might go about setting up merge request preview environments with Netlify.

This can be set up to also leave comments as shown above, and notify us if our website deployed successfully. Which will look like so:

However, there is currently no way to update our CI pipeline/GitLab on the main branch when our app is deployed on Netlify ¹. Netlify does allow us to set up webhooks depending on the different statuses of our builds. So what we need is to set up webhooks on Netlify, then have that call something say a serverless function which will then call the GitLab to commit status API. When we update the commit status it looks something like this:

GitLab CI

Netlify GitLab Commit Status

I have created this simple script in Golang here. Which does this for us. In my case, I have it deployed as a google cloud function and have set up CI in that project so it automatically deploys when we push to the main branch.

deploy:
  image: google/cloud-sdk:latest
  only:
    - main
  script:
    - gcloud auth activate-service-account --key-file $GCLOUD_SERVICE_KEY
    - gcloud config set project haseeb-majid
    - gcloud functions deploy netlify-build-commit-status --region us-central1 --gen2 --runtime go119 --trigger-http --allow-unauthenticated --entry-point=webhook

Limitation