In this post, I will go over how you can use docker-compose and Gitlab CI. In this example, we will be running playwright tests directly on the Gitlab runner. The tests will start a SvelteKit server also running on the Gitlab runner. The SvelteKit server will connect to PocketBase (backend) running in docker-compose.

So essentially we need a way for something running locally to connect to something running in docker in Gitlab CI (on a runner). This is a pattern I am using in my new app Bookmarkey ¹.

Let’s pretend we have a docker-compose.yml file which looks something like this:

services:
  pocketbase:
    image: ghcr.io/muchobien/pocketbase:latest
    ports:
      - '9090:8090'
    volumes:
      - ./pb_data:/pb_data
      - /pb_public

and our package.json scripts section looks like this:

{
    "scripts": {
        "test": "playwright test",
    },
}

Finally the most important file let’s look at the gitlab ci file .gitlab-ci.yml:

stages:
  - test

tests:e2e:
  stage: test
  only:
    - merge_request
  image: mcr.microsoft.com/playwright:v1.29.0-jammy
  services:
    - docker:dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_HOST: tcp://docker:2375
    VITE_POCKET_BASE_URL: 'http://docker:9090'
  script:
    - # ... Installing docker and docker compose
    - docker compose up -d
    - npm run test

Since all Gitlab CI jobs run in Docker. If we want to run docker-compose inside a job we need to use the dind service ². We can then use docker normally i.e. using docker compose to start PocketBase. The most important line here is normally to connect to PocketBase I would use http://localhost:8090. However, since we are using the dind service we need to use docker instead of localhost. Hence this VITE_POCKET_BASE_URL: 'http://docker:9090', passed to my SvelteKit app.

I spent about two days debugging this issue, even though I’d solved this problem before 🤦‍♂️. So I decided to make a quick post so hopefully you can avoid wasting your time.

This post in fact will be a scheduled post I’m writing it on the 20th of November but it will be published on the 23rd of November. Assuming we are using Netlify to deploy our website, what we will need to do is trigger a rebuild of our site every day using Gitlab CI. Since Hugo builds a static site to publish our content after a set date we will need to rebuild our site. So let’s see how we can automate that. Else the post won’t be visible until I next make a commit to my main branch.

I use the date field to determine when to make this post visible on my blog, set in the front matter of the post (markdown).

---
title: "TIL: You Can Schedule Posts on Your Hugo Blog using Netlify and Gitlab CI"
canonicalURL: https://haseebmajid.dev/posts/2022-11-23-til-you-can-schedule-posts-on-your-hugo-blog-using-netlify-and-gitlab-ci/
date: 2022-11-23
---

We can do this by first creating a build hook on Netlify, go to your Site Settings (on Netlify) then Build and Deploy. Copy the build hook cURL request example which will look like curl -X POST -d {} https://api.netlify.com/build_hooks/[id].

Next, go to Gitlab and open your Settings > CI/CD settings and add a new variable called NETLIFY_DEPLOY_WEBHOOK and copy your build hook URL https://api.netlify.com/build_hooks/[id]. Mark the variable as protected.

Then update our .gitlab-ci.yml with the following:

stages:
  - deploy

deploy_site:
  stage: deploy
  image: curl
  variables:
    GIT_STRATEGY: none
  script:
    - curl -X POST -d '{}' $NETLIFY_BUILD_WEBHOOK
  only:
    - schedules

This job will run on a schedule, i.e. once a day, that will trigger the build hook and cause Netlify to rebuild our site. This means any posts due to be published that day will be visible on our site.

Finally, let’s schedule our CI job, go to CI/CD > Schedules, and then create a new schedule. Set the cron to whatever you want. My pipeline schedule is set as so:

This will cause our website to build once a day, triggering a build on Netlify. So when the Gitlab CI job runs at midnight on the 23rd of November this post will be visible to you all.

You can test your website using the --buildFuture flag i.e. hugo server -D --buildFuture when testing your site locally.

Appendix

• Inspired by

For most of my common dev tasks, I’ve started to rely on docker/docker compose to run commands locally. I have also started using vscode’s .devcontainers, to provide a consistent environment for all developers using a project.

The main reason for this is to avoid needing to install dependencies on my host machine. In theory, all I should need is a Docker daemon and a CLI (docker CLI) to interact with that Daemon. This also makes it far easier for any new developer to start working on my project and get set up.

What inspired me to do this change now (in my banter bus project) was I wanted to upgrade to python 3:10 to use some of the new typing features released. However when I tried to upgrade my CI pipeline started failing, after hours of trying to debug it. I ended up using Docker and everything ran smoothly.

Now to have a more consistent environment between my local environment and CI. So in theory, it means less chance of something passing locally but failing in CI.

Now we know why we want to do it. let’s look at how we do it.

Before

Let’s take a look at what a typical CI pipeline may look for a Python project (using banter bus). In this example, we will be using a FastAPI web service which uses Poetry to manage its dependencies.

image: python:3.10.5

variables:
  DOCKER_DRIVER: overlay2
  PIP_CACHE_DIR: "${CI_PROJECT_DIR}/.cache/pip"
  PIP_DOWNLOAD_DIR: ".pip"
  DOCKER_HOST: tcp://docker:2375

cache:
  key: "${CI_JOB_NAME}"
  paths:
    - .cache/pip
    - .venv

.test:
  services:
    - name: mongo:4.4.4
      alias: banter-bus-database
    - name: redis:6.2.4
      alias: banter-bus-message-queue
    - name: registry.gitlab.com/banter-bus/banter-bus-management-api:test
      alias: banter-bus-management-api
    - name: registry.gitlab.com/banter-bus/banter-bus-management-api/database-seed:latest
      alias: banter-bus-database-seed
  variables:
    MONGO_INITDB_ROOT_USERNAME: banterbus
    MONGO_INITDB_ROOT_PASSWORD: banterbus
    MONGO_INITDB_DATABASE: test
    BANTER_BUS_MANAGEMENT_API_DB_USERNAME: banterbus
    BANTER_BUS_MANAGEMENT_API_DB_PASSWORD: banterbus
    BANTER_BUS_MANAGEMENT_API_DB_HOST: banter-bus-database
    BANTER_BUS_MANAGEMENT_API_DB_PORT: 27017
    BANTER_BUS_MANAGEMENT_API_DB_NAME: test
    BANTER_BUS_MANAGEMENT_API_WEB_PORT: 8090
    BANTER_BUS_MANAGEMENT_API_CLIENT_ID: client_id
    BANTER_BUS_MANAGEMENT_API_USE_AUTH: "False"
    MONGO_HOSTNAME: banter-bus-database:27017
    BANTER_BUS_CORE_API_DB_USERNAME: banterbus
    BANTER_BUS_CORE_API_DB_PASSWORD: banterbus
    BANTER_BUS_CORE_API_DB_HOST: banter-bus-database
    BANTER_BUS_CORE_API_DB_PORT: 27017
    BANTER_BUS_CORE_API_DB_NAME: test
    BANTER_BUS_CORE_API_MANAGEMENT_API_URL: http://banter-bus-management-api
    BANTER_BUS_CORE_API_MANAGEMENT_API_PORT: 8090
    BANTER_BUS_CORE_API_CLIENT_ID: client_id
    BANTER_BUS_CORE_API_USE_AUTH: "False"
    BANTER_BUS_CORE_API_MESSAGE_QUEUE_HOST: banter-bus-message-queue
    BANTER_BUS_CORE_API_MESSAGE_QUEUE_PORT: 6379

stages:
  - test

before_script:
  - pip download --dest=${PIP_DOWNLOAD_DIR} poetry
  - pip install --find-links=${PIP_DOWNLOAD_DIR} poetry
  - poetry config virtualenvs.in-project true
  - poetry install -vv

test:lint:
  stage: test
  only:
    - merge_request
  script:
    - poetry run pre-commit run --all-files

test:unit-tests:
  stage: test
  only:
    - merge_request
  script:
    - poetry run pytest -v tests/unit

test:integration-tests:
  stage: test
  only:
    - merge_request
  extends:
    - .test
  script:
    - poetry run pytest -v tests/integration

The above looks quite complicated, but very simply we install our dependencies for each job the before_script section is used in all jobs. All jobs also use python:3.9.8 image, this is where our code is cloned into the CI pipeline.

Where our .pre-commit-config.yaml looks something like this:

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.3.0
    hooks:
      - id: check-yaml
        args: ["--allow-multiple-documents"]
  - repo: local
    hooks:
      - id: forbidden-files
        name: forbidden files
        entry: found copier update rejection files; review them and remove them
        language: fail
        files: "\\.rej$"
      - id: black
        name: black
        entry: poetry run black
        language: system
        types: [python]
      - id: flake8
        name: flake8
        entry: poetry run flake8
        language: system
        types: [python]
      - id: isort
        name: isort
        entry: poetry run isort --settings-path=.
        language: system
        types: [python]
      - id: pyupgrade
        name: pyupgrade
        entry: poetry run pyupgrade
        language: system
        types: [python]
        args: [--py310-plus]
      - id: mypy
        name: mypy
        description: Check python types.
        entry: poetry run mypy
        language: system
        types: [python]

pre-commit is a library we can use to add pre-commit hooks before we commit our code to git. Adding some checks that the code is consistent with the rules we defined. We can also just use it as a lint job, multiple linting tools together. Simplified. Hence here we are checking for code formatting, linting, import sorting etc. The details don’t matter but at the moment we need to have a virtualenv locally to run this.

Integration Tests

A slightly more interesting job is integration tests, it requires other docker containers, as our tests need Postgres and Redis to run. We can define these as services and then reference them in our job like so:

test:integration-tests:
  stage: test
  only:
    - merge_request
  extends:
    - .test
  script:
    - poetry run pytest -v tests/integration

Note the extends clause, which essentially merges the .test section with our job so it will look something like:

test:integration-tests:
  stage: test
  only:
    - merge_request
  services:
    - name: mongo:4.4.4
      alias: banter-bus-database
    - name: redis:6.2.4
      alias: banter-bus-message-queue
    - name: registry.gitlab.com/banter-bus/banter-bus-management-api:test
      alias: banter-bus-management-api
    - name: registry.gitlab.com/banter-bus/banter-bus-management-api/database-seed:latest
      alias: banter-bus-database-seed
  variables:
    MONGO_INITDB_ROOT_USERNAME: banterbus
    MONGO_INITDB_ROOT_PASSWORD: banterbus
    MONGO_INITDB_DATABASE: test
    BANTER_BUS_MANAGEMENT_API_DB_USERNAME: banterbus
    BANTER_BUS_MANAGEMENT_API_DB_PASSWORD: banterbus
    BANTER_BUS_MANAGEMENT_API_DB_HOST: banter-bus-database
    BANTER_BUS_MANAGEMENT_API_DB_PORT: 27017
    BANTER_BUS_MANAGEMENT_API_DB_NAME: test
    BANTER_BUS_MANAGEMENT_API_WEB_PORT: 8090
    BANTER_BUS_MANAGEMENT_API_CLIENT_ID: client_id
    BANTER_BUS_MANAGEMENT_API_USE_AUTH: "False"
    MONGO_HOSTNAME: banter-bus-database:27017
    BANTER_BUS_CORE_API_DB_USERNAME: banterbus
  	BANTER_BUS_CORE_API_DB_PASSWORD: banterbus
  	BANTER_BUS_CORE_API_DB_HOST: banter-bus-database
  	BANTER_BUS_CORE_API_DB_PORT: 27017
  	BANTER_BUS_CORE_API_DB_NAME: test
  	BANTER_BUS_CORE_API_MANAGEMENT_API_URL: http://banter-bus-management-api
  	BANTER_BUS_CORE_API_MANAGEMENT_API_PORT: 8090
  	BANTER_BUS_CORE_API_CLIENT_ID: client_id
  	BANTER_BUS_CORE_API_USE_AUTH: "False"
  	BANTER_BUS_CORE_API_MESSAGE_QUEUE_HOST: banter-bus-message-queue
  	BANTER_BUS_CORE_API_MESSAGE_QUEUE_PORT: 6379
  script:
    - poetry run pytest -v tests/integration

We also need to define a bunch of environment variables in this case so our containers can communicate with each other. Now, these are of course specific to my apps. But you can imagine a real-life project also needing a bunch of environment variables. As you can see this can get a bit messy and what is running locally may differ slightly from what is running in CI.

I have been caught out by these env variables in the past. Note variables like BANTER_BUS_CORE_API_MANAGEMENT_API_URL: http://banter-bus-management-api. The name of the container must match the URL we have provided

    - name: registry.gitlab.com/banter-bus/banter-bus-management-api:test
      alias: banter-bus-management-api

Docker DNS (link to DNS) is clever enough to work out the IP address. This is also different now to how we are running it locally.

After

Now we are running all our dev tasks in docker. We will use docker-compose to manage all of the containers, docker-compose makes managing multiple containers a lot easier. We define all of them in our docker-compose.yml file.

services:
  app:
    container_name: banter-bus-core-api
    build:
      context: .
      dockerfile: Dockerfile
      target: development
      cache_from:
        - registry.gitlab.com/banter-bus/banter-bus-core-api:development
    environment:
      XDG_DATA_HOME: /commandhistory/
      BANTER_BUS_CORE_API_DB_USERNAME: banterbus
      BANTER_BUS_CORE_API_DB_PASSWORD: banterbus
      BANTER_BUS_CORE_API_DB_HOST: banter-bus-database
      BANTER_BUS_CORE_API_DB_PORT: 27017
      BANTER_BUS_CORE_API_DB_NAME: test
      BANTER_BUS_CORE_API_MANAGEMENT_API_URL: http://banter-bus-management-api
      BANTER_BUS_CORE_API_MANAGEMENT_API_PORT: 8090
      BANTER_BUS_CORE_API_CLIENT_ID: client_id
      BANTER_BUS_CORE_API_USE_AUTH: "False"
      BANTER_BUS_CORE_API_MESSAGE_QUEUE_HOST: banter-bus-message-queue
      BANTER_BUS_CORE_API_MESSAGE_QUEUE_PORT: 6379
    ports:
      - 127.0.0.1:8080:8080
    volumes:
      - ./:/app
      - /app/.venv/ # This stops local .venv getting mounted
    depends_on:
      - database
      - management-api
      - message-queue
      - database-seed

  management-api:
    container_name: banter-bus-management-api
    image: registry.gitlab.com/banter-bus/banter-bus-management-api:test
    environment:
      BANTER_BUS_MANAGEMENT_API_DB_USERNAME: banterbus
      BANTER_BUS_MANAGEMENT_API_DB_PASSWORD: banterbus
      BANTER_BUS_MANAGEMENT_API_DB_HOST: banter-bus-database
      BANTER_BUS_MANAGEMENT_API_DB_NAME: banter_bus_management_api
      BANTER_BUS_MANAGEMENT_API_WEB_PORT: 8090
      BANTER_BUS_MANAGEMENT_API_CLIENT_ID: client_id
      BANTER_BUS_MANAGEMENT_API_USE_AUTH: "False"
    ports:
      - 127.0.0.1:8090:8090
    depends_on:
      - database

  database:
    container_name: banter-bus-database
    image: mongo:4.4.4
    environment:
      MONGO_INITDB_ROOT_USERNAME: banterbus
      MONGO_INITDB_ROOT_PASSWORD: banterbus
      MONGO_INITDB_DATABASE: banterbus
    volumes:
      - /data/db
    ports:
      - 27017:27017

  database-gui:
    container_name: banter-bus-database-gui
    image: mongoclient/mongoclient:4.0.1
    depends_on:
      - database
    environment:
      - MONGOCLIENT_DEFAULT_CONNECTION_URL=mongodb://banterbus:banterbus@banter-bus-database:27017
    volumes:
      - /data/db mongoclient/mongoclient
    ports:
      - 127.0.0.1:4000:3000

  database-seed:
    container_name: banter-bus-database-seed
    image: registry.gitlab.com/banter-bus/banter-bus-management-api/database-seed:latest
    environment:
      MONGO_INITDB_ROOT_USERNAME: banterbus
      MONGO_INITDB_ROOT_PASSWORD: banterbus
      MONGO_INITDB_DATABASE: banter_bus_management_api
      MONGO_HOSTNAME: banter-bus-database:27017
    depends_on:
      - database

  message-queue:
    container_name: banter-bus-message-queue
    image: redis:6.2.4
    volumes:
      - /data/datastore /data
    ports:
      - 127.0.0.1:6379:6379

Note: This file was already defined just not used in CI because I wanted to provide an easy way to start up my “tech stack”. So the file had gone unused.

How do run our dev tasks?

• lint: docker compose run app poetry run pre-commit run --all-files
• integration tests: docker compose run app poetry run pytest -v tests/integration

Then our CI pipelines could look simply like this:

image: docker

services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_HOST: tcp://docker:2375

before_script:
  - docker compose build

stages:
  - test

test:lint:
  stage: test
  only:
    - merge_request
  script:
    - docker compose run app poetry run pre-commit run --all-files

test:unit-tests:
  stage: test
  only:
    - merge_request
  script:
    - docker compose run app poetry run pytest -v tests/unit

test:integration:
  stage: test
  only:
    - merge_request
  script:
    -  docker compose run app poetry run pytest -v tests/integration

Now before job we build our docker images, docker compose build. Then to run the dev task we do something like:

docker compose run app <command to run>

So to run unit tests we could do:

docker compose run app poetry run pytest -v tests/unit

Aside

We could simplify this if we use makefile and make the target be poetry run pytest -v tests/unit.

.PHONY: unit_tests
unit_tests: ## Run all the unit tests
	@poetry run pytest -v tests/unit

Then our ci job would look something like:

test:unit-tests:
  stage: test
  only:
    - merge_request
  script:
    - make unit_tests

Which I think is a lot more readable and a lot easier to type. We can also leverage auto-complete on the terminal and add help targets. So a user can see all the targets they can run.

This will be a slightly shorter article. In this article I will show you how I’ve managed to do some end-to-end testing with Gitlab CI services.

I’m building a browser-based multiplayer game called Banter Bus. Banter Bus consists of three main components,

• gui: A SvelteKit based frontend the user will interact with to play the game
• core-api: A Socketio API written in Python
• management-api: A simple RESTful API written in Python (FastAPI)

Now say I want to write some e2e Cypress tests, that will test all of these components interacting with each other. Which mainly will look something like gui -> core-api -> management-api.

Each of these project deploys its own Docker container, which we can then use for testing it. So how can we do this with Gitlab CI ?

Gitlab Services

What is a Gitlab CI service ?

The services keyword defines a Docker image that runs during a job linked to the Docker image that the image keyword defines. This allows you to access the service image during build time. - https://docs.gitlab.com/ee/ci/services/

Essentially they are Docker containers we can use in our CI jobs.

package.json

For the examples below assume our package.json scipts section looks something like:

{
  "dev": "svelte-kit dev",
  "e2e": "cypress run --browser chrome",
  "e2e:ci": "start-server-and-test dev http://localhost:3000 e2e"
}

Gitlab CI

Let’s take a look at an example .gitlab-ci.yml file:

stages:
  - test

cypress-e2e-chrome:
  image: cypress/browsers:node14.17.0-chrome88-ff89
  stage: test
  variables:
	BANTER_BUS_CORE_API_MANAGEMENT_API_URL: http://banter-bus-management-api
	BANTER_BUS_CORE_API_DB_HOST: banter-bus-database
	FF_NETWORK_PER_BUILD: 1
	# Hidden the rest of the variables as not to clutter the file
  services:
    - name: mongo:4.4.4
      alias: banter-bus-database
    - name: registry.gitlab.com/banter-bus/banter-bus-core-api:test
      alias: banter-bus-core-api
    - name: registry.gitlab.com/banter-bus/banter-bus-management-api:test
      alias: banter-bus-management-api
    - name: registry.gitlab.com/banter-bus/banter-bus-management-api/database-seed:latest
      alias: banter-bus-database-seed
  script:
	- npm ci
    - export VITE_BANTER_BUS_CORE_API_URL=http://banter-bus-core-api:8080
    - echo fs.inotify.max_user_watches=524288 | tee -a /etc/sysctl.conf && sysctl -p
    - npm run e2e:ci
  artifacts:
    expire_in: 1 week
    when: always
    paths:
      - cypress/screenshots
      - cypress/videos
    reports:
      junit:
        - results/TEST-*.xml

Services

Let’s break file down a bit, these are essentially all the dependencies of our gui application. We need all of these containers running.

In this case we need four containers (this doesn’t really matter):

• banter-bus-database: A database for the core-api and management-api
• banter-bus-core-api: The main API the gui will interact with
• banter-bus-management-api: Used to help manage our available games, questions etc
• banter-bus-database-seed: A short lived container which pre-fills the database with some values

services:
  - name: mongo:4.4.4
    alias: banter-bus-database
  - name: registry.gitlab.com/banter-bus/banter-bus-core-api:test
    alias: banter-bus-core-api
  - name: registry.gitlab.com/banter-bus/banter-bus-management-api:test
    alias: banter-bus-management-api
  - name: registry.gitlab.com/banter-bus/banter-bus-management-api/database-seed:latest
    alias: banter-bus-database-seed

In our examples the name field is the image name, this is the same name you’d use when using the docker pull command. The next field is alias this is the name we’ll use to reference that container. This is the container name.

To see how the alias is used, is to look at the environment variables we have provided for the job BANTER_BUS_CORE_API_DB_HOST: banter-bus-database. So core-api will try to connect to database using this host. You can read more about Docker is able to resolve this to an IP address here. Another example is how the URL core-api will use to connect to the management-api BANTER_BUS_CORE_API_MANAGEMENT_API_URL: http://banter-bus-management-api.

We’ve discussed the most important part of the CI file, but lets quickly discuss the rest for completeness

Variables

We’ve already spoken about this above, but lets take a quick look at the variables section.

variables:
  BANTER_BUS_CORE_API_MANAGEMENT_API_URL: http://banter-bus-management-api
  BANTER_BUS_CORE_API_DB_HOST: banter-bus-database
  FF_NETWORK_PER_BUILD: 1
  # Hidden the rest of the variables as not to clutter the f

These are environment variables that are shared both with the job and the services. Some of these are config passed to the application, such as BANTER_BUS_CORE_API_MANAGEMENT_API_URL and BANTER_BUS_CORE_API_DB_HOST.

Script

Since we are using the cypress/browsers:node14.17.0-chrome88-ff89 image, we have access to chrome (headless) browser we can use with Cypress.

So we can do something like so:

script:
  - npm ci
  - export VITE_BANTER_BUS_CORE_API_URL=http://banter-bus-core-api:8080
  - npm run e2e:ci

• npm ci: Installs our npm dependencies for the gui app
• export VITE_BANTER_BUS_CORE_API_URL=http://banter-bus-core-api:8080 exports an enviroment variable which will be used by the gui app so it knows the URL of the core-api. Note the use of the alias name here (and port :8080 default port for the core-api)
• npm run e2e:ci: Starts the dev server and then runs the cypress test, see start-server-and-test dev http://localhost:3000 e2e where e2e is cypress run --browser chrome

Artifacts

Finally, the artifacts are “things” that are left over after the build. In this case we use them in two ways:

• One to generate a coverage report with junit
• Two to save our Cypress screenshots and videos

The downloadable artifacts will expire after 1 week. The Cypress files can be useful when debugging a problem with your tests. You get a video of perhaps why the tests failed.

artifacts:
  expire_in: 1 week
  when: always
  paths:
    - cypress/screenshots
    - cypress/videos
  reports:
    junit:
      - results/TEST-*.xml

Appendix

• Example Project
• Example Job

Introduction

Gatsby

Gatsby is a static site generator (SSG) built upon React. It allows us to create “blazing” fast websites. In this example, we will use a simple blog starter template available and add a Cypress test.

Cypress

Fast, easy and reliable testing for anything that runs in a browser. - Cypress README

Cypress allows us to test a web application, how a real user would use the application. Cypress will be used to test our Gatsby application, though if it’s a site you can test it using Cypress.

Gitlab CI

Gitlab CI is a continuous integration pipeline that will allow us to run our tests automatically, such as when we merge code into the master branch.

Getting Started

Gatsby

Create a new Gatsby site, using this default Gatsby starter:

gatsby new gatsby-starter-blog https://github.com/gatsbyjs/gatsby-starter-blog
cd gatsby-starter-blog

(Optional) Typescript

Adding Typescript to a Gatsby web application.

yarn add typescript @types/react @types/react-dom @types/node -D
yarn add gatsby-plugin-typescript

Add the following to your gatsby-config.js.

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-typescript`,
      options: {
        isTSX: true, // defaults to false
        jsxPragma: `jsx`, // defaults to "React"
        allExtensions: true, // defaults to false
      },
    },
  ],
}

Then create a new file tsconfig.json (in the project root, where the gatsby-config.js is).

{
  "compilerOptions": {
    "module": "commonjs",
    "target": "esnext",
    "jsx": "preserve",
    "lib": ["dom", "esnext"],
    "strict": true,
    "noEmit": true,
    "isolatedModules": true,
    "esModuleInterop": true,
    "noUnusedLocals": false,
    "allowJs": true
  },
  "exclude": ["node_modules", "public", ".cache"]
}

Cypress

Now to finally add Cypress to our application so we can test it. First, install the dependencies.

yarn add -D cypress cypress-axe axe-core start-server-and-test
# Add types
yarn add -D @types/cypress-axe

Next, let’s create a cypress.json folder in the project root.

{
  "baseUrl": "http://localhost:8000/",
   "integrationFolder": "cypress/e2e"
}

Next, let’s add some new “scripts” to the package.json file.

  "scripts": {
    "cy:open": "cypress open",
    "cy:run": "cypress run",
    "build": "gatsby build",
    "develop": "gatsby develop",
    "format": "prettier --write \"**/*.{js,jsx,ts,tsx,json,md}\"",
    "start": "npm run develop",
    "serve": "gatsby serve",
    "clean": "gatsby clean",
    "test": "echo \"Write tests! -> https://gatsby.dev/unit-testing\" && exit 1",
    "test:e2e": "start-server-and-test 'yarn develop' http://localhost:8000 'yarn cy:open'",
    "test:e2e:ci": "start-server-and-test 'yarn develop' http://localhost:8000 'yarn cy:run'"
  }

These scripts allow us to start Cypress, cy:open opens a GUI to visualise our tests whereas cy:run does it all in the terminal (the browser runs in headless mode). Where we will run test:e2e:ci in our CI pipeline, here we use the start-server-and-test command to start our Gatsby server using yarn develop. Then we run cy:run to start our tests.

Structure

Create a new folder called cypress which will look something like this.

.
├── e2e
│   └── accessibility.test.ts
├── fixtures
│   └── graphql.json
├── plugins
│   └── index.js
├── support
│   ├── commands.js
│   ├── index.d.ts
│   └── index.js
└── tsconfig.json

mkdir -p cypress/support

Create a file at cypress/support/commands.js

Cypress.Commands.add(`assertRoute`, (route) => {
  cy.url().should(`equal`, `${window.location.origin}${route}`);
});

Add some custom types for Cypress index.d.ts if you are using Typescript.

/// <reference types="cypress" />

declare namespace Cypress {
  interface Chainable<Subject> {
    /**
     * Assert the current URL
     * @param route
     * @example cy.assertRoute('/page-2')
     */
    assertRoute(route: string): Chainable<any>;

    /**
     * Waits for Gatsby to finish the route change, in order to ensure event handlers are properly setup
     */
    waitForRouteChange(): Chainable<any>;
  }
}

Next create the index.js file, which should look something like this.

import "cypress-axe"
import "./commands"

Next, let’s create a plugin folder mkdir -p cypress/plugins.

// / <reference types="cypress" />
// ***********************************************************
// This example plugins/index.js can be used to load plugins
//
// You can change the location of this file or turn off loading
// the plugins file with the 'pluginsFile' configuration option.
//
// You can read more here:
// https://on.cypress.io/plugins-guide
// ***********************************************************

// This function is called when a project is opened or re-opened (e.g. due to
// the project's config changing)

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config
  return config
}

Now finally let’s create our tests folder mkdir -p cypress/e2e.

cypress-axe

In this blog post we won’t go over any complicated Cypress test we will simply use cypress-axe to test the accessibility of our (a11y) of our website.

/// <reference types="../support/index" />
/// <reference types="cypress" />
/// <reference types="@types/cypress-axe" />

describe("Component accessibility test", () => {
  it("Main Page", () => {
    cy.visit("/")
    cy.wait(500)
    cy.injectAxe()
    cy.checkA11y({
      include: [["#___gatsby"]],
    })
  })
})

Note the /// comments at the top used to add types for cypress. The test above will go to our home page and test if it has any a11y violations and if so will fail the test.

We can now run our tests locally by running this command:

yarn run test:e2e

Gitlab CI

Now how can we automate this, so the tests will run say every time we make changes on the master branch to make sure we haven’t broken any a11y. Create a new .gitlab-ci.yml or add the following job to an existing CI file.

image: node:12.14.1

variables:
  CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/cache/Cypress"

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - cache/Cypress
    - node_modules

stages:
  - test

before_script:
  - yarn install

tests:
  image: cypress/browsers:node12.14.1-chrome83-ff77
  stage: test
  script:
    - yarn test:e2e:ci

I won’t go into the details of what makes up a Gitlab CI file. At the top of the file, we will cache the node_modules file so we can share it between the job and the Cypress cache. The job itself is very simple it uses a cypress/browsers:node12.14.1-chrome83-ff77 Docker image which provides a headless chrome browser that Cypress can leverage to run the tests. As we won’t have access to a GUI in the Gitlab CI runner. The tests job is very simple it runs yarn test:e2e:ci to run our Cypress tests.

That’s it, quite simple to add Cypress tests that run in our CI pipeline.

Appendix

• Example source code
• Cypress
• Example Job

So what this entails is the source code for my Gatsby site is in a repo called portfolio-site on Gitlab. Then I have another repo for all of my blog posts (in markdown) called articles. During build time of the Gatsby blog, we will import the markdown files from our articles git repo and use it as a source of data for our Gatsby blog.

Git Plugin

First, install the Gatsby git plugin, so that we can source our data from git.

yarn add gatsby-source-git

Then add the plugin to your gatsby-config.js to tell it where to source its data from.

{
  resolve: `gatsby-source-git`,
  options: {
    name: `Articles`,
    remote: "https://gitlab.com/hmajid2301/articles.git",
    branch: `master`,
    patterns: ["**/*", "!**/*/index.md"],
  },
},

In our example, I will use this, the same repo this blog post originates from. You can specify the branch to use if you want. The most interesting bit is the patterns section. This is where you can specify which files to include and which to ignore ["**/*", "!**/*/index.md"]. In this example, I want to ignore all files called index.md because these are the ones that I use in my example Gatsby blogs in this repo for. You can read more about the pattern here.

GraphQL

We can now check if the articles are being imported correctly by using the GraphQL IDE that comes with Gatsby.

yarn develop

# Go to localhost:8000/__graphql

Then run the following query.

query MyQuery {
  allMarkdownRemark {
    edges {
      node {
        fileAbsolutePath
      }
    }
  }
}

You should see output like this, where it will list all of your blog posts/markdown files. Here you can verify your git repo is being sourced correctly.

{
  "data": {
    "allMarkdownRemark": {
      "edges": [
        {
          "node": {
            "fileAbsolutePath": "/home/haseeb/projects/personal/articles/35. Gatsby source git/source_code/.cache/gatsby-source-git/Articles/1. Expo with VirtualBox and Genymotion/README.md"
          }
        },
        {
          "node": {
            "fileAbsolutePath": "/home/haseeb/projects/personal/articles/35. Gatsby source git/source_code/.cache/gatsby-source-git/Articles/11. React Navigation with React Native/README.md"
          }
        },
        {
          "node": {
            "fileAbsolutePath": "/home/haseeb/projects/personal/articles/35. Gatsby source git/source_code/.cache/gatsby-source-git/Articles/13. REST API using OpenAPI, Flask & Connexions/README.md"
          }
        }
        // ...
      ]
    }
  }
}

Gatsby Node

Now make sure you have logic in your gatsby-node.js file to create a blog post page for every markdown file that we source, i.e. one blog post for every item in the list above.

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions;

  const blogPost = path.resolve(`./src/templates/blog-post.js`);
  const result = await graphql(
    `
      {
        allMarkdownRemark(
          sort: { fields: [frontmatter___date], order: DESC }
          limit: 1000
        ) {
          edges {
            node {
              frontmatter {
                title
                slug
              }
            }
          }
        }
      }
    `
  );

  if (result.errors) {
    throw result.errors;
  }

  // Create blog posts pages.
  const posts = result.data.allMarkdownRemark.edges;

  posts.forEach((post, index) => {
    const previous = index === posts.length - 1 ? null : posts[index + 1].node;
    const next = index === 0 ? null : posts[index - 1].node;

    createPage({
      path: post.node.frontmatter.slug,
      component: blogPost,
      context: {
        slug: post.node.frontmatter.slug,
        previous,
        next,
      },
    });
  });
};

Gitlab CI

So every time we make a change in our article repo we want to trigger a rebuild of our site. Since I use Gitlab, I will show you how you can do this with Gitlab CI. Every commit on the master branch, on our repo that contains our articles, will trigger a rebuild on the Gatsby repo.

For example, in my use case the article repo will trigger a rebuild for the Gatsby repo.

First, go to your Gatsby repo then go to Settings > CI/CD > Pipeline triggers. Then create a new pipeline trigger, save the newly created token to your CI/CD variables.

Then also copy the cURL command shown and add the following to your .gitlab-ci.yml, with the cURL command.

stages:
  - build

rebuild:portfolio-site:
  stage: build
  image: curlimages/curl
  only:
    - master
  script:
    - "curl -X POST -F token=${TRIGGER_TOKEN} -F ref=master https://gitlab.com/api/v4/projects/19260161/trigger/pipeline"

Make sure you replace 19260161 with the project ID of your Gatsby blog, as this is the repo we want to trigger a rebuild of. This means every time we push a new commit (i.e. an article) to the master branch of the articles, it will trigger the pipeline to run on our Gatsby blog.

This will mean when it runs yarn build or gatsby build it’ll source the markdown data from the latest commit on our article git repo and will have the new article or whatever changes were made. The .gitlab-ci for our Gatsby blog may look something like this:

image: node:12.13.0
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules

stages:
  - build
  - deploy

before_script:
  - yarn install

build:site:
  stage: build
  only:
    - master
  script:
    - yarn run build
  artifacts:
    paths:
      - public

deploy:site:
  stage: deploy
  only:
    - master
  script:
    - npm i -g netlify-cli
    - yarn deploy --site $NETLIFY_SITE_ID --auth $NETLIFY_PERSONAL_TOKEN --message "$CI_COMMIT_TITLE"
  dependencies:
    - build:site

I build and deploy the site from Gitlab CI to save the build minutes on Netlify. All you need to do this is to get your NETLIFY_SITE_ID and create a NETLIFY_PERSONAL_TOKEN that can make the API request to publish the site on your behalf.

Netlify

If you don’t want to use Gitlab CI to build and publish your Gatsby blog and want to force a rebuild on Netlify. Then you can do the following; Every time we push a new commit to the master branch on the article repo. We can use a webhook to trigger a rebuild of our site on Netlify. To do this select your website on the Netlify GUI. Then Settings > Build & deploy > Build hooks. Add a new build hook. Then copy the cURL command, so your article repo .gitlab-ci.yml now looks something like:

stages:
  - build

rebuild:portfolio-site:
  stage: build
  image: curlimages/curl
  only:
    - master
  script:
    - curl -X POST -d {} https://api.netlify.com/build_hooks/5f5e9c4f495aebe573c39aef

You will want to turn 5f5e9c4f495aebe573c39aef into a CI/CD variable, else anyone can force a rebuild of your site.

That’s it, we learnt how we can manage our markdown articles in a separate repo to our Gatsby blog! We went over how we can also automate the rebuild of our site using Gitlab CI and Netlify.

Appendix

• Example source code
• Example Gatsby blog repo
• Example Articles repo

(Optional) Git Feature Branches

Feel free to skip this section if you are already familar with feature branch, skip to the Gitlab Auto MR section

Before I introduce what the script does and how we use it, let’s go over why you might need to use it. Say you’re working on a project with multiple other people and you want to make sure you keep your master/production branch clean. One way to do that is everyone uses “feature” branches. So for every feature being added to the project you create a new short-lived branch off of the master branch. Then typically one developer will work on a feature and make a merge request when ready to get the branch merged into the main master branch, to integrate their work with everyone else’s.

This means your changes can get reviewed before they merged into the master branch. This helps keep the master branch “cleaner”, it should have fewer bugs etc. One good way to visualise this is in the diagram above. Let’s say we want to add reaction buttons to a blog. We would create a new branch called something like feature/add-reaction-buttons. Then we would commit our changes on the branch.

Usually, this is coupled with a CI pipeline which will auto-run against our code. It may run jobs like unit tests, linting and static code analysis. This acts as a kind of first step review, we need to make sure it’s passing (and is green) before people even start to review our code, as seen in the above diagram. Once the CI pipeline is working the merge request can be reviewed. After the merge request has been approved it can be merged into the master branch and we will add the new feature to our codebase, which will eventually get deployed to production. We can also then delete our old branch. You can read more about feature branches here.

Note there are many other git flows. This can project can be used in conjunction with all which will use some form of feature branching.

CI/CD

Feel free to skip this section if you are already familiar with CI/CD, Git and Gitlab CI.

Continuous Integration (CI) is typically defined as making sure all code being integrated into codebase works. It usually involves running a set of jobs referred to as a CI pipeline. Some jobs we may run include linting our code and running unit tests. This is usually done automatically using a tool such as Travis, Circle or even Gitlab.

One use case for this is when others are adding new features to our codebase and we want to check it still works. We can create a CI pipeline that will run unit tests against the new code automatically when a pull request (GitHub) or merge request (Gitlab) is opened. This saves us a lot of time, rather than having to copy the new features/code and then run the tests ourselves on our machine.

Continuous Delivery (CD) is typically an extension of CI to make sure that you can release new changes quickly. This means automating your release process, such that you can deploy your application at any point of time just by clicking on a button.

Continuous Deployment takes CD one step further by requiring no human intervention in deploying our application. You can read more about this here

Gitlab Auto MR

I created a simple CLI script, that I run during on Gitlab CI, which will auto-create merge requests every time you create a new branch. So you could have something like this:

stages:
  - pre

create:merge-request:
  image: registry.gitlab.com/gitlab-automation-toolkit/gitlab-auto-mr
  stage: pre
  except:
    - master
    - tags
  script:
    - gitlab_auto_mr -t master -c WIP -d .gitlab/merge_request_templates/merge_request.md -r -s --use-issue-name

To use this tool you will need to create a personal access token so that the tool can access the GitLab API on your behalf. You can find out how to do that here. In the the example above I have set the private token in my CI/CD variables as GITLAB_PRIVATE_TOKEN but you can also pass in the --private-token argument instead.

If you use registry.gitlab.com/gitlab-automation-toolkit/gitlab-auto-mr this Docker image, the cli tool already comes preinstalled with all of the dependencies as well. Else you can also install it manually using pip install gitlab-auto-mr.

This particular job will only be run on a branch that is not called a master. This is because of the except clause we defined above. Hence it should only run when we create a new feature branch. It will create a new merge request if one does not already exist between this new branch and the master branch. The target branch it will create the merge request to is set by the -t option, in this example -t master. For example, source branch -> target branch.

Next, we append all of our new merge requests with WIP, set by the -c WIP argument, where WIP typically means Work in progress so other devs know not to review our MR as it’s not ready. Next, the tool also allows you to specify a merge request template to use by passing the -d and giving it the path to a file. In this example .gitlab/merge_request_templates/merge_request.md looks like this:

# Description

<!-- please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. -->

## Type

- [ ] Bug Fix
- [ ] Improvement
- [ ] New Feature

Fixes #<!-- Issue Number -->

The other arguments do the following:

• -r: Will remove the source branch (our feature branch) after the MR has been approved
• -s: Will squash our commits into a single commit, so each feature branch will appear as a single commit onto the master branch
• --use-issue-name: If set and you do something like feature/#6 it will search for the issue with id 6 and pull information from there like labels and milestones etc. It will then assign those to this MR, an example of this can be seen with MR here where the issue #211 is here

You can get a full list of options available using this tool here at its project page.

Appendix

• This project was originally inspired by this other project and this post
• Project Page
• An example project
• Example MRs created with this tool
• Example pipeline running this tool

The diagram above is a visualisation of what we are trying to achieve. We want to spawn Docker containers using docker-compose from within our job. The spawning and destruction of these Docker containers will be done via our Python code. We can achieve this by using dind (Docker in Docker). I have written a previous article on this topic which you can read more about here. This article assumes you already somewhat familiar with Docker, docker-compose and Pytest.

This compose file will be used to start our Docker containers.

version: "3"

services:
  service1:
    container_name: container1
    image: docker
    command: ["tail", "-f", "/dev/null"]

  service2:
    container_name: container2
    image: docker
    command: ["tail", "-f", "/dev/null"]

Gitlab CI

Next, we have our .gitlab-ci.yml file, this file is used to tell Gitlab CI what our CI jobs should do. In this example, we have one job called test:integration which will run our integration tests. But before we do that we need a way to access the Docker daemon from within Gitlab CI, this can be done by using the docker:dind service.

The docker:dind image automatically using its entry point starts a docker daemon. We need to use this daemon to start/stop our Docker images within CI. The docker:dind (dind = Docker in Docker) image is almost identical to the docker image. The difference being the dind image starts a Docker daemon. In this example, the job will use the docker image as the client and connect to the daemon running in this container.

services:
  - docker:dind

tests:integration:
  stage: test
  image: hmajid2301/dind-docker-compose
  script:
    - pip install pytest docker lovely-pytest-docker
    - pytest -s tests/test_integration.py

The job itself is very simple, it uses a container which already comes with docker and docker-compose. Next we install the dependencies we need for our tests. Then it runs our tests.

Tests

Now onto our actual tests file. It looks more complicated than it is:

import docker as docker_py
import pytest

docker_client = docker_py.from_env()
docker_compose = None


@pytest.fixture(scope="session", autouse=True)
def docker(docker_services):
    global docker_compose
    docker_compose = docker_services


@pytest.fixture(scope="session", autouse=True)
def setup():
    docker_compose.start()
    yield
    docker_compose.shutdown()


def kill_container(container_name):
    container = get_container(container_name)
    container.kill()
    container.remove()


def get_container(container_name):
    containers = docker_client.containers.list()
    for container in containers:
        if container.name == container_name:
            return container


def start_container(service_name):
    docker_compose.start(service_name)


def test_two_containers():
    containers = docker_client.containers.list()
    assert len(containers) == 2


def test_kill_container1():
    kill_container("container1")
    containers = docker_client.containers.list()
    container1 = get_container("container1")
    assert len(containers) == 1
    assert not container1

def test_start_container1():
    start_container("service1")
    containers = docker_client.containers.list()
    container1 = get_container("container1")
    assert len(containers) == 2
    assert container11

The first part is the setup, we will use the python Docker library,docker. Which allow us to use Python code to control our Docker daemon. The first @pytest-fixture called docker allows us to use the lovely-pytest-docker library. To give us a docker_compose object which will allow us to again use Python code to control our docker-compose.yml file (start-up/stop containers). The library also has some very nice features such as waiting for containers or executing commands within containers. You can find the available functions here. Now we can access the docker_compose object by using our docker_compose global variable.

The reason we have both a library for Docker and docker-compose is because at the moment there is no way to use lovely-pytest-docker (as far as I’m aware) to stop a single container. So we need to use the standard docker library to do that. We also use the standard docker library to find out if a container is running.

Next, we have the setup() fixture which we auto use, this means the fixture is run before our test, normally a fixture would only be called once it has been referred to within another function. In this function, we start both of our containers in our docker-compose file. This is the same as running docker-compose up --build -d. Next we yield, how exactly the yield command works I won’t go over in this article, all you have to know is that everything after the yield will only be run after all of our tests. In this case we teardown our containers (stop them). This is the same as running docker-compose down.

import docker as docker_py
import pytest

docker_client = docker_py.from_env()
docker_compose = None


@pytest.fixture(scope="session", autouse=True)
def docker(docker_services):
    global docker_compose
    docker_compose = docker_services


@pytest.fixture(scope="session", autouse=True)
def setup():
    docker_compose.start()
    yield
    docker_compose.shutdown()

The next part of our file contains some helper functions I’ve written. These functions can be used by multiple tests. This helps our tests file stay more DRY (do not repeat yourself). You may well want to make these part of a (helper) class that you expose as a fixture. If you wanted to structure these properly so they can be accessed by more than one file. Also whilst we are on this topic, we may want to move our fixture to conftest.py, again to allow other files to use the same fixture we have defined here. But to keep this example simpler we will leave it here.

def kill_container(container_name):
    container = get_container(container_name)
    container.kill()
    container.remove()


def get_container(container_name):
    containers = docker_client.containers.list()
    for container in containers:
        if container.name == container_name:
            return container


def start_container(service_name):
    docker_compose.start(service_name)

Finally, onto our actual tests, in reality, these tests are very boring and not super useful but they should give you an idea what you can do. Such as killing the database container and then check how your Python application responds. Then you can start the database container and again check how your Python application responds. You can read more about how you can test Python Flask applications with pytest here.

So what do our tests do? Well, the first one checks the number of containers running is equal to 2. The next one kills container1 and checks that only one container is running and it’s not container1. Our final test starts container1 and checks that it is running and the number of containers running is back to 2. After this final test has completed then the setup fixture will run its docker_compose.shutdown() command.

def test_two_containers():
    containers = docker_client.containers.list()
    assert len(containers) == 2


def test_kill_container1():
    kill_container("container1")
    containers = docker_client.containers.list()
    container1 = get_container("container1")
    assert len(containers) == 1
    assert not container1

def test_start_container1():
    start_container("service")
    containers = docker_client.containers.list()
    container1 = get_container("container1")
    assert len(containers) == 2
    assert container1

That’s it we’ve managed to start/stop Docker containers from within Gitlab CI, using DinD. In a future article I will explain how you could run your tests within a Docker container you’ve started. Say you had three containers nginx, flask and postgres and you wanted to run your tests within the flask container. But for now that’s it, thanks for reading!

Appendix

• Example source code

A quick aside on terminology related to Docker:

• container: An instance of an image is called a container (docker run)
• image: A set of immutable layers (docker build)
• hub: The official registry where you can get more Docker images from (docker pull)

Example

Here is an example .gitlab-ci.yml file which can be used to build and push your Docker images to the Gitlab registry.

variables:
  DOCKER_DRIVER: overlay2

services:
  - docker:dind

stages:
  - publish

publish-docker:
  stage: publish
  image: docker
  script:
    - export VERSION_TAG=v1.2.3
    - docker login ${CI_REGISTRY} -u gitlab-ci-token -p ${CI_BUILD_TOKEN}
    - docker build -t ${CI_REGISTRY_IMAGE}:latest -t ${CI_REGISTRY_IMAGE}:${VERSION_TAG}  .
    - docker push ${CI_REGISTRY_IMAGE}:latest
    - docker push ${CI_REGISTRY_IMAGE}:${VERSION_TAG}

Explained

The code above may be a bit confusing, it might be a lot to take in. So now we will break it down line by line.

variables:
  DOCKER_DRIVER: overlay2

In our first couple of lines, we define some variables which will be used by all our jobs (the variables are global). We define a variable DOCKER_DRIVER: overlay2, this helps speed our Docker containers a bit because by default it uses vfs which is slower learn more here.

random-job:
  stage: publish
  variables:
    DOCKER_DRIVER: overlay2
  script:
    - echo "HELLO"

Note we could just as easily define variables just within our job as well like you see in the example above.

services:
  - docker:dind

The next couple of lines define a service. A service is a Docker image which links during our job(s). Again in this example, it is defined globally and will link to all of our jobs. We could very easily define it within our job just like in the variables example. The docker:dind image automatically using its entrypoint starts a docker daemon. We need to use this daemon to build/push our Docker images within CI.

The docker:dind (dind = Docker in Docker) image is almost identical to the docker image. The difference being the dind image starts a Docker daemon. In this example, the job will use the docker image as the client and connect to the daemon running in this container.

We could also just use the dind image in our job and simply start dockerd (& = in the background) in the first line. The dockerd command starts the Docker daemon as a client, so we can then communicate with the other Docker daemon. It would achieve the same outcome. I think the service approach is a bit cleaner but as already stated either approach would work.

publish-docker:
  stage: publish
  image: docker:dind
  script:
    - dockerd &
    ...
    - docker push ${CI_REGISTRY_IMAGE}:${VERSION_TAG}

Info: One common use case of Gitlab CI services is to spin up databases like MySQL. We can then connect to it within our job, run our tests. It can simplify our jobs by quite a bit.

Note: There are several other ways we could also build/push our images. This is the recommended approach.

stages:
  - publish

Next, we define our stages and give them names. Each job must have a valid stage attached to it. Stages are used to determine when a job will be run in our CI pipeline. If two jobs have the same stage, then they will run in parallel. The stages defined earlier will run first so order does matter. However in this example, we only have one stage and one job so this isn’t super important, more just something to keep in mind.

publish-docker:
  stage: publish
  ...

Now we define our job, where publish-docker is the name of our job on Gitlab CI pipeline. We then define what stage the job should run in, in this case, this job will run during the publish stage.

publish-docker:
  ...
  image: docker
  ...

Then we define what Docker image to use in this job. In this job, we will use the docker image. This image has all the commands we need to build and push our Docker images. It will act as the client making requests to the dind daemon.

script:
  - export VERSION_TAG=v1.2.3
  - docker login ${CI_REGISTRY} -u gitlab-ci-token -p ${CI_BUILD_TOKEN}
  - docker build -t ${CI_REGISTRY_IMAGE}:latest -t ${CI_REGISTRY_IMAGE}:${VERSION_TAG}  .
  - docker push ${CI_REGISTRY_IMAGE}:latest
  - docker push ${CI_REGISTRY_IMAGE}:${VERSION_TAG}

Finally, we get to the real meat and potatoes of the CI file. The bit of code that builds and pushes are Docker images to the registry:

- export VERSION_TAG=v1.2.3

It is often a good idea to tag our images, in this case, I’m using a release name. You could get this from say your setup.py or package.json file as well. In my Python projects I usually use this command export VERSION_TAG=$(cat setup.py | grep version | head -1 | awk -F= '{ print $2 }' | sed 's/[",]//g' | tr -d "'"), to parse my setup.py for the version number. But this can be whatever you want it to be. Here we have just kept it static to make things simpler but in reality, you’ll probably want to retrieve it programmatically (the version number).

- docker login ${CI_REGISTRY} -u gitlab-ci-token -p ${CI_BUILD_TOKEN}

Then we log in to our Gitlab registry, the environment variables $CI_REGISTRY and CI_BUILD_TOKEN are predefined Gitlab variables that are injected into our environment. You can read more about them here. Since we are pushing to our Gitlab registry we can just use the credentials defined within environment i.e. username=gitlab-ci-token and password a throwaway token.

Note: You can only do this on protected branches/tags.

- docker build -t ${CI_REGISTRY_IMAGE}:latest -t ${CI_REGISTRY_IMAGE}:${VERSION_TAG}  .
- docker push ${CI_REGISTRY_IMAGE}:latest
- docker push ${CI_REGISTRY_IMAGE}:${VERSION_TAG}

Finally, we run our normal commands to build and push our images. The place where you can find your images will depend on the project name and your username but it should follow this format

registry.gitlab.com/<username>/<project_name>/<tag>

(Optional) Push to DockerHub

- docker login -u hmajid2301 -p ${DOCKER_PASSWORD}
- export IMAGE_NAME="hmajid2301/example_project"
- docker build -t ${IMAGE_NAME}:latest -t ${IMAGE_NAME}:${VERSION_TAG}  .
- docker push ${IMAGE_NAME}:latest
- docker push ${IMAGE_NAME}:${VERSION_TAG}

We can also push our images to DockerHub, with the code shown above. We need to first login to DockerHub. Then change the name of our image <username>/<project_name>.

Appendix

• A good Stackoverflow Post
• Gitlab CI Docs
• Gitlab Example

Docz

Docz is a tool powered by Gatsby, it aims to make it easier to document your project. It uses a language called mdx which is like normal markdown with some extra features, i.e. md + jsx. The main advantage of using Docz is you can render components “live”, if you put them with the <playground> tags. A basic example may look like this:

---
name: Button
route: /
---

import { Playground, Props } from 'Docz'
import { Button } from './'

# Button

<Props of={Button} />

## Basic usage

<Playground>
  <Button>Click me</Button>
  <Button kind="secondary">Click me</Button>
</Playground>

I chose to use Docz because it was simple to set up and looks very nice. I was already writing my documentation in markdown so it seemed like a perfect fit, even though in my use case I didn’t use the playground to render components.

Example

You can find an example project using Docz here. This is one of my projects where I deployed the documentation using Gitlab Pages. You can find it here.

Getting Started

Ok, now let’s get into how we can add Docz to an existing project. We also need to have react-dom and react installed.

yarn add Docz

# or

npm install Docz

So our package.json looks like:

{
  "name": "example_app",
  "scripts": {
    "docs-dev": "docz dev",
    "docs-build": "docz build"
  },
  "dependencies": {
    "Docz": "2.2.0",
    "react": "16.9.0",
    "react-dom": "16.8.0"
  }
}

We then need to create our Doczrc.js configuration file, like so:

export default {
  src: "docs",
  description: "Example Documentation",
  menu: ["Introduction", "Contributing"],
  themeConfig: {
    initialColorMode: "dark",
  },
};

We will keep all our .mdx files in a folder called docs hence src: 'docs'. Each file will be shown as a page on our website. The description option will be the name of our website in an open browser tab. The menu option is the order in which our pages will show up in the sidebar (on the left-hand side by default). The names used in this menu option must match the names used in the frontmatter of that file should match the front matter in mdx pages. Finally, I want to use the dark mode by default which is the final option.

---
name: Button
route: /
---

We can view our current docz website by running docs-dev and then go to localhost:3000 on our dev machine.

Adding Pages

Ok, now let’s add our actual “pages” to our Docz website. First create a new folder called docs in your project root. Then we will create our first page called Introduction.mdx (this name doesn’t matter so much), where the page’s contents look something like:

---
name: Introduction
route: /
---

# Example

Welcome to this example app, in this app we will show examples.

Note: The front matter name must match the name we defined in the Doczrc.js menu option.

The route defines the path the user will see, i.e. in the stegappasaurus example this page will be shown on https://stegappasaurus.haseebmajid.dev/.

Next let’s create a second page called Contributing.mdx, which looks like:

---
name: Contributing
route: /contributing
---

# Contributing

Three main ways to contribute to this project are;

- **Adding a new feature:** Adding a new feature to the project, such as allow encoding of audio files alongside images
- **Improving a feature:** Extend/improve an existing feature, such as a small UI change
- **Fix an issue:** We have a list of [issues](https://gitlab.com/hmajid2301/stegappasaurus/issues), or you can fix your issue.

This page can be found on /contributing i.e. https://stegappasaurus.haseebmajid.dev/contributing. The page may look something like the image below. The titles are shown as sub-menus.

Gitlab Pages

Now that we have our Docz website and it is working locally, how can we deploy on Gitlab pages for all the world to see? Well first we need to add a job titled pages to our .gitlab-ci.yml file, then we need to store all of our static assets in a folder called public and make that an artifact of this job. This will tell Gitlab CI, that we want to publish this “website” to Gitlab Pages. Here is an example of what it may look like:

pages:
  only:
    - master
  script:
    - yarn
    - yarn docs-build
    - mv .Docz/dist/* public/
  artifacts:
    paths:
      - public

This will publish our “website” to https://.gitlab.io/<project_name>. So, for example, my username is hmajid2301 and my project name is stegappasaurus hence the website URL is https://hmajid2301.gitlab.io/stegappasaurus.

Custom Domain (Optional)

If you have your own custom domain, you can “host” the page under two URLs. In my case, I own haseebmajid.dev and I wanted to host it under a subdomain within that domain, like so https://stegappasaurus.haseebmajid.dev/. Gitlab makes this surprisingly easy to do:

Note: In this example, I am assuming we are using a subdomain and not the root domain i.e. example_app.haseebmajid.dev not haseebmajid.dev.

• First go to your project on Gitlab
• Next Settings (Left sidebar) > Pages > New Domain (Top Righthand Corner)
• Enter the domain name you’d like to use, i.e. example_app.haseebmajid.dev
• Press Create New Domain
• Copy the TXT record to verify ownership.
• Next go your domain provider, in my case it’s Google
• Go to your DNS settings for that domain

We will now create two DNS records a TXT record which verifies ownership of the domain. Go to settings for “pages” which can be found https://gitlab.com/<username>/<project_name>/pages. Then copy the data next to Verification status into your DNS settings.

Next we will create a CNAME record, a CNAME is used to point one domain record to another, i.e. a user comes to stegappasaurus.haseebmajid.dev CNAME -> hmajid2301.gitlab.io -> Resolve IP Address. Then the browser will take the user to the correct IP address. You can copy the CNAME data next to the DNS field.

After we’ve added the details it will take a few minutes for verification and for Let's encrypt to create a certificate for our website. Then you should be able to view your Docz website using both domains listed under the page settings of your project, i.e the gitlab.io url and your custom domain.

You can find more information here, with regards to Gitlab pages and custom domains.

Note: .dev domains always needs to be HTTPS encrypted (need a certificate).

Note: For Google Domains this data must be added in the DNS > Custom resource records.

That’s it. We have deployed a docz website using Gitlab CI onto Gitlab Pages and even added our custom domain to it.

Appendix

• Example Project
• Gitlab Tutorial

CI/CD

Before we dive straight into the CI file itself, let’s do a quicker refresher on some basic concepts. Feel free to skip this section if you are already familiar with CI/CD, Git and Gitlab CI.

Continuous Integration (CI), is typically defined as making sure all code being integrated into a codebase works. It usually involves running a set of jobs referred to as a CI pipeline. Some jobs we may run include linting our code and running unit tests. This is usually done automatically using a tool such as Travis, Circle or even Gitlab.

One particularly useful use case for this is when others are adding new features to our codebase and we want to check it still works. We can create a CI pipeline that will run unit tests against the new code automatically when a pull request (GitHub) or merge request (Gitlab) is opened. This saves us a lot of time, rather than having to copy the new features/code and then run the tests our selves on our machine.

Continuous Delivery (CD), is typically an extension of CI to make sure that you can release new changes quickly. This means automating your release process, such that you can deploy your application at any point of time just by clicking on a button.

Continuous Deployment takes CD one step further by requiring no human intervention in deploying our application. You can read more about this here.

Git

Git is a version control system (VCS), it is heavily tied in with CI. In git, we can make “commits” which are snapshots of our project at its current state. We can later revert back to older commits and compare files between commits (and much more). Usually every commit we push to Gitlab triggers a CI pipeline run against that current commit. Git also has this concept of branches, where usually the master branch contains our production-ready code and the other branches have new features being worked on. When our feature branches are ready they are merged into the master branch. Usually, the CI pipeline needs to be successfully running (green ticks) before this can happen, however.

Gitlab CI

Gitlab CI, is defined as a YAML file. In the file, we define “jobs” which can do various different task. You can read more here. Full reference docs here, which details all the different parameters we can use. To use Gitlab CI within our projects is very straight forward, create a new file .gitlab-ci.yml in our project root and then define our jobs (we will see this a bit later in the article).

Example

The image above shows an example of a workflow we may use. So we create a new branch for our feature called feature/add-x. We then create our commits (with our new code) and push them to Gitlab. Open a merge request, this triggers the CI pipeline (from the .gitlab-ci.yml) file. In this example, the pipeline fails, perhaps because a unit test failed. This causes the whole pipeline to fail.

We then fix our code so the unit tests pass and then create more commits and push them. This then triggers the pipeline to run again, this time it passes. Now our code is ready to be reviewed and merged into the main branch. After the code review, it will be merged onto the master (main) branch. Then we will trigger the deployment process, this can also be defined within our CI file.

.gitlab-ci.yml

Now onto the real meat and potatoes of this article, our .gitlab-ci.yml file for React Native apps. Taking a look at an example application. You can find the .gitlab-ci.yml and package.json in the appendix below or follow the link above. Now let’s take a look the .gitlab-ci.yml file.

setup

image: node:8

stages:
  - pre
  - test
  - publish
  - post

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/

variables:
  DOCKER_DRIVER: overlay2

before_script:
  - yarn generate-dotenv
  - yarn

First, we specify an image, this is the default docker image we will use for our “jobs”. Unless a job specifies an image explicitly in its definition it will use this one. In this example, we will use node 8 it already has node, npm and yarn installed. We could probably upgrade this to node 10 or even node 12 (long term releases of node).

Next, we define all the stages of our pipeline, any jobs in the same stage will run in parallel (at the same time). If a job in an earlier stage fails the pipeline won’t carry on to the next and will stop running at the current stage. The stages defined first such as pre and test run before stages defined later such as publish. Each job must be given a stage.

Next, we define a cache, we will cache the node_modules for future jobs (in this pipeline). Gitlab CI injects some predefined environment variables, one of them being CI_COMMIT_REF_SLUG.

We then define a variable DOCKER_DRIVER: overlay2, this helps speed our docker containers a bit because by default it uses vfs which is slower learn more here.

Finally, we define before_script which will run before every job unless we specify a before_script within the jobs themselves. In this example, we install our node_modules using yarn and create a .env file, we need the .env file for a few our the jobs. The .env file is used by React Native to set configuration within the app.

{
  "name": "stegappasaurus",
  "scripts": {
    "generate-dotenv": "sh util/generate-dotenv.sh > .env"
  }
}

Where BUGSNAG_API_KEY and CAT_API_KEY are environment variables which are injected by Gitlab more information here.

#!/usr/bin/env bash

cat  << EOF
BUGSNAG_API_KEY=${BUGSNAG_API_KEY}
CAT_API_KEY=${CAT_API_KEY}
EOF

Where the generated .env file will look like.

BUGSNAG_API_KEY=1232541
CAT_API_KEY=abxc-71379991

jobs

Note: For the example application I am showing it has two branches production (main) and master.

pre

Now, let’s take a look at our jobs in the CI file. The first job is used to close issues automatically on Gitlab if there is an issue number in the git commit. It uses the following tool gitlab-auto-close-issue. Which provides a docker image which contains the script to auto-close your issues. It will also remove labels from the issue if you want such as “Doing”. This job is only run on the master branch of our project.

Since we don’t need to install any dependencies to run the job before_script: [] is an empty list, therefore the default before_script defined above won’t run in this job. Also since we define a docker image within the job we don’t use the default docker image node:8.

close:issue:
  image: registry.gitlab.com/gitlab-automation-toolkit/gitlab-auto-close-issue
  stage: pre
  before_script: []
  only:
    - master
  script:
    - apk add --no-cache --upgrade grep
    - ISSUE=$(echo $CI_COMMIT_MESSAGE | grep -oP "(?<=Fixes \#)[0-9]+" || echo '1')
    - gitlab_auto_close_issue --issue $ISSUE --remove-label "Doing" --remove-label "To Do"

The next job automatically creates a merge request (MR) if the commits are not being pushed to master or production branches. It will create an MR as WIP with a template we defined in the .gitlab folder. We also set the option --use-issue-name where if we have a branch called say feature/#211 where #211 is an issue number (for that project). It will take certain bits of information from that issue and set it on the MR such as labels. More information about the tool gitlab-auto-mr.

create:merge-request:
  image: registry.gitlab.com/gitlab-automation-toolkit/gitlab-auto-mr
  stage: pre
  before_script: []
  except:
    - production
    - master
    - tags
  script:
    - gitlab_auto_mr -t master -c WIP -d .gitlab/merge_request_templates/merge_request.md -r -s --use-issue-name

Where the template could look something like this.

# Description

<!-- please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. -->

## Type

- [ ] Bug Fix
- [ ] Improvement
- [ ] New Feature

Fixes #<!-- Issue Number -->

test

This job called lint only runs on MRs not on the master branch i.e. it won’t run if create an MR from master to production. Hence the except clause. Finally we run the lint command which is defined in our package.json file as eslint src/**/*.{ts,tx,tsx}. This will run eslint against all of the code within our src folder.

lint:
  stage: test
  only:
    - merge_requests
  except:
    variables:
      - $CI_COMMIT_REF_NAME =~ /^master/
  script:
    - yarn lint

Then lint:code-formatter checks our code against prettier and see’s if it’s compliant with the code formatter.

lint:code-formatter:
  stage: test
  only:
    - merge_requests
  except:
    variables:
      - $CI_COMMIT_REF_NAME =~ /^master/
  script:
    - yarn code-formatter-check

Then we check all of our TS is valid, by running tsc --project . --noEmit --pretty --skipLibCheck. To make sure there aren’t any type mismatches.

lint:types:
  stage: test
  only:
    - merge_requests
  except:
    variables:
      - $CI_COMMIT_REF_NAME =~ /^master/
  script:
    - yarn types-check

We run our unit tests using jest (our test runner). We also use the --silent flag to hide various warnings raised by components we are testing. Like all the other jobs in this stage we only run this job in an MR.

tests:unit:
  stage: test
  only:
    - merge_requests
  except:
    variables:
      - $CI_COMMIT_REF_NAME =~ /^master/
  script:
    - yarn tests --silent

Finally, almost the same as the job above, except it only runs on the master branch it gets the code coverage from unit tests and stores the result using coverage (with some Regex). Where the coverage script is defined as jest --coverage in package.json. More information here. The code coverage can be shown on a badge, such as here.

tests:coverage:
  stage: test
  only:
    - master
  script:
    - yarn coverage --silent
  coverage: /All\sfiles.*?\s+(\d+.\d+)/

publish

Then on to our next stage. The job below actually publishes our app to the Play Store. It will only run when we’ve tagged one of our commits for release i.e. release/1.0.0. This will only be done on the production branch. We are also using another docker image which has Android and various our dependencies need for our React Native app.

I won’t do a massive deep dive into this job because I’ve already written an article about it here. But essentially what happens is we have various variables defined in our project in Gitlab such as our Keystore stored in base64 and the Keystore setting such as the username and password. To use the tool to auto-publish our app I need to have a play-store.json file and because my app uses react-native-firebaseI need agoogle-services.json` file.

I then generate a licenses.json file using the following command npm-license-crawler -onlyDirectDependencies --omitVersion -json src/data/licenses.json, there is a license view within my application which lists all of the main dependencies so I can properly credit those libraries this task generates that file.

I then generate a gradle.propeties file using sh util/generate-gradle-properties.sh > android/gradle.properties. Very similar to the .env the script we looked at above. Where the file looks something like:

#!/usr/bin/env bash

cat  << EOF
android.useAndroidX=true
android.enableJetifier=true
org.gradle.jvmargs=-Xms1g
MYAPP_RELEASE_STORE_FILE=stegappasaurus.keystore
MYAPP_RELEASE_STORE_PASSWORD=${ANDROID_KEYSTORE_PASSWORD}
MYAPP_RELEASE_KEY_ALIAS=${ANDROID_KEYSTORE_ALIAS}
MYAPP_RELEASE_KEY_PASSWORD=${ANDROID_KEYSTORE_KEY_PASSWORD}
EOF

This means we can reference the variables for the keystore within our build.gradle files without needing to hardcode the values and once again this file is generated from CI variables stored on the project itself. For example the app/build.gradle I have the following defined.

android {
    signingConfigs {
        release {
            if (project.hasProperty("MYAPP_RELEASE_STORE_FILE")) {
                storeFile file(MYAPP_RELEASE_STORE_FILE)
                storePassword MYAPP_RELEASE_STORE_PASSWORD
                keyAlias MYAPP_RELEASE_KEY_ALIAS
                keyPassword MYAPP_RELEASE_KEY_PASSWORD
            }
        }
    }
}

We then publish the application using publish-package script which runs yarn run bundle && bash util/publish-package.sh. Where publish-package.sh looks like

#!/usr/bin/env bash

echo $CI_COMMIT_TAG

if [[ $CI_COMMIT_TAG == *"alpha"* ]]; then
    echo "Publishing Package: Alpha"
    cd android && ./gradlew publish --track alpha
elif [[ $CI_COMMIT_TAG == *"beta"* ]]; then
    echo "Publishing Package: Beta"
    cd android && ./gradlew publish --track beta
elif [[ $CI_COMMIT_TAG == *"release"* ]]; then
    echo "Publishing Package: Production"
    cd android && ./gradlew publish --track production
else
    echo "Publishing Package: Internal"
    cd android && ./gradlew publish --track internal
fi

If the git tag is release/1.0.0 then we will publish this directly onto the production track. It also check if the tag contains alpha or beta if so then we publish it to different tracks.

echo "Publishing Package: Production"
cd android && ./gradlew publish --track production

Finally we make the assets and build folders available as artifacts for jobs in future stages.

publish:android:package:
  stage: publish
  image: reactnativecommunity/react-native-android
  only:
    - /^release/.*$/
  script:
    - echo fs.inotify.max_user_watches=524288 | tee -a /etc/sysctl.conf && sysctl -p
    - cd android
    - base64 -d $ANDROID_KEYSTORE > app/stegappasaurus.keystore
    - export VERSION=$(cat app.json | jq -r .version)
    - cat $PLAY_STORE_JSON > app/play-store.json
    - cat $FIREBASE_GOOGLE_SERVICES_JSON > app/google-services.json
    - yarn generate-licenses
    - yarn generate-gradle-properties
    - yarn publish-package --no-daemon
  artifacts:
    paths:
      - ./android/app/build/
      - ./android/app/src/main/assets/

post

Onto our final stage, the first job creates a Gitlab release. This job is again only run on release tags, but only for “final” release hence the except clause. It won’t run if the git tag contains beta or alpha in its name. The gitlab-auto-release tool much like the other tools above is used to automate this part of the Gitlab workflow.

The script also can use CHANGELOG.md, if it follows keepachangelog format. It takes the changelog from that file and copies into our release. Only for the matching version name i.e. release/1.0.0, would look for 1.0.0 in our changelog file. You can find an example release created by this script here. Also if you specify a job name after the --artifacts argument it will link that jobs artifacts in this release (if it was run in the same pipeline as this job). In this example, we want to include our Android app build (APK/AAB).

create:gitlab:release:
  image: registry.gitlab.com/gitlab-automation-toolkit/gitlab-auto-release
  stage: post
  only:
    - /^release/.*$/
  except:
    variables:
      - $CI_COMMIT_TAG =~ /beta/
      - $CI_COMMIT_TAG =~ /alpha/
  before_script: []
  script:
    - gitlab_auto_release -c CHANGELOG.md -d "This was auto-generated by the gitlab-auto-release tool, https://gitlab.com/gitlab-automation-toolkit/gitlab-auto-release." --artifacts "publish:android:package"

Our final job in this stage again only runs on release tags. It publishes our source maps to Bugsnag. Which is a bug tracking tool. When our app is published to the Play store the JavaScript is minified and so Bugsnag cannot give us a proper stack trace without the source maps. We must “tag” each upload with a version, hence we look in app.json file for the current app version. This job requires artifacts from the previous android publishing job publish:android:package, hence we mark it a dependency in dependencies. We need access to the bundle generated in the assets folder from the previous job. Rather than repeat the same “actions” here to generate the files we need. To speed up our CI we will just “copy” the files into the job by using artifacts.

publish:bugsnag:soucemaps:
  stage: post
  only:
    - /^release/.*$/
  script:
    - apt update && apt install -y jq
    - export VERSION=$(cat app.json | jq -r .version)
    - curl https://upload.bugsnag.com/react-native-source-map -F apiKey=${BUGSNAG_API_KEY} -F appVersion=${VERSION} -F dev=false -F platform=android -F sourceMap=@android/app/src/main/assets/index.map -F bundle=@android/app/src/main/assets/index.bundle -F projectRoot=`pwd`
    - yarn run bugsnag-sourcemaps upload --api-key=${BUGSNAG_API_KEY} --app-version=${VERSION} --minifiedFile=android/app/build/generated/assets/react/release/index.android.bundle --source-map=android/app/build/generated/sourcemaps/react/release/index.android.bundle.map --minified-url=index.android.bundle --upload-sources
  dependencies:
    - publish:android:package

other

Finally, we have a Gitlab defined job called pages, where using Gitlab pages we will publish documentation for this application. It will publish a static website present in the public. The documentation is built using docz. By default, you can access pages at https://hmajid2301.gitlab.io/stegappasaurus, i.e. username.gitlab.io/project_name but I have a google domain and using a CNAME you can also view the website on https://stegappasaurus.haseebmajid.dev/.

Since this is a special job this job is run at the very end of our pipeline, also this job on runs on the master branch.

pages:
  only:
    - master
  before_script:
    - yarn
  script:
    - yarn docs-build
    - mv .docz/dist/* public/
  artifacts:
    paths:
      - public

Finally it’s done! That’s it! That is one example of a .gitlab-ci.yml file you can use to for your React Native projects.

Appendix

• Example Project
• Cover image

Firebase is a cloud-based platform developed by Google to aid in the development of Web and Mobile applications. It is tightly coupled with the Google Cloud Platform (GCP), so much so that there are certain actions you can only do using the GCP GUI, such as increasing the RAM of your cloud function “containers”.

Note: We will be using Typescript in this article

Firebase Cloud Functions

Firebase Cloud Functions can be referred to as serverless or as Functions-as-a-service (FaaS). This means we simply deploy our code as a function, the tool (Firebase) installs our dependencies and set up the environment. Essentially all we manage is the “code” and let the platform manage the actual server/environment.

Note: You can deploy cloud functions in Python and Golang, however, you must do this through the GCP GUI. The functions will show up on your Firebase GUI after you’ve created them. You can view your Firebase project within GCP hence you can make changes to it such as increasing the RAM (from 512MB to 2GB) from within GCP.

Let’s now take a look at a simple app we will deploy to Firebase Cloud Functions.

Structure

Our project structure will look something like this:

├── firebase.json
├── .firebaserc
├── functions
│   ├── index.ts
│   ├── middleware
│   ├── node_modules
│   ├── package.json
│   ├── tsconfig.json
│   └── yarn.lock
├── .gitignore
└── .gitlab-ci.yml

This setup will look very similar to the tutorial available here.

Firebase Misc Files

This file contains some configuration options but for most projects, it will just contain the project name (the one we want to publish our changes to on Firebase, as we could be working on multiple projects).

{
  "projects": {
    "default": "ExampleProject"
  }
}

This file is important as it defines the actions that will happen before we deploy a new version of the cloud functions. In this case, we run yarn run build, within the functions folder. It compiles our TypeScript (TS) into regular JavaScript (JS) so that it can be run as a cloud function. You could do various other actions such as lint your code etc.

{
  "functions": {
    "predeploy": [
      "yarn --cwd \"$RESOURCE_DIR\" run build"
    ]
  }
}

Gitlab CI

Now you’re probably wondering how do we get our Cloud Functions from our dev machine (computer) to the Firebase servers. We run the deploy script command. Now we could do this every time we make a change, however, I prefer to automate this process.

We will use Gitlab CI to automatically publish changes to Firebase. First, we will need a deploy token as we cannot enter our username and password within GitLab CI to do this run yarn firebase login:ci. Then log in to your Firebase account after you’ve done this you will get a deploy token (shown in the terminal), then;

• Open your Gitlab project in a web browser
• Go to Settings (left-hand sidebar) > CI/CD
• Variables -> Expand
• Add a new variable, with Type: Variable, Key: FIREBASE_DEPLOY_TOKEN, Value: your deploy token here, and toggle protected and masked as true (blue).

This now means you can access the token within the Gitlab CI as an environment variable, and it will allow us to authenticate with Firebase and push changes to Firebase.

image: node:8

stages:
  - publish

publish:firebase:functions:
  stage: publish
  only:
    - /^release/.*$/
  script:
    - cd functions
    - yarn
      - yarn run deploy -m "Pipeline $CI_PIPELINE_ID, build $CI_BUILD_ID" --non-interactive --token $FIREBASE_DEPLOY_TOKEN