I was having issues with my debugger today (well technically yesterday because I am publishing this a day later to spread out my blog posts but same difference) and it took me a few hours to realise what was going on. In my case, I was trying to debug a test written in Golang using nvim-dap-go on Neovim.

The reason the test was failing because the test file had build tags so delve (the debugger) couldn’t compile the file, i.e. using the unit build tag in the example below.

//go:build unit
// +build unit

package options_test

When we set up the dag-go plugin, we need to pass an extra field called build_flags:

lua require('dap-go').setup {
  delve = {
    build_flags = "-tags=unit,integration,e2e",
  },
}

Replace the above with the tags you use! If you are using NixVim you should be able to do something like this ¹:

{
  programs.nixvim = {
    plugins = {
      dap.extensions.dap-go = {
        enable = true;
        delve = {
          path = "${pkgs.delve}/bin/dlv";
          buildFlags = "-tags=unit,integration,e2e";
        };
      };
    };
  };
}

Then I have a keybinding to debug the nearest test like so, again using NixVim.

  {
    action = "<cmd> lua require('dap-go').debug_test()<CR>";
    key = "<leader>td";
    options = {
      desc = "Debug Nearest (Go)";
    };
    mode = [
      "n"
    ];
  }

in lua:

vim.keymap.set("n", "<leader>td", "<cmd>lua require('dap-go').debug_test()<CR>")

I am using the LazyVim neovim distribution, where these two come as easily installed extra plugins. However, it should be easy enough for you to copy the config over to your neovim lua config. LazyVim even has the config available on its website, you don’t even need to deep dive into the code 😄.

For example with dap. Here are some examples of what it can look like. The two main bits we will look at today don’t come by default with LazyVim but are “extras” we can add them

Neotest

To set up Neotest on LazyVim, go to your LazyVim starter repo. Then open our lazy.lua file and add the following lines:

spec = {
    -- add LazyVim and import its plugins
    { "LazyVim/LazyVim", import = "lazyvim.plugins" },
    -- import any extras modules here
    { import = "lazyvim.plugins.extras.test.core" },
    { import = "lazyvim.plugins.extras.lang.go" },
    -- import/override with your plugins
    { import = "plugins" },
},

Here you can see the output from a test at the bottom (from Neotest) and a test summary (on the right). Using the test summary we can easily run one test at a time, even subtests.

The config that LazyVim sets the following key binds.

keys = {
    { "<leader>tt", function() require("neotest").run.run(vim.fn.expand("%")) end, desc = "Run File" },
    { "<leader>tT", function() require("neotest").run.run(vim.loop.cwd()) end, desc = "Run All Test Files" },
    { "<leader>tr", function() require("neotest").run.run() end, desc = "Run Nearest" },
    { "<leader>ts", function() require("neotest").summary.toggle() end, desc = "Toggle Summary" },
    { "<leader>to", function() require("neotest").output.open({ enter = true, auto_close = true }) end, desc = "Show Output" },
    { "<leader>tO", function() require("neotest").output_panel.toggle() end, desc = "Toggle Output Panel" },
    { "<leader>tS", function() require("neotest").run.stop() end, desc = "Stop" },
},

So we can get the test summary by pressing <space> + t + s. Or test output using <space> + t + shift + o. Where <space> is my leader key. Also if a test fails when I run the nearest test <space> + t + n, the output panel (at the bottom) auto opens.

DAP Core

Similar to above to add debugging support we can add an extra plugin like so:

spec = {
    -- add LazyVim and import its plugins
    { "LazyVim/LazyVim", import = "lazyvim.plugins" },
    -- import any extras modules here
    { import = "lazyvim.plugins.extras.dap.core" },
    -- import/override with your plugins
    { import = "plugins" },
},

To debug our tests first we can add a breakpoint using <space> + d + b. Then we can run our test in debug mode using <space> + t + d, which will run the nearest test in debug mode.

When we start to debug our tests it looks something like this. If you have used the VSCode debugger this will look very similar. We can explore the current state, the values various variables have etc. Run commands in the debugger at the bottom.

That’s it! We can improve our testing experience in Go using neovim with Neotest and DAP. I am also able to replicate some features I used to have in VSCode and now also have in neovim 💗.

Appendix

• This video helping me get set up by Elijah Manor (he does great videos btw)

In this post, I will show you how to get code coverage reports from your Playwright tests in your SvelteKit app. So let’s imagine we are starting with the basic SvelteKit template. First, we need to install:

npm i -D vite-plugin-istanbul

We need the vite plugin to instrument our code using Istanbul. Istanbul is a tool which allows us to instrument our code such that it can determine which lines were covered by our tests.

vite.config.ts

First, let’s update our vite.config.ts (or .js) file:

import { sveltekit } from "@sveltejs/kit/vite";
import { defineConfig } from "vitest/config";
import istanbul from "vite-plugin-istanbul";

export default defineConfig({
  build: {
    sourcemap: true,
  },
  plugins: [
    sveltekit(),
    istanbul({
      include: "src/*",
      exclude: ["node_modules", "test/"],
      extension: [".ts", ".svelte"],
      requireEnv: false,
      forceBuildInstrument: true,
    }),
  ],
  test: {
    include: ["src/**/*.{test,spec}.{js,ts}"],
  },
});

forceBuildInstrument - Optional boolean to enforce the plugin to add instrumentation in build mode. Defaults to false.

In theory, we should just be able to use requireEnv: true and then pass the VITE_COVERAGE=true environment variable. To our playwright tests job, so instead I just set the forceBuildInstrument. Now, this always instrument our builds with Istanbul. However, we can do something like process.env.NODE_ENV === "test". Which will only run when NODE_ENV is test. In this example, we will keep it simple and leave it as is.

tests

Now let’s go to our tests folder we need to add a new file called baseFixtures.ts [1], which looks like this:

import * as fs from 'fs';
import * as path from 'path';
import * as crypto from 'crypto';
import { test as baseTest } from '@playwright/test';

const istanbulCLIOutput = path.join(process.cwd(), '.nyc_output');

export function generateUUID(): string {
  return crypto.randomBytes(16).toString('hex');
}

export const test = baseTest.extend({
  context: async ({ context }, use) => {
    await context.addInitScript(() =>
      window.addEventListener('beforeunload', () =>
        (window as any).collectIstanbulCoverage(JSON.stringify((window as any).__coverage__))
      ),
    );
    await fs.promises.mkdir(istanbulCLIOutput, { recursive: true });
    await context.exposeFunction('collectIstanbulCoverage', (coverageJSON: string) => {
      if (coverageJSON)
        fs.writeFileSync(path.join(istanbulCLIOutput, `playwright_coverage_${generateUUID()}.json`), coverageJSON);
    });
    await use(context);
    for (const page of context.pages()) {
      await page.evaluate(() => (window as any).collectIstanbulCoverage(JSON.stringify((window as any).__coverage__)))
    }
  }
});

export const expect = test.expect;

We will use test and expect functions from this module instead of the playwright ones. As this module, will collect the corresponding coverage files into .nyc_output in a JSON file.

Now open our tests file and update them to use the base fixture module from this:

import { expect, test } from '@playwright/test';

to this:

import { expect, test } from "./baseFixtures.js";

Run the Tests

Now we can run the tests like npm run test, this should create a new file .nyc_output. To see the actual coverage we need to use nyc we can do that:

npx nyc report --report-dir ./coverage --temp-dir .nyc_output --reporter=text --exclude-after-remap false

# output
--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |     100 |        0 |     100 |     100 |                   
 +page.svelte |     100 |        0 |     100 |     100 | 16                
 +page.ts     |     100 |      100 |     100 |     100 |                   
--------------|---------|----------|---------|---------|-------------------

C8

If we want to generate a C8 (Coberatura) report we can do:

npx nyc report --report-dir ./coverage --temp-dir .nyc_output --reporter=cobertura --exclude-after-remap false

This will save the code coverage data in the C8 format in the coverage folder.

JUnit

We can also generate a JUnit file, first we need to add the following to our playwright.config.ts file: reporter: [["junit", { outputFile: "results.xml" }]]. Then to generate the report we need to run PLAYWRIGHT_JUNIT_OUTPUT_NAME=results.xml npm run test, we pass an environment variable to tell playwright where to save the JUnit file.

(Optional) Run in GitLab CI

If we put all of this together we can run the tests in GitLab CI and get the code coverage, JUnit and Coberatura reports like so:

image: node

stages:
  - test

tests:e2e:
  stage: test
  script:
    - npx playwright install
    - npm run test -- --reporter=junit
    - npx nyc report --report-dir ./coverage --temp-dir .nyc_output --reporter=cobertura --exclude-after-remap false
    - npx nyc report --report-dir ./coverage --temp-dir .nyc_output --reporter=text --exclude-after-remap false
  coverage: /All files[^|]*\|[^|]*\s+([\d\.]+)/
  artifacts:
    reports:
      junit: results.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
    when: always
    paths:
      - test-results/
    expire_in: 1 week

GitLab will be able to show some useful information using this code coverage data, such as lines covered in an open MR. It can show if the code coverage has gone up or down within the MR as well.

That’s it! Hopefully, this post has helped you set up retrieving code coverage from your playwright tests!

Appendix

• Example source code
• Playwright Discussion
• Inspired By this repo

Recently I have been creating a SvelteKit app, when creating a new SvelteKit app you get a choice of different things you can add. Such as using vitest for unit testing.

I needed to spy on/mock a method in a class, to see if it was called when a button was pressed and, it was called with the correct arguments.

Let’s say we have a Button component which looks like this:

<!-- button.svelte -->
<script lang="ts">
    import { API } from "./api"

    const api = new API()
</script>

<button on:click={api.create}>Press Me!</button>

Then the api.ts looks something like this:

export class API {
	async create() {
        // ... does something
	}
}

So how do we test that the create method is called? Let’s assume we will be using the svelte testing library ¹.

// __tests__/button.test.ts
import { render } from "@testing-library/svelte";
import userEvent from "@testing-library/user-event";
import { describe, expect, test, vi } from "vitest";

import Button from "../Button.svelte";
import { API } from "./api";

describe("Button", () => {
	test("Successfully render Button", async () => {
		const user = userEvent.setup();
		const mock = vi.spyOn(API.prototype, "create");

		const { getByRole, getByLabelText } = render(Button, { props: {} });

		const button = getByRole("button", { name: "Add Bookmark" });
		await user.click(button);
		expect(mock).toHaveBeenCalled();
	});
});

The key being this line vi.spyOn(API.prototype, "create"), where we need to use .prototype. So we can access the methods in the class. Without the .prototype we will not find the method on the object itself.

That’s it! Thanks for reading.

Appendix

• Example Test from my App

In this post, I will quickly show you how you can reduce boilerplate code to log in to your app in Playwright tests. Log in to our app is a very common action that will likely be required in most of our tests.

You can read this documentation here, which will explain how you can set this up. However, what do you do if you want to say test the login flow or the register flow? You cannot use the same auth file as all the other tests as you will already be logged in.

It is a great way to reduce boilerplate code from our tests, which would require us to write code to log in before each test. You could log yourself in the test but, then we are adding more boilerplate code again.

What you can do is create a 2nd file say called playwright/.auth/not_logged_in.json which looks like this:

{
	"cookies": [],
	"origins": []
}

In other words there is no state, so the user will not be logged in. Then we can do something like this in say our login.test.ts:

import type { Page, expect, test } from "@playwright/test";

test.describe(() => {
	let page: Page;

	test.beforeEach(async ({ browser }) => {
		const loginContext = await browser.newContext({
			storageState: "playwright/.auth/not_logged_in.json"
		});
		page = await loginContext.newPage();
	});

	test("Successfully login to app", async ({ baseURL }) => {
		await page.goto("/login");

		const email = "test@bookmarkey.app";
		await page.locator('[name="email"]').type(email);

		const password = "password@11";
		await page.locator('[name="password"]').type(password);

		await page.locator('button[type="submit"]').click();
		await page.waitForURL(`${baseURL}/my/collections/0`);
	});

	test("Fail to login to app using incorrect credentials", async ({ baseURL }) => {
		await page.goto("/login");

		const email = "test@bookmarkey.app";
		await page.locator('[name="email"]').type(email);

		const password = "wrong_password";
		await page.locator('[name="password"]').type(password);

		await page.locator('button[type="submit"]').click();

		const toastMessage = await page.locator(".message").innerText();
		expect(toastMessage).toBe("Wrong email and password combination.");
		await page.waitForURL(`${baseURL}/login`);
	});
});

Rather than using the “normal” page argument provided by Playwright, we will create our context. Using this stateless file, will mean the user is not logged in.

If we still want a test to use to automatically logged in we can do:

import { test } from "@playwright/test";

test.describe(() => {
	test("Successfully load collections in side bar", async ({ page, baseURL }) => {
		await page.goto(`${baseURL}/my/collections/0`);
		await page.getByRole("link", { name: "folder closed test" }).click();
		await page.waitForURL(`${baseURL}/my/collections/46lfmlwhymhv6xl`);
	});
});

That’s it!

Appendix

• Used in the Bookmarkey Project

This will be a slightly shorter article. In this article I will show you how I’ve managed to do some end-to-end testing with Gitlab CI services.

I’m building a browser-based multiplayer game called Banter Bus. Banter Bus consists of three main components,

• gui: A SvelteKit based frontend the user will interact with to play the game
• core-api: A Socketio API written in Python
• management-api: A simple RESTful API written in Python (FastAPI)

Now say I want to write some e2e Cypress tests, that will test all of these components interacting with each other. Which mainly will look something like gui -> core-api -> management-api.

Each of these project deploys its own Docker container, which we can then use for testing it. So how can we do this with Gitlab CI ?

Gitlab Services

What is a Gitlab CI service ?

The services keyword defines a Docker image that runs during a job linked to the Docker image that the image keyword defines. This allows you to access the service image during build time. - https://docs.gitlab.com/ee/ci/services/

Essentially they are Docker containers we can use in our CI jobs.

package.json

For the examples below assume our package.json scipts section looks something like:

{
  "dev": "svelte-kit dev",
  "e2e": "cypress run --browser chrome",
  "e2e:ci": "start-server-and-test dev http://localhost:3000 e2e"
}

Gitlab CI

Let’s take a look at an example .gitlab-ci.yml file:

stages:
  - test

cypress-e2e-chrome:
  image: cypress/browsers:node14.17.0-chrome88-ff89
  stage: test
  variables:
	BANTER_BUS_CORE_API_MANAGEMENT_API_URL: http://banter-bus-management-api
	BANTER_BUS_CORE_API_DB_HOST: banter-bus-database
	FF_NETWORK_PER_BUILD: 1
	# Hidden the rest of the variables as not to clutter the file
  services:
    - name: mongo:4.4.4
      alias: banter-bus-database
    - name: registry.gitlab.com/banter-bus/banter-bus-core-api:test
      alias: banter-bus-core-api
    - name: registry.gitlab.com/banter-bus/banter-bus-management-api:test
      alias: banter-bus-management-api
    - name: registry.gitlab.com/banter-bus/banter-bus-management-api/database-seed:latest
      alias: banter-bus-database-seed
  script:
	- npm ci
    - export VITE_BANTER_BUS_CORE_API_URL=http://banter-bus-core-api:8080
    - echo fs.inotify.max_user_watches=524288 | tee -a /etc/sysctl.conf && sysctl -p
    - npm run e2e:ci
  artifacts:
    expire_in: 1 week
    when: always
    paths:
      - cypress/screenshots
      - cypress/videos
    reports:
      junit:
        - results/TEST-*.xml

Services

Let’s break file down a bit, these are essentially all the dependencies of our gui application. We need all of these containers running.

In this case we need four containers (this doesn’t really matter):

• banter-bus-database: A database for the core-api and management-api
• banter-bus-core-api: The main API the gui will interact with
• banter-bus-management-api: Used to help manage our available games, questions etc
• banter-bus-database-seed: A short lived container which pre-fills the database with some values

services:
  - name: mongo:4.4.4
    alias: banter-bus-database
  - name: registry.gitlab.com/banter-bus/banter-bus-core-api:test
    alias: banter-bus-core-api
  - name: registry.gitlab.com/banter-bus/banter-bus-management-api:test
    alias: banter-bus-management-api
  - name: registry.gitlab.com/banter-bus/banter-bus-management-api/database-seed:latest
    alias: banter-bus-database-seed

In our examples the name field is the image name, this is the same name you’d use when using the docker pull command. The next field is alias this is the name we’ll use to reference that container. This is the container name.

To see how the alias is used, is to look at the environment variables we have provided for the job BANTER_BUS_CORE_API_DB_HOST: banter-bus-database. So core-api will try to connect to database using this host. You can read more about Docker is able to resolve this to an IP address here. Another example is how the URL core-api will use to connect to the management-api BANTER_BUS_CORE_API_MANAGEMENT_API_URL: http://banter-bus-management-api.

We’ve discussed the most important part of the CI file, but lets quickly discuss the rest for completeness

Variables

We’ve already spoken about this above, but lets take a quick look at the variables section.

variables:
  BANTER_BUS_CORE_API_MANAGEMENT_API_URL: http://banter-bus-management-api
  BANTER_BUS_CORE_API_DB_HOST: banter-bus-database
  FF_NETWORK_PER_BUILD: 1
  # Hidden the rest of the variables as not to clutter the f

These are environment variables that are shared both with the job and the services. Some of these are config passed to the application, such as BANTER_BUS_CORE_API_MANAGEMENT_API_URL and BANTER_BUS_CORE_API_DB_HOST.

Script

Since we are using the cypress/browsers:node14.17.0-chrome88-ff89 image, we have access to chrome (headless) browser we can use with Cypress.

So we can do something like so:

script:
  - npm ci
  - export VITE_BANTER_BUS_CORE_API_URL=http://banter-bus-core-api:8080
  - npm run e2e:ci

• npm ci: Installs our npm dependencies for the gui app
• export VITE_BANTER_BUS_CORE_API_URL=http://banter-bus-core-api:8080 exports an enviroment variable which will be used by the gui app so it knows the URL of the core-api. Note the use of the alias name here (and port :8080 default port for the core-api)
• npm run e2e:ci: Starts the dev server and then runs the cypress test, see start-server-and-test dev http://localhost:3000 e2e where e2e is cypress run --browser chrome

Artifacts

Finally, the artifacts are “things” that are left over after the build. In this case we use them in two ways:

• One to generate a coverage report with junit
• Two to save our Cypress screenshots and videos

The downloadable artifacts will expire after 1 week. The Cypress files can be useful when debugging a problem with your tests. You get a video of perhaps why the tests failed.

artifacts:
  expire_in: 1 week
  when: always
  paths:
    - cypress/screenshots
    - cypress/videos
  reports:
    junit:
      - results/TEST-*.xml

Appendix

• Example Project
• Example Job

We will be using pytest as our testing framework.

main.py

import socketio
import uvicorn
from pydantic import BaseModel


async def startup():
    print("Starting Application")


sio = socketio.AsyncServer(async_mode="asgi")
app = socketio.ASGIApp(sio, on_startup=startup)


class FooEvent(BaseModel):
    name: str


@sio.on("FOO")
async def foo_event(sid, *args, **kwargs):
    data = FooEvent(**args[0])
    await sio.emit("BAR", {"foo": data.name})


if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8080)

Let’s take a look at our socketio app. Which is a very simple web app, that listens to one event FOO and responds with a BAR event. It is just this single file.

conftest.py

The conftest.py file is automatically run by pytest and allows our test modules to access fixtures defined in this file. One of the best features of Pytest is fixtures. Fixture are functions that have re-usable bits of code we can run in our tests, such as static data used by tests.

import asyncio
from typing import Any, AsyncIterator, Awaitable, List, Optional

import pytest
import socketio
import uvicorn
from app import main
from socketio import ASGIApp
from socketio.asyncio_client import AsyncClient

PORT = 8000
LISTENING_IF = "127.0.0.1"
BASE_URL = f"http://{LISTENING_IF}:{PORT}"


class UvicornTestServer(uvicorn.Server):
    def __init__(self, app: ASGIApp = main.app, host: str = LISTENING_IF, port: int = PORT):
        self._startup_done = asyncio.Event()
        self._serve_task: Optional[Awaitable[Any]] = None
        super().__init__(config=uvicorn.Config(app, host=host, port=port))

    async def startup(self) -> None:
        """Override uvicorn startup"""
        await super().startup()
        self.config.setup_event_loop()
        self._startup_done.set()

    async def start_up(self) -> None:
        """Start up server asynchronously"""
        self._serve_task = asyncio.create_task(self.serve())
        await self._startup_done.wait()

    async def tear_down(self) -> None:
        """Shut down server asynchronously"""
        self.should_exit = True
        if self._serve_task:
            await self._serve_task


@pytest.fixture(scope="session")
def event_loop():
    loop = asyncio.get_event_loop()
    yield loop
    loop.close()


@pytest.fixture(autouse=True, scope="session")
async def startup_and_shutdown_server():
    server = UvicornTestServer()
    await server.start_up()
    yield
    await server.tear_down()


@pytest.fixture(scope="session")
async def client() -> AsyncIterator[AsyncClient]:
    sio = socketio.AsyncClient()
    await sio.connect(BASE_URL)
    yield sio
    await sio.disconnect()

Quick Aside FastAPI Testing

Now when testing say a FastAPI application, it has a builtin test client we can use. This means we don’t actually have to spin up a Uvicorn server to test our application. We can simply pretend to send requests to the FastAPI web service and it will handle the routing behind the scenes.

We can do something like this, where httpx is a async HTTP client (think like the requests library).

import pytest
from asgi_lifespan import LifespanManager
from httpx import AsyncClient

from app.main import app



@pytest.fixture()
async def client() -> AsyncIterator[AsyncClient]:
    async with LifespanManager(app):
        async with AsyncClient(app=app, base_url="http://localhost") as client:
            yield client

Then we can use it like so in our tests:

from fastapi import status
from httpx import AsyncClient

@pytest.mark.asyncio
async def test_add_game(client: AsyncClient):
    response = await client.post("/game", json=request_data)
    assert response.status_code == status.HTTP_201_CREATED

However socketio at the moment does not provide us with a test client we can use. So we will start and stop a Uvicorn server and send actual Socketio requests from a Socketio client. There is a Socketio client library we can use to do this, available in Python.

class UvicornTestServer(uvicorn.Server):
    def __init__(self, app: ASGIApp = main.app, host: str = LISTENING_IF, port: int = PORT):
        self._startup_done = asyncio.Event()
        self._serve_task: Optional[Awaitable[Any]] = None
        super().__init__(config=uvicorn.Config(app, host=host, port=port))

    async def startup(self) -> None:
        """Override uvicorn startup"""
        await super().startup()
        self.config.setup_event_loop()
        self._startup_done.set()

    async def start_up(self) -> None:
        """Start up server asynchronously"""
        self._serve_task = asyncio.create_task(self.serve())
        await self._startup_done.wait()

    async def tear_down(self) -> None:
        """Shut down server asynchronously"""
        self.should_exit = True
        if self._serve_task:
            await self._serve_task

This is a test class which we can use to start and stop the Uvicorn server. Note that the class inherits from uvicorn.server, we need to overwrite the startup() method as we want to change the startup a bit.

Before explaining the code above let’s take a look at how we may use it:

@pytest.fixture(scope="session")
def event_loop():
    loop = asyncio.get_event_loop()
    yield loop
    loop.close()


@pytest.fixture(autouse=True, scope="session")
async def startup_and_shutdown_server():
    server = UvicornTestServer()
    await server.start_up()
    yield
    await server.tear_down()

@pytest.fixture(scope="session")
async def client() -> AsyncIterator[AsyncClient]:
    sio = socketio.AsyncClient()
    await sio.connect(BASE_URL)
    yield sio
    await sio.disconnect()

What we have done is created two pytest fixtures, the first simply starts an event loop so we can test async code.

Tangent on asyncio

To test async code with pytest we need to install the pyest-asyncio library. By default this will give us an event_loop fixture that runs on scope of function. So it will start and stop after each test function. However if you want to use fixtures that aren’t of scope function i.e. session or module. Then we need to redefine the event_loop function as we have done in the example above.

Okay back to our code above. The main bit we are interested in is the startup_and_shutdown_server function, here we start the server before all of our tests and due to how yield, you can read more about how yield works here, we will stop our server after all of our tests have run.

This happens automatically without calling the function because of the decorator we have provided @pytest.fixture(autouse=True, scope="session"). Again we are using scope session so that this function isn’t called either for every function (which would slow down our tests). We could’ve set it to module but again if we have multiple test files we don’t want to run this function for every file (module).

Deeper diver into UvicornTestServer

Let’s take a look at the first two methods

  def __init__(self, app: ASGIApp = main.app, host: str = LISTENING_IF, port: int = PORT):
      self._startup_done = asyncio.Event()
      self._serve_task: Optional[Awaitable[Any]] = None
      super().__init__(config=uvicorn.Config(app, host=host, port=port))

  async def startup(self) -> None:
      """Override uvicorn startup"""
      await super().startup()
      self.config.setup_event_loop()
      self._startup_done.set()

The __init__ magic dunder method creates an asyncio event asyncio.Event(). These events are often used to:

An asyncio event can be used to notify multiple asyncio tasks that some event has happened. - https://docs.python.org/3/library/asyncio-sync.html#asyncio.Event

Then we create a variable self._serve_task: Optional[Awaitable[Any]] = None, we will see how this used later. Finally we call the parent calls __init__ method (super().__init__()). This calls the __init__ function of the uvicorn.Server class. We do this to set the uvicorn.Config, which includes our app and which host and port to start the server.

Onto the second method startup this also overwrites a method in the parent class. In fact the first we do is call the parent class’s startup method (await super().startup()). Then we start the event loop ourselves self.config.setup_event_loop(), where our web app will run.

Finally we do self._startup_done.set(), we are setting this event as true i.e. is complete. So any coroutines waiting until this set can be carry on their execution.

Yet another tangent on run() method

Now the parent class does have a run method we could use, which would start the event loop for us. This however won’t work, lets pretend we change startup_and_shutdown_server function too look like this (server.run()).

@pytest.fixture(autouse=True, scope="session")
async def startup_and_shutdown_server():
    server = UvicornTestServer()
    await server.run()
    yield
    await server.tear_down()

We would get the following error RuntimeError: asyncio.run() cannot be called from a running event loop. This because if we take a look at the run method in the parent class it contains something like this line return asyncio.run(self.serve(...)).

This is why we need to write our own code to handle starting the Uvicorn server.

start_up and tear_down

Okay let’s move and take a look at the start_up and tear_down methods

async def start_up(self) -> None:
    self._serve_task = asyncio.create_task(self.serve())
    await self._startup_done.wait()

async def tear_down(self) -> None:
    self.should_exit = True
    if self._serve_task:
        await self._serve_task

Remember these are the two methods we will call in our “startup and shutdown” fixture. The start_up method, creates a task and assigns it to our empty variable from the __init__ method self._serve_task = asyncio.create_task(self.serve()). It calls the serve method to start the Uvicorn server.

Then we await self._startup_done.wait(), this is the event we created earlier. It will wait until the set() function has been called in the in the startup method above.

Now onto the tear_down method where we set the should_exit to true. There is a main_loop method called by our serve method in the parent class. This main_loop calls an on_tick function which returns if self.should_exit is true. So the call chain looks like: serve -> main_loop -> on_tick. When on_tick returns should_exist as true, it exits it main loop:

async def main_loop(self) -> None:
    counter = 0
    should_exit = await self.on_tick(counter)
    while not should_exit:
        counter += 1
        counter = counter % 864000
        await asyncio.sleep(0.1)
        should_exit = await self.on_tick(counter)

Client Fixture

Finally lets take a look at our final fixture, here we create a client that can be used to make requests with socketio. We use a similar technique with yields so we return a socketio client. We will see how this used in one of our tests.

@pytest.fixture(scope="session")
async def client() -> AsyncIterator[AsyncClient]:
    sio = socketio.AsyncClient()
    await sio.connect(BASE_URL)
    yield sio
    await sio.disconnect()

test_room.py

import asyncio

import pytest
from socketio.asyncio_client import AsyncClient


@pytest.mark.asyncio
async def test_success(client: AsyncClient):
    future = asyncio.get_running_loop().create_future()

    @client.on("BAR")
    def _(data):
        future.set_result(data)

    await client.emit("FOO", {"name": "haseeb"})
    await asyncio.wait_for(future, timeout=5.0)
    result = future.result()
    assert result == {"foo": "haseeb"}

Since we need to wait for the FOO event to return a BAR event we use a future to await until we get a response then set the return data in the future

@client.on("BAR")
def _(data):
    future.set_result(data)

We await asyncio.wait_for(future, timeout=5.0) for the future to have data set on it.

That’s it, the code itself is fairly simple once everything is setup in conftest to actually do the test.

Appendix

• Example source code
• Github Issue: UvicornTestServer
• Async Create Task SO
• Real application using this testing pattern

Introduction

Gatsby

Gatsby is a static site generator (SSG) built upon React. It allows us to create “blazing” fast websites. In this example, we will use a simple blog starter template available and add a Cypress test.

Cypress

Fast, easy and reliable testing for anything that runs in a browser. - Cypress README

Cypress allows us to test a web application, how a real user would use the application. Cypress will be used to test our Gatsby application, though if it’s a site you can test it using Cypress.

Gitlab CI

Gitlab CI is a continuous integration pipeline that will allow us to run our tests automatically, such as when we merge code into the master branch.

Getting Started

Gatsby

Create a new Gatsby site, using this default Gatsby starter:

gatsby new gatsby-starter-blog https://github.com/gatsbyjs/gatsby-starter-blog
cd gatsby-starter-blog

(Optional) Typescript

Adding Typescript to a Gatsby web application.

yarn add typescript @types/react @types/react-dom @types/node -D
yarn add gatsby-plugin-typescript

Add the following to your gatsby-config.js.

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-typescript`,
      options: {
        isTSX: true, // defaults to false
        jsxPragma: `jsx`, // defaults to "React"
        allExtensions: true, // defaults to false
      },
    },
  ],
}

Then create a new file tsconfig.json (in the project root, where the gatsby-config.js is).

{
  "compilerOptions": {
    "module": "commonjs",
    "target": "esnext",
    "jsx": "preserve",
    "lib": ["dom", "esnext"],
    "strict": true,
    "noEmit": true,
    "isolatedModules": true,
    "esModuleInterop": true,
    "noUnusedLocals": false,
    "allowJs": true
  },
  "exclude": ["node_modules", "public", ".cache"]
}

Cypress

Now to finally add Cypress to our application so we can test it. First, install the dependencies.

yarn add -D cypress cypress-axe axe-core start-server-and-test
# Add types
yarn add -D @types/cypress-axe

Next, let’s create a cypress.json folder in the project root.

{
  "baseUrl": "http://localhost:8000/",
   "integrationFolder": "cypress/e2e"
}

Next, let’s add some new “scripts” to the package.json file.

  "scripts": {
    "cy:open": "cypress open",
    "cy:run": "cypress run",
    "build": "gatsby build",
    "develop": "gatsby develop",
    "format": "prettier --write \"**/*.{js,jsx,ts,tsx,json,md}\"",
    "start": "npm run develop",
    "serve": "gatsby serve",
    "clean": "gatsby clean",
    "test": "echo \"Write tests! -> https://gatsby.dev/unit-testing\" && exit 1",
    "test:e2e": "start-server-and-test 'yarn develop' http://localhost:8000 'yarn cy:open'",
    "test:e2e:ci": "start-server-and-test 'yarn develop' http://localhost:8000 'yarn cy:run'"
  }

These scripts allow us to start Cypress, cy:open opens a GUI to visualise our tests whereas cy:run does it all in the terminal (the browser runs in headless mode). Where we will run test:e2e:ci in our CI pipeline, here we use the start-server-and-test command to start our Gatsby server using yarn develop. Then we run cy:run to start our tests.

Structure

Create a new folder called cypress which will look something like this.

.
├── e2e
│   └── accessibility.test.ts
├── fixtures
│   └── graphql.json
├── plugins
│   └── index.js
├── support
│   ├── commands.js
│   ├── index.d.ts
│   └── index.js
└── tsconfig.json

mkdir -p cypress/support

Create a file at cypress/support/commands.js

Cypress.Commands.add(`assertRoute`, (route) => {
  cy.url().should(`equal`, `${window.location.origin}${route}`);
});

Add some custom types for Cypress index.d.ts if you are using Typescript.

/// <reference types="cypress" />

declare namespace Cypress {
  interface Chainable<Subject> {
    /**
     * Assert the current URL
     * @param route
     * @example cy.assertRoute('/page-2')
     */
    assertRoute(route: string): Chainable<any>;

    /**
     * Waits for Gatsby to finish the route change, in order to ensure event handlers are properly setup
     */
    waitForRouteChange(): Chainable<any>;
  }
}

Next create the index.js file, which should look something like this.

import "cypress-axe"
import "./commands"

Next, let’s create a plugin folder mkdir -p cypress/plugins.

// / <reference types="cypress" />
// ***********************************************************
// This example plugins/index.js can be used to load plugins
//
// You can change the location of this file or turn off loading
// the plugins file with the 'pluginsFile' configuration option.
//
// You can read more here:
// https://on.cypress.io/plugins-guide
// ***********************************************************

// This function is called when a project is opened or re-opened (e.g. due to
// the project's config changing)

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config
  return config
}

Now finally let’s create our tests folder mkdir -p cypress/e2e.

cypress-axe

In this blog post we won’t go over any complicated Cypress test we will simply use cypress-axe to test the accessibility of our (a11y) of our website.

/// <reference types="../support/index" />
/// <reference types="cypress" />
/// <reference types="@types/cypress-axe" />

describe("Component accessibility test", () => {
  it("Main Page", () => {
    cy.visit("/")
    cy.wait(500)
    cy.injectAxe()
    cy.checkA11y({
      include: [["#___gatsby"]],
    })
  })
})

Note the /// comments at the top used to add types for cypress. The test above will go to our home page and test if it has any a11y violations and if so will fail the test.

We can now run our tests locally by running this command:

yarn run test:e2e

Gitlab CI

Now how can we automate this, so the tests will run say every time we make changes on the master branch to make sure we haven’t broken any a11y. Create a new .gitlab-ci.yml or add the following job to an existing CI file.

image: node:12.14.1

variables:
  CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/cache/Cypress"

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - cache/Cypress
    - node_modules

stages:
  - test

before_script:
  - yarn install

tests:
  image: cypress/browsers:node12.14.1-chrome83-ff77
  stage: test
  script:
    - yarn test:e2e:ci

I won’t go into the details of what makes up a Gitlab CI file. At the top of the file, we will cache the node_modules file so we can share it between the job and the Cypress cache. The job itself is very simple it uses a cypress/browsers:node12.14.1-chrome83-ff77 Docker image which provides a headless chrome browser that Cypress can leverage to run the tests. As we won’t have access to a GUI in the Gitlab CI runner. The tests job is very simple it runs yarn test:e2e:ci to run our Cypress tests.

That’s it, quite simple to add Cypress tests that run in our CI pipeline.

Appendix

• Example source code
• Cypress
• Example Job

• Implementing a Simple REST API using OpenAPI, Flask & Connexions
• Testing with pytest-mock and pytest-flask

The example app we will be writing tests for is a very simple CRUD API managing a pet store. It allows us to add pets, remove pets, update pets and query pets we have in the store.

Structure

You can find the source code here. Our project structure looks like this:

.
├── openapi
│   └── specification.yml
├── requirements.txt
├── test_api
│   ├── core
│   │   ├── __init__.py
│   │   ├── pets.json
│   │   └── pets.py
│   ├── __init__.py
│   ├── run.py
│   └── web
│       ├── controllers
│       │   ├── __init__.py
│       │   └── pets_controller.py
│       ├── encoder.py
│       ├── __init__.py
│       ├── models
│       │   ├── base_model_.py
│       │   ├── __init__.py
│       │   ├── pet.py
│       │   └── pets.py
│       └── util.py
└── tests
    ├── conftest.py
    ├── __init__.py
    └── test_pets_controller.py

API

Here is our controller module called web/controller/pets_controller.py. This is where connexion routes are requests to:

import connexion
import six

from ..models.pet import Pet  # noqa: E501
from ..models.pets import Pets  # noqa: E501
from .. import util

from test_api.core import pets


def get_pet(pet_id):  # noqa: E501
    """Get a pet in the store

     # noqa: E501

    :param pet_id: The id of the pet to retrieve
    :type pet_id: str

    :rtype: Pet
    """
    try:
        pet = pets.get_pet(pet_id)
        response = Pet(id=pet.id, breed=pet.breed, name=pet.name, price=pet.price), 200
    except KeyError:
        response = {}, 404

    return response

Connexion uses the open API specification openapi/specification.yml, to work out which function to route requests for the path /pet/{pet_id}. It uses the operationId alongside the x-swagger-router-controller to determine the function to call in the pets_controller.py module.

/pet/{pet_id}:
  get:
    tags:
      - "pet"
    summary: "Get a pet in the store"
    operationId: "get_pet"
    parameters:
      - name: "pet_id"
        in: "path"
        description: "The id of the pet to retrieve"
        required: true
        type: "string"
    responses:
      200:
        description: "Successfully retrived pet"
        schema:
          $ref: "#/definitions/Pet"
      404:
        description: "Pet doesn't exist"
    x-swagger-router-controller: "test_api.web.controllers.pets_controller"

Tests

Now onto our tests!

Libraries

pytest-flask allows us to specify an app fixture and then send API requests with this app. Usage is similar to the requests library when sending HTTP requests to our app.

pytest-mock is a simple wrapper around the unit test mock library, so anything you can do using unittest.mock you can do with pytest-mock. The main difference in usage is you can access it using a fixture mocker, also the mock ends at the end of the test. Whereas with the normal mock library if you say mock the open() function, it will be mocked for the remaining duration of that test module, i.e. it will affect other tests.

conftest.py

The conftest.py file is automatically run by pytest and allows our test modules to access fixtures defined in this file. One of the best features of Pytest is fixtures. Fixture are functions that have re-usable bits of code we can run in our unit tests, such as static data used by tests.

import os
import json

import pytest

from test_api.run import create_app


@pytest.fixture(scope="session")
def app():
    abs_file_path = os.path.abspath(os.path.dirname(__file__))
    openapi_path = os.path.join(abs_file_path, "../", "openapi")
    os.environ["SPEC_PATH"] = openapi_path

    app = create_app()
    return app


@pytest.fixture(scope="session", autouse=True)
def clean_up():
    yield
    default_pets = {
        "1": {"name": "ginger", "breed": "bengal", "price": 100},
        "2": {"name": "sam", "breed": "husky", "price": 10},
        "3": {"name": "guido", "breed": "python", "price": 518},
    }

    abs_file_path = os.path.abspath(os.path.dirname(__file__))
    json_path = os.path.join(abs_file_path, "../", "test_api", "core", "pets.json")
    with open(json_path, "w") as pet_store:
        json.dump(default_pets, pet_store, indent=4)

app()

In this file, we have two functions: the app allows users to pass the client argument to other tests and then we can test our web application. You can get more information here about how Flask apps can be tested. Essentially we don’t need to start/stop a server before/after our tests.

By giving it the scope=session the fixture will be created once before all of our tests run. Our run.py file looks like this:

import os

import connexion

from .web import encoder


def create_app():
    if "SPEC_PATH" in os.environ:
        openapi_path = os.environ["SPEC_PATH"]
    else:
        abs_file_path = os.path.abspath(os.path.dirname(__file__))
        openapi_path = os.path.join(abs_file_path, "../", "../", "openapi")
    app = connexion.FlaskApp(
        __name__,
        specification_dir=openapi_path,
        options={"swagger_ui": False, "serve_spec": False},
    )
    app.add_api("specification.yml", strict_validation=True)
    flask_app = app.app
    flask_app.json_encoder = encoder.JSONEncoder

    return flask_app

The create_app function creates our web application and returns a Flask object. Remember the Connexion library is just a wrapper around Flask. Connexion just reduces the boilerplate code we wrote. Again you can have a read of the article above to get more details about how it works.

clean_up()

@pytest.fixture(scope="session", autouse=True)
def clean_up():
    yield
    default_pets = {
        "1": {"name": "ginger", "breed": "bengal", "price": 100},
        "2": {"name": "sam", "breed": "husky", "price": 10},
        "3": {"name": "guido", "breed": "python", "price": 518},
    }

    abs_file_path = os.path.abspath(os.path.dirname(__file__))
    json_path = os.path.join(abs_file_path, "../", "test_api", "core", "pets.json")
    with open(json_path, "w") as pet_store:
        json.dump(default_pets, pet_store, indent=4)

The second fixture we define is called clean_up, because of the yield line, this function will run after all of our tests have completed. The yield command is related to generators, you can read more here. In our case, it’s used in Pytest fixtures so that we can run some cleanup jobs after our test is completed. In this example, I am simply replacing the contents of the JSON file which acts as a data store (like a database), to its default values before the test was run.

Since pytest-3.0, fixtures using the normal fixture decorator can use a yield statement to provide fixture values and execute teardown code - Pytest Docs

test_pets_controller.py

Now we have gone over the setup required for our tests, let’s take a look at how we can test our code. So our first test looks like:

def test_get_all_pets(client):
    url = "/api/v1/pet"
    expected_json = [
        {"id": "1", "name": "ginger", "breed": "bengal", "price": 100},
        {"id": "2", "name": "sam", "breed": "husky", "price": 10},
        {"id": "3", "name": "guido", "breed": "python", "price": 518},
    ]
    response = client.get(url)
    assert response.json == expected_json

It’s a very simple test, here we use the app fixture we defined above. This client fixture can be used because we are using the pytest-flask library. As you can see it looks very similar to requests, where we give it a path /API/v1/pet and then tell it what kind of request to make client.get. Whilst the syntax between the requests library and the client fixture is almost identical. One big difference that always seems to trip me up is, in requests to get the JSON data from the response object would be response.json() i.e it is a function. However in client (pytest-flask) fixture do get the JSON data we do response.json which is just an attribute of the object not a function.

The test itself is very simple, it’s making a request to get all pets in the pet store. We then compare that with= what we expect to be in the pet store assert response.json == expected_json.

The next test we have looks like this:

@pytest.mark.parametrize(
    "pet_data, expected_status, expected_data",
    [
        ({"name": "Yolo", "breed": "shorthair", "price": 100}, 201, {"id": 4}),
        ({}, 400, {}),
        ({"a": "b"}, 400, {}),

    ]
)
def test_add_a_pet(client, pet_data, expected_status, expected_data):
    url = "/api/v1/pet"
    response = client.post(url, json=pet_data)
    assert response.status_code == expected_status
    if response.status_code == 200:
        assert response.json == expected_data

This test is attempting to add a new pet to the store. It’s similar to the other test we still use the client fixture to make the request. This time we also give it some json data hence we provide the json argument json=pet_data this automatically sets the headers correctly so the server knows it’s receiving JSON data.

We also use a decorate called @pytest.mark.parametrize. This allows us to run our tests against a list of data. So we don’t have to write the same test x number of times. We just pass the test different arguments. Pytest will run this test x number of times once for each item in the list. So, for example, the first time the test runs:

pet_data = {"name": "Yolo", "breed": "shorthair", "price": 100}
expected_status = 200
expected_data = {"id": 4}

The second like this:

pet_data = {}
expected_status = 200
expected_data = {}

And so on and so on. This helps keep our test file smaller and keeps the DRY (do not repeat yourself). A very nice feature of Pytest and one I use heavily.

The final test we have in this file looks like:

def test_add_pet_fail_json(client, mocker):
    pet_data = {"name": "Yolo", "breed": "shorthair", "price": 100}
    url = "/api/v1/pet"
    mock = mocker.patch("connexion.request")
    mock.is_json = False
    response = client.post(url, json=pet_data)
    assert response.status_code == 400

At last, we see pytest-mock being used via the mocker fixture we automatically get access to. The mocker is just a simple wrapper around the unittest.mock module. The main difference being the mock on exists for the duration of that test. Mocking is often used when unit testing and we cannot rely on external dependencies such as database connections or another web service.

def add_pet(body):  # noqa: E501
    # ...
    if connexion.request.is_json:
        body = Pet.from_dict(connexion.request.get_json())  # noqa: E501
    # ...

In this example, we want to mock the part of connexion that checks if the data being sent is valid JSON. We want the connexion.request.is_json to return False, we can do this like so:

mock = mocker.patch("connexion.request")
mock.is_json = False

Since is_json is an attribute of the connexion.request module and not a function we need to set it false on another line. If is_json was a function that we wanted to return False we could’ve done mocker.patch("connexion.request.is_json") instead.

You can run the tests locally by running the pytest command or if you want to run the code in this article, you can by doing the following:

gcl https://gitlab.com/hmajid2301/articles.git
cd 27.\ Mocking\ in\ Flask\ with\ Pytest/source_code
virtualenv .venv
source .venv/bin/activate
pip install -r requirements.txt
pytest

That’s it, the examples above cover most of the things you’ll need to mock and test your connexion web service.

INFO: pytest-flask provides a whole bunch of other features that may be useful, you can find the full list here

Appendix

• Example source code

In this example, we will be testing a very simple Flask app I created source code here. If you want to learn more about the Flask app you can read my previous article here :plug: :plug:. All you really need to know for this article is that it involves using a very simple RESTful API for an imaginary cat store. Essentially all the API does is it interacts with a database, to get current cats, add new cats, edit already existing cats and remove cats from the store.

Prerequisites

• Your own Flask app
• Install the following dependencies, using pip install -r requirements.txt (or pip3 instead of pip)

Where requirements.txt is:

attrs==19.3.0
click==7.1.2
Flask==1.1.2
Flask-SQLAlchemy==2.4.3
itsdangerous==1.1.0
Jinja2==2.11.2
MarkupSafe==1.1.1
more-itertools==8.4.0
packaging==20.4
pluggy==0.13.1
psycopg2==2.8.5
py==1.8.2
pyparsing==2.4.7
pytest==5.4.3
pytest-flask==1.0.0
pytest-mock==3.1.1
six==1.15.0
SQLAlchemy==1.3.17
wcwidth==0.2.4
Werkzeug==1.0.1

Libraries

pytest-flask: Allows us to specify an app fixture and then send API requests with this app. Usage is similar to requests library, when sending HTTP requests to our flask app.

pytest-mock: Is a simple wrapper around the unittest mock library, so anything you can do using unittest.mock you can do with pytest-mock. The main difference in usage is you can access it using a fixture mocker, also the mock ends at the end of the test. Whereas with the normal mock library if you mock say the open() function, it will be mocked for the remaining duration of that test module, i.e. it will effect other tests.

pytest-flask example

import pytest

from example_app import create_app


@pytest.fixture
def app():
    app = create_app()
    return app


def test_example(client):
    response = client.get("/")
    assert response.status_code == 200

To use pytest-flask we need to create a fixture called app() which creates our Flask server. We can then use this fixture by passing client as an argument to any test. Then we can send various http requests using client.

Above is a very simple example using pytest-flask, we send a GET request to our app, which should return all cats in the database. We then check that the status code returned from the server was a 200 (OK). This is great except how can we mock out certain features within our code?

pytest-mock

We can mock out certain parts of our code using the pytest-mock library, but we have to mock inside the app() fixture. Since the rest of our tests will just be making HTTP requests to our Flask server. In this example say we don’t want to mock a connection to the database, we can use the following lines of code.

mocker.patch("flask_sqlalchemy.SQLAlchemy.init_app", return_value=True)
mocker.patch("flask_sqlalchemy.SQLAlchemy.create_all", return_value=True)
mocker.patch("example.database.get_all", return_value={})

What this bit of code is doing is any time the any of the mocked functions are called, say init_app we mock it so it will always return true. We have to give it the “full path” to the function, it’s the same as if you had to import the function itself.

import pytest

import example.app


@pytest.fixture
def app(mocker):
    mocker.patch("flask_sqlalchemy.SQLAlchemy.init_app", return_value=True)
    mocker.patch("flask_sqlalchemy.SQLAlchemy.create_all", return_value=True)
    mocker.patch("example.database.get_all", return_value={})
    return example.app.app


def test_example(client):
    response = client.get("/")
    assert response.status_code == 200

In this example it’s very boring as when we send an HTTP GET request to the app it will not interact with the database since we’ve mocked this out but instead just return an empty dict ({}). In reality, this is not a very good test, you would make it a bit more interesting.

virtualenv .venv
source .venv/bin/activate
pip install -e .
pip install -r requirements.txt
export $(xargs < database.conf)
pytest

Appendix

• Example source code

So I ended up creating a simplified mocked version in Python. This way I could run automated tests on my code without needing to either install/rely on this server already existing there. Essentially I ended up creating a TCP server in Python, which would receive some input and respond how I would expect the real TCP server too.

I write my tests using Pytest for several reasons, I prefer it to unittest because it required less boilerplate code. It also has lots of useful features such as fixtures. However, when creating tests for my API I realised that I had to first start the fake TCP server. But since it was always listening 👂 for traffic 🚦 it would block the rest of my tests from running. So in this example, I will show you how you can create a Pytest fixture and run it on a background thread (TCP server) and still run your tests as normal.

Code

import socket


class TCPServer:
    def __init__(self):
            self._sock = socket.socket(socket.AF_INET, 
                                       socket.SOCK_STREAM)
            self._sock.setsockopt(socket.SOL_SOCKET, 
                                  socket.SO_REUSEADDR, 1)
            
    def __enter__(self):
            self._sock.bind(('127.0.0.1', 9500))
            return self
            
    def __exit__(self, exception_type, exception_value, traceback):
            self._sock.close()
            
    def listen_for_traffic(self):
            while True:
                self._sock.listen(5)
                connection, address = self._sock.accept()
                message = connection.recv(1024)
                response = "Received"
                connection.send(response.encode())
                connection.close()

This is a very basic TCP server, it received some input but always replies with “Received”. Also, I created the class to be used with the context manager so that the socket can be closed, hence the __enter__() and __exit__() dunder (magic) method. This TCP server will always listen on localhost on port 9500, this is the same port the client will also have to connect to, to send messages to this TCP server.

The main function of interest is listen_for_traffic(), which loops forever whilst listening for traffic. If it receives any data, it always replies with “Received”, of course with a real server we might want to do something more interesting but to keep this example simple that’s all this server will do.

import socket
import threading

import pytest

from .tcp_server import TCPServer


@pytest.fixture(autouse=True)
def dummy_tcp_server():
    tcp_server = TCPServer()
    with example_server as tcp_server:
        thread = threading.Thread(target=example_server.listen_for_traffic)
        thread.daemon = True
        thread.start()
        yield example_server


def test_example():
    HOST = '127.0.0.1'
    PORT = 9500

    data = ""
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect((HOST, PORT))
        s.sendall(b'Hello, world')
        data = s.recv(1024)

    assert data.decode() == "Received"

When you run the Pytest command it will automatically run tests on all modules and method that start with “test_”. I have created a fixture (using the fixture decorator), fixtures allow for code reuse within a Pytest module. Fixtures are typically used to connect to databases, fixtures are the run before any tests hence we can also use them to setup is code.

So taking a look at the dummy_tcp_server() function, first we use a context manager (with) to set up our TCP server.

with tcp_server as example_server:

This calls the __enter__() dunder method, which starts up our TCP server and so it’s now listening for any traffic on port 9500.

thread = threading.Thread(target=example_server.listen_for_traffic)
thread.daemon = True
thread.start()

This is the bit of code that actually runs our TCP server on another (background) thread. So the in the first line we tell the thread what function to call, listen_for_traffic (without brackets) has an infinite while loops just waiting to listen for traffic arriving on port 9500. We set the thread as a daemon so that when we close the main Python program the thread will automatically kill itself, this means we don’t have to manage it yourself.

yield example_server

The yield is very similar to a return statement. The main reason I choose to include is that if the tests for some reason raise an exception, the yield will still allow the context manager to exit. If we had any tear down code after the yield even with an exception, this code will still run after all the tests have finished. Also yields (and return statements) allow individual tests to access functions from the TCPServer class. For example, if you had a function (in the TCPServer class) that stored every message sent to the server. If you pass the fixture as a parameter to any tests then you could access this function.

def test_example():
    HOST = '127.0.0.1'
    PORT = 9500

    data = ""
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect((HOST, PORT))
        s.sendall(b'Hello World')
        data = s.recv(1024)

    assert data.decode() == "Received"

The part of our code is the actual test. It sends “Hello World” to the TCP server, remember if you sent it as string (without b’’) then you’ll have to encode that string first (“Hello World”.encode()). Then we capture the response from the server and decode it (because it’s in binary, this converts it into a string) and assert or check that the data is the string “Received” as we expect..

Appendix

• Example source code
• Pytest