ZOE

• I work for ZOE 🥑
  • https://joinzoe.com
  • Personalised nutrition product
  • Health study

My Blood Sugar Levels

Who Is This Talk For?

• Have used Docker
  • But not an expert
• Know basic CLI commands
• Want to use Docker in CI

Example Code

• Simple FastAPI web-service
  • Interacts with DB
• It allows us to get and add new users
• Poetry for dependency management

Why Docker?

• Reproducible builds
  • Easy setup for developers
  • OS independent

My First Image

# Dockerfile

FROM python:3.9.8

ENV PYTHONUNBUFFERED=1 \
	PYTHONDONTWRITEBYTECODE=1 \
	PYTHONPATH="/app" \
	PIP_NO_CACHE_DIR=off \
	PIP_DISABLE_PIP_VERSION_CHECK=on \
	PIP_DEFAULT_TIMEOUT=100 \
	\
	POETRY_VERSION=1.1.11 \
	POETRY_HOME="/opt/poetry" \
	POETRY_VIRTUALENVS_IN_PROJECT=true \
	PYSETUP_PATH="/opt/pysetup" \
	POETRY_NO_INTERACTION=1 \
	\
	VENV_PATH="/opt/pysetup/.venv"

ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"

WORKDIR $PYSETUP_PATH
COPY pyproject.toml poetry.lock ./

RUN pip install poetry==$POETRY_VERSION && \
	poetry install

WORKDIR /app
COPY . .

CMD [ "bash", "/app/start.sh" ]

Let’s Run It

docker build --tag app .
docker run --publish 80:80 app

# Access app on http://localhost

App Dependencies

• App depends on a database
  • Dockerise it

Without Docker

sudo apt update
sudo apt install postgresql postgresql-contrib
sudo systemctl start postgresql.service

sudo -u postgres createuser --interactive
sudo -u postgres createdb test

With Docker

  docker run --volume "postgres_data:/var/lib/postgresql/data" \
  --environment "POSTGRES_DATABASE=postgres" \
  --environment "POSTGRES_PASSWORD=postgres" \
  --publish "5432:5432" \
    postgres:13.4

  
  # Start Commands:
  docker network create --driver bridge workspace_network
  docker volume create  postgres_data
  docker build -t app .
  docker run --environment "POSTGRES_USER=postgres" \
    --environment "POSTGRES_HOST=postgres" \
    --environment "POSTGRES_DATABASE=postgres" \
    --environment "POSTGRES_PASSWORD=postgres" \
    --environment "POSTGRES_PORT=5432" \
    --volume "./:/app" --publish "80:8080" \
    --network workspace_network --name workspace_app \
    --detach app
  docker run --volume "postgres_data:/var/lib/postgresql/data" \
  --environment "POSTGRES_DATABASE=postgres" \
  --environment "POSTGRES_PASSWORD=postgres" \
  --publish "5432:5432" --network workspace_network \
  --name workspace_postgres --detach postgres:13.4

  # Delete Commands:
  docker stop workspace_app
  docker rm workspace_app
  docker stop workspace_postgres
  docker rm workspace_postgres
  docker network rm workspace_network
  

Docker Compose

• Manage multiple Docker containers
• Existing tool docker-compose
  • V2 called docker compose
• Use docker compose today

# docker-compose.yml

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    command: bash /app/start.sh --reload
    volumes:
      - ./:/app
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_HOST=postgres
      - POSTGRES_DATABASE=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_PORT=5432
    ports:
      - 127.0.0.1:80:80

  postgres:
    image: postgres:13.4
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DATABASE=postgres
      - POSTGRES_PASSWORD=postgres
    ports:
      - 127.0.0.1:5432:5432

Run It!!!

docker compose up --build

docker compose down

Summary

• Dockerise your app
• Dockerise dependencies (DB)
• Use docker compose
  • Manage multiple containers

Running Tests

• Run tests in Docker
  • pytest runner
• Consistent environment

docker compose run app pytest

# docker-compose.yml

services:
 app:
    build:
      context: .
      dockerfile: Dockerfile
    depends_on:
      - postgres
    # ...

  postgres:
	# ...

CI Pipeline

• Docker running locally
• Can we use Docker in CI?

Before

# .github/workflows/branch.yml

name: Check changes on branch

on:
  push:
    branches:
      - "*"
      - "!main"

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:13.4
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python 3.9
        uses: actions/setup-python@v3
        with:
          python-version: 3.9
      - name: Install dependencies
        run: |
          pip install poetry=1.11.0
          poetry install
      - name: Test with pytest
        run: |
        export DB_USERNAME=postgres
        export DB_PASSWORD=postgres
        export DB_HOST=postgres
        export DB_PORT=5432
        export DB_NAME=postgres
        pytest

After

# .github/workflows/branch.yml

name: Check changes on branch

#...

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v3
      - name: Run Tests
        run: docker compose run app pytest

Summary

• Dockerise development tasks
  • Tests
  • Linting
  • DB migrations
• Use Docker on CI
  • Local environment = CI enviromnent

Smaller Image

• Remove redundant dependencies
  • Fewer security vectors
• Less storage

# Dockerfile

FROM python:3.9.8-slim

# ...

WORKDIR $PYSETUP_PATH
COPY pyproject.toml poetry.lock ./

RUN pip install poetry==$POETRY_VERSION && \
	poetry install

WORKDIR /app
COPY . .

CMD [ "bash", "/app/start.sh" ]

Comparison

Summary

• Aim to use smaller base images
• Remove unnecessary dependencies
• Reduce build time

Dependencies

• Dev dependencies in Docker image
  • Don’t need pytest in prod

Multistage Builds

# Dockerfile

FROM python:3.9.8-slim as base

ARG PYSETUP_PATH
ENV PYTHONPATH="/app"
ENV PIP_NO_CACHE_DIR=off \
	PIP_DISABLE_PIP_VERSION_CHECK=on \
	PIP_DEFAULT_TIMEOUT=100 \
	\
	POETRY_VERSION=1.1.11 \
	POETRY_HOME="/opt/poetry" \
	POETRY_VIRTUALENVS_IN_PROJECT=true \
	PYSETUP_PATH="/opt/pysetup" \
	POETRY_NO_INTERACTION=1 \
	\
	VENV_PATH="/opt/pysetup/.venv"

ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"


FROM base as builder

RUN pip install poetry==$POETRY_VERSION

WORKDIR $PYSETUP_PATH
COPY poetry.lock pyproject.toml ./

RUN poetry install --no-dev


FROM builder as development

RUN poetry install

WORKDIR /app
COPY . .

EXPOSE 80
CMD ["bash", "/app/start.sh", "--reload"]


FROM base as production

COPY --from=builder $VENV_PATH $VENV_PATH

WORKDIR /app
COPY . .

EXPOSE 80
CMD ["bash", "/app/start.sh"]

# docker-compose.yml

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      target: development
    command: bash /app/start.sh --reload
    depends_on:
      - postgres
    environment:
      - # ...
    volumes:
      - ./:/app
    ports:
      - 127.0.0.1:80:80

Comparison

Cache From

# docker-compose.yml

services:
  app:
    build:
      context: .
      target: development
      cache_from:
        - registry.gitlab.com/haseeb-slides/developing-with-docker-slides/python-image:latest
    command: bash /app/start.sh --reload
    # ....

Private Deps

• Private git repository
• Inject an SSH key
  • At build time

poetry add [email protected]:zoe/pubsub.git

  [tool.poetry.dependencies]
  python = "^3.9"
  fastapi = "^0.70.0"
  pubsub = { git = "ssh://[email protected]:zoe/pubsub.git",
              rev = "0.2.5" }
  psycopg2-binary = "^2.9.3"
  SQLAlchemy = "^1.4.36"
  uvicorn = "^0.17.6"

FROM base as builder

RUN apt-get update && \
    apt-get install openssh-client git -y && \
    mkdir -p -m 0600 \
    ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts && \
    pip install poetry==$POETRY_VERSION

WORKDIR $PYSETUP_PATH
COPY poetry.lock pyproject.toml ./

RUN --mount=type=ssh poetry install --no-dev

First add our ssh key

ssh-add ~/.ssh/id_rsa

Then we can do

docker compose build --ssh default

CI Changes

  
# .github/workflows/branch.yml

jobs:
  # ...
  test:
    # ...
    steps:
      - uses: actions/checkout@v3
      - uses: webfactory/[email protected]
        with:
          ssh-private-key: ${{ secrets.PRIVATE_SSH_KEY }}
      - name: Build Image
        run: docker compose build --ssh default
      - name: Run Tests
        run: docker compose run app pytest
  

Comparison

Summary

• Use multistage builds
  • Slimmer production images
• Leverage SSH injection
  • During build time

Any Questions?

• Code: https://gitlab.com/hmajid2301/talks/docker-as-a-dev-tool
• Slides: https://docker-as-a-dev-tool.haseebmajid.dev/

Extra Reading

• Breaking Down Docker by Nawaz Siddiqui
• Announcing Compose V2 General Availability
• Caching Docker layers on serverless build hosts with multi-stage builds
• Using Alpine can make Python Docker builds 50× slower

Useful Tools

• Dive
• Anchore image scan

Appendix

• Container meme from
• Memes generated at
• Hand illustration
• People Hands Up Vectors by Vecteezy
• FlatIcon
  • Box icons created by Freepik - Flaticon
  • Arrow icons created by rang - Flaticon
  • One icons created by Hight Quality Icons - Flaticon
  • Two icons created by Hight Quality Icons - Flaticon