From Coffee Break to Instant Feedback: Rapid Integration Testing in Go

Introduction

Who is this for?

  • Go developers with slow test suites
  • Teams using PostgreSQL for integration tests
  • Anyone tired of waiting for CI

The Pain of Slow Integration Tests

“I pushed to CI and went to make coffee… then forgot what I was working on”

The Real Cost

  • Lost focus and context switching
  • Slower feedback loops
  • Reduced confidence in changes
  • Delayed deployments

Impact on CI/CD

  • Long CI times block other PRs
  • Developers batch changes (riskier)
  • “Skip tests” commits appear
  • Deployment pipeline bottleneck

How to Profile Our Tests

go test -json ./... | go-test-trace

Test Timing

go test -v ./... 2>&1 | grep -E '(PASS|FAIL).*\('

Finding the Slowest

go test ./... -json | \
  jq -r 'select(.Action == "pass") |
  [.Elapsed, .Package] | @tsv' | \
  sort -rn | head -20

Visual Test Explorer

go install github.com/lamarios/vgt@latest

vgt

Why Are Database Tests So Slow?

  • Running migrations every test
  • Seeding test data repeatedly
  • Teardown and cleanup
  • Sequential execution

Go’s Built-in Tools

Two main levers:

  • t.Parallel() - parallel tests within package
  • -p flag - parallel package execution

t.Parallel()

func TestUserCreation(t *testing.T) {
    t.Parallel()

    // Test runs concurrently with other parallel tests
}

Understanding -p Flag

# Run 4 packages concurrently
go test -p 4 ./...

# Check what -p would do without running
go test -p 4 -n ./...

Combining Both

go test -p 4 -parallel 8 ./...
  • -p 4: 4 packages at once
  • -parallel 8: 8 tests per package

What Are Template Databases?

PostgreSQL feature: clone databases instantly

CREATE DATABASE testdb
TEMPLATE template1;

The Template Pattern

  1. Run migrations once on template
  2. Clone template for each test
  3. Drop test database after test
  4. Repeat

Performance Difference

Traditional approach:

Test 1: 100ms migrations + 50ms test
Test 2: 100ms migrations + 50ms test
Total: 300ms

Template approach:

Setup: 100ms migrations (once)
Test 1: 10ms clone + 50ms test
Test 2: 10ms clone + 50ms test
Total: 220ms

Live Demo

Setting up template databases in Go

Creating Template Database

var initDatabaseOnce sync.Once

func setupTemplate(t *testing.T) {
    initDatabaseOnce.Do(func() {
        db := connectToPostgres()
        defer db.Close()

        // Run migrations on template1
        runMigrations(db, "template1")
    })
}

Creating Test Database

func createTestDB(t *testing.T) *sql.DB {
    setupTemplate(t)

    dbName := fmt.Sprintf("test_%s_%d",
        t.Name(),
        time.Now().UnixNano(),
    )

    // Terminate existing connections
    terminateConnections(dbName)

    // Create from template
    db := connectToPostgres()
    db.Exec(fmt.Sprintf(
        "CREATE DATABASE %s TEMPLATE template1",
        dbName,
    ))

    return connectToTestDB(dbName)
}

Cleanup

func TestUser(t *testing.T) {
    t.Parallel()

    db := createTestDB(t)
    t.Cleanup(func() {
        dbName := db.Stats().Name
        db.Close()

        admin := connectToPostgres()
        defer admin.Close()

        terminateConnections(dbName)
        admin.Exec(fmt.Sprintf(
            "DROP DATABASE %s",
            dbName,
        ))
    })

    // Your test here
}

Docker Optimizations

Beyond template databases

Disable fsync

services:
  postgres:
    image: postgres:16
    command: postgres -c fsync=off -c full_page_writes=off

Use tmpfs

services:
  postgres:
    image: postgres:16
    tmpfs:
      - /var/lib/postgresql/data

Combined Configuration

services:
  postgres:
    image: postgres:16
    command: postgres -c fsync=off
    tmpfs:
      - /var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: template1

Common Pitfalls

Hard-learned lessons

Connection Leaks

// Bad: connection not closed
db := createTestDB(t)
// test code...

// Good: ensure cleanup
db := createTestDB(t)
t.Cleanup(func() {
    db.Close()
})

Template Corruption

// Bad: modifying template
db := connectTo("template1")
db.Exec("INSERT INTO users...")

// Good: only read from template
db := connectTo("template1")
// Read-only operations or create test DB

Naming Conflicts

// Bad: predictable name
dbName := "test_db"

// Good: unique name
dbName := fmt.Sprintf("test_%s_%d",
    sanitize(t.Name()),
    time.Now().UnixNano(),
)

Connection Pool Limits

postgres:
  environment:
    POSTGRES_MAX_CONNECTIONS: 200

// In tests
db.SetMaxOpenConns(5)
db.SetMaxIdleConns(2)

Real Results

From the articles

Before and After

  • maragu.dk: 33.3s → 9.8s (3.4x)
  • mikecann.blog: 15x faster locally, 3x in CI
  • victoronsoftware: Large suite split into parallel jobs

My Results

# Before
go test ./...
# 45 seconds

# After: templates + parallel
go test -p 4 ./...
# 6 seconds

Summary

Three key techniques:

  1. Template databases (avoid repeated migrations)
  2. Go’s parallelization (t.Parallel() + -p)
  3. Docker optimizations (fsync, tmpfs)

Getting Started

  1. Profile your tests
  2. Add template database helper
  3. Mark tests t.Parallel()
  4. Optimize Docker config
  5. Adjust -p flag

The Payoff

  • Instant feedback
  • More confident refactoring
  • Faster CI/CD
  • Better developer experience

Resources

  • maragu.dk: Template database pattern
  • mikecann.blog: Docker fsync optimization
  • rotational.io: Parallel testing strategies
  • gajus.com: Template + tmpfs setup

Code Examples

Example repo: https://gitlab.com/hmajid2301/banterbus

Slides: https://haseebmajid.dev/slides/faster-go-tests-and-postgres/

Questions?