Slides

A Slidev deck covering event-driven architectures, Go, Kafka, PostgreSQL, outbox, retries and DLQs.

What I learnt building a web app with Go and HTMX Introduction Haseeb Majid Backend Software Engineer at Nala Loves cats 🐱 Avid cricketer 🏏 #BazBall 🌱 Amateur Gardener Who Is This For? Backend engineers Simple Stack Best fit when backend already owns most of the flow Banter Bus got me here, LettuceGo made the same lessons easier to show. What is HTMX? Small library Interact with backend Using HTML attributes Raise your hands sequenceDiagram participant User participant HTMX participant Server User->>HTMX: Click / submit HTMX->>Server: HTTP request Server-->>HTMX: HTML fragment HTMX-->>User: DOM swap <form hx-post="/plants/timeline" hx-target="#timeline" > <div class="flex flex-col gap-4"> <input name="event_type" type="radio" value="repotted" checked /> <textarea name="notes" rows="3"></textarea> <button type="submit">Add Event</button> </div> </form> ...

What I learnt building a web app with Go and HTMX Introduction Haseeb Majid Backend Software Engineer at Nala https://haseebmajid.dev Loves cats 🐱 Avid cricketer 🏏 #BazBall Who is this for? Backend Developers No JS Manage state in one place side projects struggled with frontend structure Story Time Tech Stack (Backend) Go Postgres Templ No external framework dependencies Built-in routing with Go 1.22+ patterns Tech Stack (Frontend) HTMX TailwindCSS AlpineJS HTMX A small library Extends HTML with AJAX (~14kb minified) Basic interaction via HTML attributes javascript fatigue: longing for a hypertext already in hand — htmx.org ...

Using Nix to Create Reproducible Go Development Environments Introduction Haseeb Majid Backend Software Engineer at Curve https://haseebmajid.dev Loves cats 🐱 Avid cricketer 🏏 #BazBall Who is this for? Interested in Nix Consistent development environments Old project; still works New developers “It works on my machine” Explain reproducible Explain ephemeral short-lived Looking to improve the developer experience Scared to re-run Fails on CI Credit What is Nix? Nix is a declarative package manager Nixlang the programming language that powers Nix Nixpkgs is the largest repository of Nix packages Put your hand up if you’ve heard of Nix limited lanuage Pure functional: no side effects, same inputs same outputs Lazy: only evaluates what is needed for that nix file nixos is not nix NixOS: A Linux distribution that can be configured using Nixlang nixpkgs maintained by community and NixOS foundation Declarative { wayland.windowManager.sway.enable = true; xsession.windowManager.i3.enable = false; } imperative: instructions ...

Using Nix to Create Reproducible Go Development Environments Introduction Haseeb Majid Backend Software Engineer at Curve https://haseebmajid.dev Loves cats 🐱 Avid cricketer 🏏 #BazBall Who is this for? Interested in Nix Consistent development environments Old project; still works New developers “It works on my machine” Explain reproducible Explain ephemeral short-lived Looking to improve the developer experience Scared to re-run Fails on CI Credit What is Nix? Nix is a declarative package manager Nixlang the programming language that powers Nix Nixpkgs is the largest repository of Nix packages Put your hand up if you’ve heard of Nix limited lanuage Pure functional: no side effects, same inputs same outputs Lazy: only evaluates what is needed for that nix file nixos is not nix NixOS: A Linux distribution that can be configured using Nixlang nixpkgs maintained by community and NixOS foundation Declarative { wayland.windowManager.sway.enable = true; xsession.windowManager.i3.enable = false; } imperative: instructions ...

Reproducible & Ephemeral Development Environments with Nix Introduction Haseeb Majid Backend Software Engineer at Curve https://haseebmajid.dev Loves cats 🐱 Avid cricketer 🏏 #BazBall Who is this for? Interested in Nix Consistent development environments Old project; still works New developers “It works on my machine” Explain reproducible Explain ephemeral short-lived Looking to improve the developer experience Scared to re-run Fails on CI Credit What is Nix? Nix is a declarative package manager Nixlang the programming language that powers Nix NixOS: A Linux distribution that can be configured using Nixlang limited lanuage Pure functional: no side effects, same inputs same outputs Lazy: only evaluates what is needed for that nix file nixos is not nix Declarative { wayland.windowManager.sway.enable = true; xsession.windowManager.i3.enable = false; } imperative: instructions ...

My Journey Using Docker as a Development Tool: From Zero to Hero by Haseeb Majid Presentation in Docker Code & Slides shared at the end About Me Haseeb Majid A software engineer https://haseebmajid.dev Loves cats 🐱 Avid cricketer 🏏 #BazBall ZOE I work for ZOE 🥑 https://joinzoe.com Personalised nutrition product Health study Understand how your body responds to food Use the power of community science to tackle global health issues from covid My Blood Sugar Levels Breathe Unpleasant symptoms like crashing, hunger, fatigue Using Docker at ZOE Next level Who Is This Talk For? Have used Docker But not an expert Know basic CLI commands Want to use Docker in CI Learn more about using in CI Multistage builds Example Code Simple FastAPI web-service Interacts with DB It allows us to get and add new users Poetry for dependency management FastAPI is a “async” Python Web framework, similar to Flask Postgres database Why Docker? Reproducible builds Easy setup for developers OS independent Start with this Upgrading MacOS nothing builds locally virtualenv Docker daemon running Easier to upgrade to new python versions My First Image # Dockerfile FROM python:3.9.8 ENV PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 \ PYTHONPATH="/app" \ PIP_NO_CACHE_DIR=off \ PIP_DISABLE_PIP_VERSION_CHECK=on \ PIP_DEFAULT_TIMEOUT=100 \ \ POETRY_VERSION=1.1.11 \ POETRY_HOME="/opt/poetry" \ POETRY_VIRTUALENVS_IN_PROJECT=true \ PYSETUP_PATH="/opt/pysetup" \ POETRY_NO_INTERACTION=1 \ \ VENV_PATH="/opt/pysetup/.venv" ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH" WORKDIR $PYSETUP_PATH COPY pyproject.toml poetry.lock ./ RUN pip install poetry==$POETRY_VERSION && \ poetry install WORKDIR /app COPY . . CMD [ "bash", "/app/start.sh" ] PYTHONDONTWRITEBYTECODE: prevents writing out pyc files PYTHONUNBUFFERED: sent straight to terminal real time ...

Observability Made Painless: Go, OTel & LGTM Stack LGTM: Loki, Grafana, Tempo, Mimir Introduction Haseeb Majid Backend Software Engineer at Nala https://haseebmajid.dev Loves cats 🐱 Avid cricketer 🏏 #BazBall Who is this for? New to OpenTelemetry Instrument an existing app Add observability signals Not covering PromQL What is Observability? What is going on with our app Is something wrong? What is Observability? Logs Traces Metrics Logs: Lots of context when something goes wrong Append only, detailed Metrics: ...

GitLab Runners: Docker-in-Docker Explained By Haseeb Majid What are GitLab Runners? An application which works with GitLab CI to run jobs in a pipeline 1. Types SaaS (Shared): GitLab’s own runners Enabled by default Limited by credits Self-Hosted: Runners we manage On our own infrastructure Need to register them with GitLab to use them What are Executors? Any system used to make sure the CI jobs are run 2. Each job is run separately from each other. ...

From Coffee Break to Instant Feedback: Rapid Integration Testing in Go Introduction Haseeb Majid Backend Software Engineer at Nala https://haseebmajid.dev Loves cats 🐱 Avid cricketer 🏏 #BazBall Who is this for? Go developers with slow test suites Teams using PostgreSQL for integration tests Anyone tired of waiting for CI Show of hands: who has gone to get coffee while waiting for tests? Who has lost focus because tests take too long? This talk is about getting that time back The Pain of Slow Integration Tests “I pushed to CI and went to make coffee… then forgot what I was working on” ...

An Introduction to PocketBase: A Go-Based Backend as a Service by Haseeb Majid Code & Slides shared at the end About Me Haseeb Majid Backend Software Engineer at Curve https://haseebmajid.dev Loves cats 🐱 Avid cricketer 🏏 #BazBall What is a Backend as a Service (BaaS)? Handle the basic repetitive tasks Authentication Database Management Email Verification Popular BaaS Firebase Supabase Amplify Why use PocketBase? Runs from a single binary Written in Go Extend as framework Easy to use Dashboard embeds SQLite DB UI Written in Svelte Scale can handle 10k connections on $6 VPS Super easy to self-host Dashboard Use as a Framework create a simple web application go mod init gitlab.com/hmajid2301/talks/.../example go get github.com/pocketbase/pocketbase // main.go package main import ( "log" "github.com/pocketbase/pocketbase" ) func main() { app := pocketbase.New() if err := app.Start(); err != nil { log.Fatal(err) } } go run main.go serve --http=localhost:8080 Add a Route # main.go import ( "net/http" "github.com/labstack/echo/v5" "github.com/pocketbase/pocketbase" "github.com/pocketbase/pocketbase/apis" "github.com/pocketbase/pocketbase/core" ) func main() { //... app.OnBeforeServe().Add(func(e *core.ServeEvent) error { e.Router.POST("/comment", handler, middlewares) return nil }) } echo V5 server app.OnBeforeServe().Add(func(e *core.ServeEvent) error { e.Router.POST("/comment", //handler func(c echo.Context) error { return c.NoContent(http.StatusCreated) }, //middlewares apis.ActivityLogger(app), apis.RequireRecordAuth(), ) return nil }) actual function some middlewares log request check user is auth Client Code import PocketBase from "pocketbase"; const pb = new PocketBase("http://127.0.0.1:8080"); // code to auth user // ... await pb.send("/comment", { // for all possible options check // https://developer.mozilla.org/en-US/docs/Web/API/fetch#options }); fetch Add Record to DB // ... type Comments struct { models.BaseModel Post string `db:"post" json:"post"` User string `db:"user" json:"user"` Message string `db:"message" json:"message"` } func (c *Comments) TableName() string { return "comments" } var _ models.Model = (*Comments)(nil) func main() { // ... app.OnBeforeServe().Add(func(e *core.ServeEvent) error { e.Router.POST("/comment", // handler func(c echo.Context) error { auth, _ := c.Get(apis.ContextAuthRecordKey).(*models.Record) m := fmt.Sprintf("Hi 👋 from %s", auth.Username()) commentRecord := &Comments{ User: auth.Id, Message: m, Post: "1", } err := app.Dao().Save(commentRecord) if err != nil { return err } return c.NoContent(http.StatusCreated) }, // middlewares apis.ActivityLogger(app), apis.RequireRecordAuth(), ) return nil }) } Migrations ls -al migrations/ Permissions User Group Date Modified Name .rw-r--r-- haseeb haseeb 2 Apr 22:52 1680445294_created_posts.go .rw-r--r-- haseeb haseeb 2 Apr 22:52 1680445383_created_comments.go .rw-r--r-- haseeb haseeb 2 Apr 22:52 1680445466_updated_comments.go .rw-r--r-- haseeb haseeb 2 Apr 22:52 1680445481_updated_posts.go // main.go package main import ( "log" "github.com/pocketbase/pocketbase" "github.com/pocketbase/pocketbase/plugins/migratecmd" // you must have have at least one // .go migration file in the "migrations" directory _ "gitlab.com/hmajid2301/talks/.../migrations" ) func main() { app := pocketbase.New() migratecmd.MustRegister( app, app.RootCmd, &migratecmd.Options{ // auto creates migration files // when making collection changes Automigrate: true, }) // ... } SQLite Does it Scale? Write-Ahead Logging (WAL mode) What is WAL Mode? SQLite groups rows together into 4KB chunks called “pages”. benefits POSIX system call fsync() commits buffered data to permanent storage or disk fsync() is expensive Why use WAL Mode? Is significantly faster in most scenarios. WAL uses many fewer fsync() operations Provides more concurrency as a writer does not block readers. No network file support Not atomic when commits across separate DB’s Might be slightly slower 1-2% for read heavy and very rare write apps In rollback mode, you can have concurrent readers but not readers & writers Testing package main import ( "net/http" "testing" "github.com/pocketbase/pocketbase/tests" "github.com/pocketbase/pocketbase/tokens" ) // username: test@example.com // password: password11 const testDataDir = "./tests/pb_data" func TestCommentEndpoint(t *testing.T) { recordToken, err := generateRecordToken("users", "test@example.com") if err != nil { t.Fatal(err) } setupTestApp := func() (*tests.TestApp, error) { testApp, err := tests.NewTestApp(testDataDir) if err != nil { return nil, err } bindAppHooks(testApp) return testApp, nil } scenarios := []tests.ApiScenario{ { Name: "try to get response", Url: "/comment", Method: http.MethodPost, RequestHeaders: map[string]string{ "Authorization": recordToken, }, ExpectedStatus: 201, ExpectedContent: nil, ExpectedEvents: map[string]int{"OnModelAfterCreate": 1, "OnModelBeforeCreate": 1}, TestAppFactory: setupTestApp, }, } for _, scenario := range scenarios { scenario.Test(t) } } func generateRecordToken(collectionNameOrId string, email string) (string, error) { app, err := tests.NewTestApp(testDataDir) if err != nil { return "", err } defer app.Cleanup() record, err := app.Dao().FindAuthRecordByEmail(collectionNameOrId, email) if err != nil { return "", err } return tokens.NewRecordAuthToken(app, record) } Deploy Dockerfile FROM golang:1.20-alpine as builder WORKDIR /build RUN apk update && apk upgrade && \ apk add --no-cache ca-certificates && \ update-ca-certificates COPY . . RUN CGO_ENABLED=0 GOOS=linux go build -o app main.go FROM scratch COPY --from=builder /build/app . COPY --from=builder /etc/ssl/certs/ca-certificates.crt \ /etc/ssl/certs/ ENTRYPOINT [ "./app" ] CMD ["serve", "--http=0.0.0.0:8080"] fly.io # fly.toml app = "example" kill_signal = "SIGINT" kill_timeout = 5 processes = [] [build] dockerfile = "Dockerfile" [env] ENV = "production" [experimental] allowed_public_ports = [] auto_rollback = true enable_consul = true [[services]] http_checks = [] internal_port = 8080 processes = ["app"] protocol = "tcp" script_checks = [] [services.concurrency] hard_limit = 25 soft_limit = 20 type = "connections" [[services.ports]] force_https = true handlers = ["http"] port = 80 [[services.ports]] handlers = ["tls", "http"] port = 443 [[services.tcp_checks]] grace_period = "1s" interval = "15s" restart_limit = 0 timeout = "2s" [mounts] destination = "/pb_data" source = "pb_data" fly deploy Gitlab CI deploy: stage: deploy only: - main image: docker services: - docker:dind before_script: - apk add curl - curl -L https://fly.io/install.sh | sh script: - fly deploy Other Features Expanding Relations Join tables without making additional request Uploading Files API to manage (DB) backups Caveats Need to self-host PocketHost Does not have a stable API yet Can only scale vertically LiteFS Scale horizontaly with LiteFS Any Questions? Code: https://gitlab.com/hmajid2301/talks/an-intro-to-pocketbase Slides: https://haseebmajid.dev/talks/an-intro-to-pocketbase/ Useful Links PocketBase Docs Fireship Video on PocketBase WAL Mode Explained LiteFS My App Built Using PocketBase