Since finishing the second round of development on my game/web app banter bus (currently not deployed anywhere since I took down my k8s cluster), I have been looking for something else to sink my teeth into.

I decided I was going to sell my custom-built PC and then use the money to get a Framework desktop. So I thought I might as well go with something simpler that would work just as well for development.

As part of this, I decided to finally tackle some of the tasks I wanted to complete with my Nix configuration. Mainly setting up impermanence, which means files we don’t persist will get deleted across reboots, forcing us to specify more of our machine in config/nix code. I haven’t done this yet with home-manager, but will at some point do that as well, though that will take much more effort and also be more dangerous.

Whilst doing this, I decided that I would also like to set up Secure Boot with Lanzaboote and LUKS but using TPM decryption, vs needing to type in a LUKS password then another password to login.

I set up my new device using nixos-anywhere, a great tool when you have a well-defined nix config to build from, which can build and install NixOS on a device if it can SSH to it. My favourite tool for installing NixOS whenever I need to (probably more often than I’d like to admit). Whilst this all worked great and eventually I managed to get my setup working, I am writing this blog post on my new Framework Desktop. I wondered if we could cut down the number of manual steps, originally inspired by ¹.

However, I hit some issues with all of the tools above playing nice. Mostly sops-nix with impermanence, getting the SSH keys on the machine so I could use my secrets. In this increasingly long blog post, I will show you how I got it set up. I won’t go into lots of detail about each specific tool. I will assume you are somewhat familiar with them. Each could have their own blog post (or maybe YouTube video; I should start creating videos again, I think).

I will show you the final config I managed, with minimal manual effort needed. This should work on any Framework device (especially the Secure Boot part).

Config

Now I’ll quickly show you how I have set up my Nix config for the various tools above. These configurations are interdependent: impermanence creates the persist structure, sops-nix needs persisted SSH keys for decryption, and Lanzaboote needs persisted signing keys for Secure Boot. Link to my config (at the point I published this article).

sops-nix

When we have secrets that we don’t want to keep in plaintext, we can use sops to encrypt the data with our age/SSH keys. Either host key for NixOS secrets or our age key for home-manager related secrets. For example, for defining the user’s password, which in our secrets.yaml file looks like user_password: ENC[AES256_GCM,data:wSbEwtgPzM1FLYAb3rMCXneXJr2xM4w4lydvwA+DDP1i4DNPKQvC7VUKe00wIp3rDhjLTept/WA8uIGEJsPaf30/iYc2txdDWVvjDL81UnXtIfFoXKsYWQ7vrftShMJckByMiD5uZoYkSA==,iv:0YpQ5RL5CSjfS6jfpZArore25jn42uW3IRr8xLK/798=,tag:oVyxq1uIsKfV/HQWum6LQA==,type:str]. But this can be decrypted by our Nix config and stored at /run/secrets (and other folders next to it).

This means when we do a nixos-anywhere install, we need to send it relevant SSH keys/age keys so that it can decrypt our secrets such that we can log in. Otherwise, there will be no secret and we won’t be able to log in to our machine.

In my hosts/framework/default.nix config I have:

{
  sops.secrets = {
    user_password = {
      sopsFile = ./secrets.yaml;
      neededForUsers = true;
    };
  };

  user.passwordSecretFile = config.sops.secrets.user_password.path;
}

And then when defining users, we can do something like this in modules/nixos/user/default.nix, where we can define the hash of the password above:

{
    users.mutableUsers = false;
    users.users.${cfg.name} = {
      hashedPasswordFile = cfg.passwordSecretFile;
    };
}

And finally my modules/nixos/security/sops/default.nix config:

I reference this file in the persist, which I’m not sure I need to do:

{
    sops = {
      age.sshKeyPaths = [ "/persist/etc/ssh/ssh_host_ed25519_key" ];
    };
}

Lanzaboote (Secure Boot)

We will use Lanzaboote so we can enable Secure Boot.

Secure Boot usually refers to a platform firmware capability to verify the boot components and ensure that only your own operating system is allowed to boot. - NixOS Wiki ²

Here are the relevant parts of my Nix config modules/nixos/system/boot/default.nix for Secure Boot. We want to persist the /etc/secureboot folder, which is where our signing keys will be auto-generated by Lanzaboote on our first boot. We also need to disable systemd-boot if we are using Lanzaboote.

The combination of auto-generate keys and auto-enroll means fewer manual steps we need to take post-install to enable Secure Boot. However, the first time I did this, I did set this up manually following the Lanzaboote getting started guide. Do whichever you prefer; it’s not like it’s many steps anyway.

{
  boot.lanzaboote = mkIf cfg.secureBoot {
    enable = true;
    pkiBundle = "/etc/secureboot";

    autoGenerateKeys.enable = true;
    autoEnrollKeys = {
      enable = true;
      autoReboot = true;
    };
  };

  boot.systemd-boot = {
      enable = false;
      configurationLimit = 20;
      editor = false;
  };

  environment.persistence = {
    "/persist" = {
      directories = [
        "/etc/secureboot"
      ];
    };
  };
}

Impermanence

My impermanence module is longish, so I won’t go into lots of details. But we use btrfs to be able to roll back to a clean state, then copy files we specify in our persist btrfs subvolume.

disko

I use disko with nixos-anywhere to be able to partition my disks without lots of manual commands. Here is my disko config. You can see the subvolumes we create for our btrfs setup, i.e., /persist for persisting files between reboots.

Here is where I define my LUKS settings as well, so that our data will be encrypted at rest (full-disk encryption).

PCR 15

Eventually, we will want to enroll TPM to decrypt LUKS so that it is decrypted automatically but will only work with our TPM chip soldered onto our motherboard. Part of this will be specifying the PCRs. I don’t know loads about how it works.

PCR, Platform Configuration Registers.

Platform Configuration Registers (PCR) allow binding of the encryption of secrets to specific software versions and system state via hashes, so that the enrolled key is only accessible (may be “unsealed”) if specific trusted software and/or configuration is used. - Arch Wiki

We check the previous registers before we give access to the data inside the TPM, i.e., our LUKS password stored inside the TPM. There is a really good post ³ about how TPM decryption might not be secure enough, though I don’t think we need to worry about PCR 9 ⁴ (due to checksums on initrd, from what I understand anyway).

I may do a longer post about how this all works when I learn more myself. But if we don’t check PCR 15, someone could just copy the metadata and pretend they have our disk, assuming they have physical access to our PC. The TPM is not checking the PCR 15 value, so therefore would just spit out our LUKS password, which they could then use on our original disk. Now, of course, this all depends on your threat model, but I thought it’d be fun to fix this issue, at least on my laptop.

So I copied the module. Until the expected option is configured, it won’t do anything; we won’t know this until the system has been built. So it’ll be one more manual step we need to take, but perhaps worth it to make my system more secure.

{
  security.nixicle.pcr-verification = {
    enable = true;
    # expectedPcr15 = "caf33e79c645b65849256238a11fa68ae197e5cb89730c463c1cdf1d9128376f";
  };
}

Steps

Now onto how I deploy NixOS onto a new system/device with all the above playing nice and reducing the number of manual steps we need to take. Normally, I had Claude create some scripts I can use with go-task, but for the sake of this blog post, we will do it step by step, i.e., normally I would do task install:secure. Where we could then follow the script using charm’s gum to make nice interactive scripts. This includes info like the username, IP to SSH onto.

Anyway, make sure the device can be SSH’d onto; usually you can just use a live media USB. I build my own with nixos-generators. But any live media should be fine; use NixOS if you don’t have anything set up. I do something like this: mkdir -p ~/.ssh && curl https://github.com/hmajid2301.keys > ~/.ssh/authorized_keys, so that I can SSH onto the live media. Also, take note of the IP address of the machine; you can find this by running ip addr.

For example, on my local network it might be 192.168.1.71.

Install

Enter our encryption password to a file:

MY_PASS=$(mktemp)
echo "YOUR_LUKS_PASSWORD" > $MY_PASS

Get SSH keys ready for copying onto the new machine; this is so that sops can decrypt the various secrets. Either you can copy the existing SSH keys and replace them after, or you can generate new ones and update your sops config and sops files with these new keys. The latter is the more secure option, but I went with the first one as I was feeling lazy (security through inconvenience is still security, right? Right?):

Note: If this is your first installation and you don’t have existing SSH keys at /persist/etc/ssh/, you can either generate temporary keys or skip this step and regenerate your secrets after installation with the new host keys that will be created.

SSH_KEYS=$(mktemp -d)
mkdir -p "$SSH_KEYS/persist/etc/ssh"

sudo cp /persist/etc/ssh/ssh_host_ed25519_key "$SSH_KEYS/persist/etc/ssh/"
sudo cp /persist/etc/ssh/ssh_host_ed25519_key.pub "$SSH_KEYS/persist/etc/ssh/"
sudo cp /persist/etc/ssh/ssh_host_rsa_key "$SSH_KEYS/persist/etc/ssh/" 2>/dev/null
sudo cp /persist/etc/ssh/ssh_host_rsa_key.pub "$SSH_KEYS/persist/etc/ssh/" 2>/dev/null

Update permissions on the SSH keys folder:

Note: This step is crucial for sops-nix to work with impermanence. If you skip this, you’ll need to re-encrypt your secrets with new host keys.

sudo chmod 600 "$SSH_KEYS/persist/etc/ssh"/ssh_host_*_key 2>/dev/null
sudo chown -R $(id -u):$(id -g) "$SSH_KEYS"

Then finally we can start the install process:

Note: Update the username@host as needed, nixos is the default user for the NixOS ISO.

nixos-anywhere \
--flake ".#framework" \
--disk-encryption-keys /tmp/disk-encryption.key "$MY_PASS" \
--extra-files "$SSH_KEYS" \
--build-on-remote \
"nixos@192.168.1.71"

Post Install

After the command is done, your PC will reboot. At this point, you can start to get it ready for Secure Boot if you want. For a Framework laptop to enable Secure Boot, first we need to erase the Secure Boot settings.

It may differ on your device:

Framework-specific: Enter Setup Mode

On Framework you can enter the setup mode like this:

    Select "Administer Secure Boot"
    Select "Erase all Secure Boot Settings"

When you are done, press F10 to save and exit.

Then enter your password for LUKS so it can decrypt your drive and boot like normal. You will see your PC reboot due to the auto-reboot we set above with Lanzaboote. That’s fine; it’s all very normal. If your computer starts speaking Latin or emitting smoke, that’s NOT normal. Please consult a priest or your local fire department.

On my Framework device, it then told me it’s going to enroll my keys, and I didn’t interrupt the process. Then I logged into my device and enabled TPM decryption:

Note: Replace /dev/nvme0n1p2 with your actual LUKS partition. Check your disko config to confirm the correct partition path.

# The --tpm2-pcrs=15:sha256=0000... uses all zeros as a placeholder that will be updated after first boot
sudo systemd-cryptenroll /dev/nvme0n1p2 \
    --wipe-slot=tpm2 \
    --tpm2-device=auto \
    --tpm2-pcrs=0+2+7 \
    --tpm2-pcrs=15:sha256=0000000000000000000000000000000000000000000000000000000000000000

Check and make sure it all looks normal; you should see files signed (except perhaps one old one):

❯ sudo -E sbctl verify
[sudo] password for haseeb:
Verifying file database and EFI images in /boot...
✓ /boot/EFI/BOOT/BOOTX64.EFI is signed
✓ /boot/EFI/Linux/nixos-generation-40-7ik2vgrngo25ml7c2wb43uy52pt7ahpovfrrxczp445y76u56dlq.efi is signed
✗ /boot/EFI/nixos/kernel-6.18.2-otn6nn3tkudhh5xpj5736u2q3h4kjzojd6fkg4rqzdf5l5c7gxuq.efi is not signed
✓ /boot/EFI/systemd/systemd-bootx64.efi is signed

# Verify Secure Boot is working
sudo bootctl status

Now we can enable Secure Boot; again, this may vary on your device.

On Framework you need to manually enable Secure Boot:

    Select "Administer Secure Boot"
    Enable "Enforce Secure Boot"

When you are done, press F10 to save and exit.

Finally, after logging in again, we can run the following to get the actual PCR 15 hash value:

❯ systemd-analyze pcrs 15 --json=short
[{"nr":15,"name":"system-identity","sha256":"0000000000000000000000000000000000000000000000000000000000000000"}]

Take the sha256 value from this output and update your config’s security.nixicle.pcr-verification.expectedPcr15 field with the actual hash, then run:

sudo nixos-rebuild switch --flake .#framework

Then, of course, go about setting up everything else. If you copied existing SSH keys during installation, you may want to rotate them for better security by generating new host keys and re-encrypting your sops secrets with the new keys.

That’s it! Hopefully, you found that useful! I appreciate it was very long-winded, but that’s how I was able to do a mostly unattended install with few manual steps after getting the process started. If you made it this far, you deserve a cookie 🍪 (or at least a functioning NixOS installation).

Appendix

I had to do a bunch of research whilst working on this; here are some useful links I used:

• A really interesting post about how you can break TPM encryption with Secure Boot: https://oddlama.org/blog/bypassing-disk-encryption-with-tpm2-unlock/#crude-implementation-of-pcr15-verification
• Lanzaboote getting started guide: https://github.com/nix-community/lanzaboote/blob/master/docs/getting-started/enable-secure-boot.md
• Idea for copying SSH keys: https://github.com/nix-community/nixos-anywhere/issues/604
• nixos-anywhere SSH keys copy: https://github.com/nix-community/nixos-anywhere/blob/main/docs/howtos/secrets.md#example-decrypting-an-openssh-host-key-with-pass
• Discourse post with same issue as mine: https://discourse.nixos.org/t/impermanence-sops-nix-nixos-anywhere-lead-to-missing-hashedpasswordfile-s/66472
• Another discourse post with a similar issue: https://discourse.nixos.org/t/nixos-anywhere-failing-to-deploy-secrets/68392

So I have a small home lab, where I have a few random things running, like a GitLab CI runner, a media server etc. All my home lab servers (3 of them) are running NixOS and are defined within my Nix flake what services they run.

For example: https://gitlab.com/hmajid2301/nixicle/-/blob/main/systems/x86_64-linux/ms01/default.nix?ref_type=heads

They are set up using nixos-anywhere like most of my other services. Then using deploy-rs we can ssh onto our home lab servers and deploy nix config, i.e. updating packages.

I wanted a more automated way to do, i.e. anytime nixicle repo changes, try to deploy those changes to my home lab on CI.

Tailscale

I connect to my home lab using Tailscale a very cool tool, think like a VPN, it uses WireGuard behind the scenes. With some extra stuff on top to provide a really nice user experience. Highly recommend if you are looking to get into homelabbing.

So first, our CI runner would need to be able to connect to Tailscale to be able to deploy our new config onto the home lab servers.

I will assume you have already set up your Tailscale account for the next part. I may create a tutorial later on how you can do it. Better to be honest, there is a ton of decent literature available already, including videos.

First, let’s create an auth key. Go to: https://login.tailscale.com/admin/settings/keys

We want to enable Ephemeral, so the device gets removed from our list, else every time CI runs we will get a new device in our list of machines. Then, in my case, I need to also check the pre-approve, so the admin doesn’t need to manually approve it, slightly defeating the purpose of CI.

You can also optionally create a tag for your machines using this auth key. In my case, I wanted to tag it with CI. Which we can then use for ACL rules etc.

Gitlab CI

After creating the key copy the value, you won’t be able to see it again, so keep that in mind. Add it to your CI variables, you can check the masked option, so the auth key isn’t accidentally printed in CI.

(You can access this using the left side bar; setting > CI/CD > Variables)

Then our CI file may look something like this .gitlab-ci.yml

image: nixos/nix

stages:
  - deploy

tailscale:
  stage: deploy
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - mkdir -p /etc/nix/ /etc/ssh/
    - echo "experimental-features = nix-command flakes" >> /etc/nix/nix.conf
    - nix-env -iA nixpkgs.docker nixpkgs.go-task nixpkgs.flyctl nixpkgs.tailscale nixpkgs.jq
    - ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519
    - ssh-keygen -t ed25519 -N "" -f /etc/ssh/ssh_host_ed25519_key
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo "Host *" >> ~/.ssh/config
    - echo "  StrictHostKeyChecking no" >> ~/.ssh/config
    - echo "  UserKnownHostsFile /dev/null" >> ~/.ssh/config
    - export SSH_ASKPASS_REQUIRE=never
    - eval $(ssh-agent -s)
    - ssh-add ~/.ssh/id_ed25519 2>/dev/null

    - tailscaled --state="mem:" --statedir=/var/lib/tailscale &
    - tailscale up --authkey=${TAILSCALE_AUTHKEY} --hostname="gitlab-$(cat /etc/hostname)" --accept-routes --ssh
    - nix profile install github:serokell/deploy-rs
    - deploy ".#ms01" --hostname ms01 --ssh-user nixos --skip-checks --confirm-timeout "600" --remote-build

Let’s break this file down

First, we want to create some SSH keys so that you can SSH on the CI runner onto the home lab server.

The next part we start up the Tailscale daemon in the background, and then we authenticate with Tailscale so our runner can now connect to our home lab servers

Then, finally, we install deploy-rs and use it to SSH to our machines and deploy to it.

Tailscale SSH

I also had to enable Tailscale ssh on the host devices so we could SSH from the runner without needing to include an SSH private key and adding that to the allowed SSH list on each server. Which we can do like so:

tailscale set --ssh

This lets Tailscale take over SSH for us, users can still SSH like normal, but you can also SSH like normal as well. But just make sure you are aware of the security implications before you turn it on.

Then I set up on ACLs like so:

	"tagOwners": {
		"tag:ci":           [],
		"tag:homelab":      [],
	},

	// Define access control lists for users, groups, autogroups, tags,
	// Tailscale IP addresses, and subnet ranges.
	"acls": [
		// Allow all connections.
		// Comment this section out if you want to define specific restrictions.
		{"action": "accept", "src": ["tag:ci"], "dst": ["tag:homelab:*"]},
		{
			"action": "accept",
			"src":    ["tag:homelab"],
			"dst":    ["tag:homelab:*"],
		},
	],

	// Define users and devices that can use Tailscale SSH.
	"ssh": [
		// Allow all users to SSH into their own devices in check mode.
		// Comment this section out if you want to define specific restrictions.
		{
			"action": "accept",
			"src":    ["autogroup:member", "tag:ci"],
			"dst":    ["tag:homelab"],
			"users":  ["autogroup:nonroot", "root"],
		},
		{
			"action": "accept",
			"src":    ["tag:ci"],
			"dst":    ["tag:homelab"],
			"users":  ["nixos", "root"],
		},
	],

I am no expert on ACL rules in Tailscale, but I think this allows the CI runner to SSH onto the home lab servers. Which I, of course, have already tagged with home lab.

Why Nix?

In my project, it is a go web app, so all the dependencies in the project are managed as go modules by my go.mod file. Then everything else the developer needs is set up as a dev shell. Things like the standalone tailwind CLI, I am using a nix flake which already has DaisyUI built in so it can also generate those types i.e. btn in the final styles.css (or whatever you called it). The flake is here: https://github.com/aabccd021/tailwindcss-daisyui-nix

I didn’t want to have multiple package managers, using npm or something else.

You can learn more about this here: https://www.youtube.com/watch?v=bdGfn_ihHOk

Setup

Assuming we have enabled the tailwind LSP in our Neovim config, I am using NixCats, and it looks it like this (using nvim-lspconfig):

{
    "tailwindcss",
    lsp = {},
},

You can find my full config here:

• https://gitlab.com/hmajid2301/nixicle/-/blob/8c959ca5bd6436595592d0cedf932787ba86e359/modules/home/cli/editors/neovim/lua/myLuaConf/LSPs/init.lua#L199
• https://gitlab.com/hmajid2301/nixicle/-/blob/MAJ-311/modules/home/cli/editors/neovim/default.nix#L48

So first I added this flake as an input

tailwindcss_daisy.url = "github:aabccd021/tailwindcss-daisyui-nix/lsp";
tailwindcss_daisy.inputs.nixpkgs.follows = "nixpkgs";

Then install the following packages:

tailwindcss-language-server
tailwindcss_daisy.packages.${system}.default

The tailwindcss-language-server being the LSP server for tailwind. Normally, this is installed in my dot files repo. (I linked above). But this one is an overlay from the flake. So this one also has DaisyUI in its NODE_PATH. Then it also auto-complete classes from DaisyUI.

Read this issue: https://github.com/aabccd021/tailwindcss-daisyui-nix/issues/1

Finally, to get the tailwind LSP to work, we need to create an empty tailwind.config.js. Else, the tailwind LSP does not seem to start correctly. Even though the project is configured using the CSS file (probably will be fixed)

That’s it! It took me a while to figure it out, but this seems to work pretty nicely.

Debugging Steps

This occurred off and on over the course of a few weeks. My font setup was primarily done using stylix and a bit of NixOS option config.

I noticed this only seemed to happen when the CSS on the page include a font-family like so “monospace” i.e. the monospace font in quotes. But without quotes, the font would render fine.

I made a few posts to ask for help, as I was confused, why this was happening, and found a similar issue, you can find here: https://old.reddit.com/r/NixOS/comments/1fha5hl/firefox_font_problem/.

Anyway, I didn’t get any reply, so I decided to use fc-match to see what font my system was returning.

blog on  2025-01-20-how-to-fix-firefox-rendering-emoji-font-black-bold-numbers-issue [$?] via 🐹 v1.22.10 via ❄  impure (nix-shell-env) took 19s
❯ fc-match monospace
MonoLisa-Regular.ttf: "MonoLisa" "Regular"

blog on  2025-01-20-how-to-fix-firefox-rendering-emoji-font-black-bold-numbers-issue [$?] via 🐹 v1.22.10 via ❄  impure (nix-shell-env)
❯ fc-match '"monospace"'
NotoSans[wdth,wght].ttf: "Noto Sans" "Regular"

I thought I’d found the smoking gun, ohh it was trying to use the San font to render it. I asked someone to share their working config from an Ubuntu machine and theirs also seemed to be similar to mine.

╰─ fc-match "monospace"
DejaVuSansMono.ttf: "DejaVu Sans Mono" "Book"

╰─ fc-match '"monospace"'
DejaVuSans.ttf: "DejaVu Sans" "Book"

Then whilst looking up stuff specific to how Firefox renders fonts, I realised in the DevTools there is a page for fonts, it’s just a bit hidden. I mean, it makes sense, of course there is.

So looking at this I realised it wasn’t bold text it was rendering, but it was falling back and rendering the numbers as emojis.

I then had a closer look at my font config in Nix and realised it wasn’t doing what I thought it was doing.

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
    <alias binding="weak">
        <family>monospace</family>
        <prefer>
            <family>emoji</family>
        </prefer>
    </alias>
    <alias binding="weak">
        <family>sans-serif</family>
        <prefer>
            <family>emoji</family>
        </prefer>
    </alias>
    <alias binding="weak">
        <family>serif</family>
        <prefer>
            <family>emoji</family>
        </prefer>
    </alias>
</fontconfig>

What I had done by copying and pasting some config I didn’t fully understand (tut tut tut) is if the character was available in the emoji font use that instead, i.e. the numbers or certain symbols like TM. So after deleting that config and reloading Firefox, it was rendering properly as expected.

That’s it! As expected, remember, to make sure you understand what your code is doing. I thought it was allowing emojis to be rendered with the monospace font, not replacing it or preferring it.

Relevant commit resolving the above issue: https://gitlab.com/hmajid2301/nixicle/-/commit/3168db8df339d79e838c98f8675eb21a23452d4e

There is currently a branch in home-manager we can pull into our config, which provides a convenient way to wrap these apps in nixGL rather than needing to specify say nixGLIntel kitty in our terminal or say Hyprland config (key bindings).

First, let’s import the relevant code from the branch/PR:

 imports = [
    # TODO: remove when https://github.com/nix-community/home-manager/pull/5355 gets merged:
    (builtins.fetchurl {
      url = "https://raw.githubusercontent.com/Smona/home-manager/nixgl-compat/modules/misc/nixgl.nix";
      sha256 = "01dkfr9wq3ib5hlyq9zq662mp0jl42fw3f6gd2qgdf8l8ia78j7i";
    })
  ];

Then, simply, we can do something like this:

programs = {
    kitty.package = config.lib.nixGL.wrap pkgs.kitty;
    firefox.package = config.lib.nixGL.wrap pkgs.firefox;
}

Then we should be able to use the apps like normal, such as using our app launcher like rofi. That’s It!

Appendix

• Commit with nixGL wrap in my own config

Prerequisite

In this article, I assume you are already familiar with Cloudflare Tunnels and Traefik.

Background

Cloudflare tunnels are an easy way to allow access to apps running on a server, i.e. home lab. Where you only want expose, say, a few services. When a client connects, they connect to Cloudflare and then Cloudflare makes a secure connection to your server. Which is running the cloudflared daemon.

Where my Cloudflare nix config looks something like this ¹:

{
  config,
  lib,
  ...
}:
with lib;
with lib.nixicle; let
  cfg = config.services.nixicle.cloudflared;
in {
  options.services.nixicle.cloudflared = {
    enable = mkEnableOption "Enable The cloudflared (tunnel) service";
  };

  config = mkIf cfg.enable {
    sops.secrets.cloudflared = {
      sopsFile = ../secrets.yaml;
      owner = "cloudflared";
    };

    services = {
      cloudflared = {
        enable = true;
        tunnels = {
          "ec0b6af0-a823-4616-a08b-b871fd2c7f58" = {
            credentialsFile = config.sops.secrets.cloudflared.path;
            default = "http_status:404";
          };
        };
      };
    };
  };
}

The other apps I have set up in Cloudflare tunnels point directly to the service itself. This is the first one I needed to point to Traefik, and then Traefik would forward the request to Navidrome.

Where my Traefik setup looks something ²:

{
  config,
  lib,
  ...
}:
with lib; let
  cfg = config.services.nixicle.traefik;
in {
  options.services.nixicle.traefik = {
    enable = mkEnableOption "Enable traefik";
  };

  config = mkIf cfg.enable {
    services = {
      traefik = {
        enable = true;

        staticConfigOptions = {
          # removed config to simplify

          entryPoints.web = {
            address = "0.0.0.0:80";
            http.redirections.entryPoint = {
              to = "websecure";
              scheme = "https";
              permanent = true;
            };
          };

          entryPoints.websecure = {
            address = "0.0.0.0:443";
            http.tls = {
              certResolver = "letsencrypt";
              domains = [
                {
                  main = "homelab.haseebmajid.dev";
                  sans = ["*.homelab.haseebmajid.dev"];
                }
                {
                  main = "haseebmajid.dev";
                  sans = ["*.haseebmajid.dev"];
                }
              ];
            };
          };
        };
      };
    };
  };
}

Things to note about my setup (now shown here) but Traefik is set up to be able to provision certificates from let’s encrypt using Cloudflare DNS challenge (see full config linked above).

Setup

We can see here this is how I configured my tunnel, first I point to traefik on port 443, i.e. https. As Traefik I have set up always expects traffic on 443. Also notice the origin server name is navidrome.haseebmajid.dev. This will get passed to Traefik.

Or in my Nix config I have defined it like so:

{
services = {
  cloudflared = {
    enable = true;
    tunnels = {
      "ec0b6af0-a823-4616-a08b-b871fd2c7f58" = {
        ingress = {
          "navidrome.haseebmajid.dev" = {
            service = "https://localhost";
            originRequest = {
              originServerName = "navidrome.haseebmajid.dev";
            };
          };
        };
      };
    };
  };
};
}

Finally, the last bit to connect the two bits together, the Traefik config:

{
  traefik = {
    dynamicConfigOptions = {
      http = {
        services = {
          navidrome.loadBalancer.servers = [
            {
              url = "http://localhost:4533";
            }
          ];
        };

        routers = {
          navidrome = {
            entryPoints = ["websecure"];
            rule = "Host(`navidrome.haseebmajid.dev`)";
            service = "navidrome";
            tls.certResolver = "letsencrypt";
            middlewares = ["authentik"];
          };
        };
      };
    };
  };
}

Pointing to our local service http://localhost:4533 where Navidrome is running. We also tell it that if we get a request for host navidrome.haseebmajid.dev it should forward it to that localhost service. We also tell it to provision a certificate using letsencrypt. Though you may not need this as Cloudflare does it for you.

However, Cloudflare will only do it on direct subdomains, i.e. it won’t work on navidrome.homelab.haseebmajid.dev. Because it’s a subdomain of a subdomain.

Then tell it to use the authentik middleware, which you can find out how to set up here. This tells Traefik to forward the request to Authentik, so we can correctly authenticate the user.

That’s it, we should be able to access our service Navidrome when it’s all deployed. The request is going from Client → Cloudflare →Traefik → Navidrome.

Authentik is a great app which can handle authentication for almost all of our home lab. So we only need to log in with Authentik, to log in to any of our apps. Rather than needing separate passwords and usernames for each app.

You may see an error like this, when trying to access the app:

or you may see this

Background

As per is the case with everything I do, rather than sticking to my home lab in K3S. I decided to move some of it to NixOS. The main reason for this being, it felt like there was a lot of boilerplate, sometimes 4 files just to deploy an app. I felt like I was getting more frustrated and not really spending any time setting up my home lab.

So hence I decided to have a look at how I could do it with NixOS itself. I would lose out on some of the HA features, i.e. if a node goes down, pods would get moved to another node. But for now, it would be good enough.

One thing that took me a while to figure out was how to set up forward auth like I had working with my Kubernetes cluster. Such as protecting apps from the “arr” stack, such as Sonarr.

As seems to be the norm for these blog posts it took me longer than I would care to admit, so let me show you how I did it.

Getting started

For this, I will use the embedded outpost we have with Authentik and you are already using Traefik as your reverse proxy.

Authentik

To set up Authentik, I used authentik-nix. Where my config looks something like this:

{
    services = {
      authentik = {
        enable = true;
        environmentFile = config.sops.secrets.authenik_env.path;
        settings = {
          email = {
          # email settings
          };
          disable_startup_analytics = true;
          avatars = "initials";
        };
      };
    };
}

Where my authentik_env config in sops looks like:

authenik_env: |
    AUTHENTIK_SECRET_KEY=
    AUTHENTIK_EMAIL__PASSWORD=

Then the key bit I was missing the relevant Traefik config:

• Define a new middleware
• Define a new service
• Define a new route to take us to the Authentik embedded outpost

{

 services.traefik = {
    dynamicConfigOptions = {
      http = {
        middlewares = {
          authentik = {
            forwardAuth = {
              tls.insecureSkipVerify = true;
              address = "https://localhost:9443/outpost.goauthentik.io/auth/traefik";
              trustForwardHeader = true;
              authResponseHeaders = [
                "X-authentik-username"
                "X-authentik-groups"
                "X-authentik-email"
                "X-authentik-name"
                "X-authentik-uid"
                "X-authentik-jwt"
                "X-authentik-meta-jwks"
                "X-authentik-meta-outpost"
                "X-authentik-meta-provider"
                "X-authentik-meta-app"
                "X-authentik-meta-version"
              ];
            };
          };
        };

        services = {
          auth.loadBalancer.servers = [
            {
              url = "http://localhost:9000";
            }
          ];
        };

        routers = {
          auth = {
            entryPoints = ["websecure"];
            rule = "Host(`authentik.haseebmajid.dev`) || HostRegexp(`{subdomain:[a-z0-9]+}.haseebmajid.com`) && PathPrefix(`/outpost.goauthentik.io/`)";
            service = "auth";
            tls.certResolver = "letsencrypt";
          };
        };
      };
    };
 };
}

The main bit I was missing was the address I needed to use the port 9443 to redirect to the embed outpost. Then creating the relevant auth redirect:

  auth = {
    entryPoints = ["websecure"];
    rule = "Host(`authentik.haseebmajid.dev`) || HostRegexp(`{subdomain:[a-z0-9]+}.haseebmajid.com`) && PathPrefix(`/outpost.goauthentik.io/`)";
    service = "auth";
    tls.certResolver = "letsencrypt";
  };

Where you should adjust authentik.haseebmajid.dev to match the domain you are using to access authentik on. Then you can simply follow the docs and should be able to repeat this for all apps you want to set this forward auth for.

Finally, in our Sonarr Traefik config, we would like to add our middleware that we just defined above:

  sonarr = {
    entryPoints = ["websecure"];
    rule = "Host(`sonarr.bare.homelab.haseebmajid.dev`)";
    service = "sonarr";
    tls.certResolver = "letsencrypt";
    middlewares = ["authentik"];
  };

That’s about it! The main thing I was missing was that https://localhost:9443/outpost.goauthentik.io/auth/traefik especially the port. After I added that everything worked like a dream.

Appendix

• My home lab Nix config
  • Sonarr Traefik config
  • Authentik config