So what do we do if want to connect to our K3S cluster running our RPIs, but we are not on the same network/at home. Well we can look to use a VPN, in this article we will be using tailscale. It is super easy to setup on NixOS, and we need very little config for Tailscale. It also has a generous free tier, which will be more than enough for our home lab use case.

So essentially, what we will need to do is have a Tailscale service always running on each of PI hosts, though we could have it running on our main control node only (called strawberry from me, see previous articles in this series for more context). Once we start the service we will need to auth with Tailscale the first time, manually though.

The device we want to connect to our cluster from will also need to be running the Tailscale VPN, but we only need it running when we want to access our cluster.

All we need to do is add a line to our common.nix, this line services.tailscale.enable = true;. We can then deploy this using Colmena like colmena apply switch --build-on-target. This will deploy tailscale onto each our PI cluster nodes.

Then to authenticate with Tailscale we can ssh to our node and run the following sudo tailscale up. We can then auth in the browser by logging in. We can then also see the device on the Tailscale website.

Then where before I might use strawberry.local to connect now I can use strawberry when I am on the Tailscale VPN. Note if you want self host rather than using Tailscale, there is an open source project called headscale.

That’s it! We deployed Tailscale and can connect to our PI K3S cluster, from anywhere.

Some of you maybe wondering what is K3S, it is a Kubernetes distribution which is tiny i.e. the binary is only 50 MB. It also has fewer dependencies. Make it perfect our PI cluster and home lab and IoT apps. I am still going to work out how to manage the Kubernetes cluster itself, perhaps I will use Pulumi you could also terraform if you wanted a reproducible PI cluster.

In our common.nix, let’s enable the k3s service which will start k3s for us, one of the nice things about Nix. We will also add the token that the K3S nodes will need to communicate with each other:

{
  services.k3s.enable = true;

  sops.secrets.k3s_token = {
    sopsFile = ./secrets.yaml;
  };

  services.k3s.tokenFile = config.sops.secrets.k3s_token.path;
}

This needs to be secret, so we will use the sops-nix. In my case, I deployed the main node and then took the key from the node itself and shared it with the agents, so they can join the cluster. However, when we start the main node we can also pass it a token we specify. Sops saves the token to a file; hence we use tokenFile here, as we cannot actually access the value in our nix config.

We may also want to update our firewall, and open ports so that the nodes in the cluster can communicate each other from the hosts.

{
  networking.firewall = {
    allowedTCPPorts = [
      22
      6443
      6444
      9000
    ];
    enable = true;
  };
}

We don’t need to do much else for the main node, but for each agent node we need to tell it is that it should act as an agent, and which hostname to connect to join the nodes. Which we can do something like this mango.nix;

{
  services.k3s.role = "agent";
  services.k3s.serverAddr = "https://strawberry.local:6443";
}

Where they are all running in the same local network; hence they can connect using the hostname and .local. This was avahi service we set up in a previous post.

So in each 3 of my agent nodes this config is copies, so these nodes act as agents and connect to the k8s cluster correctly. We can then deploy the app as we normally would using Colmena, colmena apply switch --build-on-target.

Then we can follow this tutorial for cluster access using kubectl. We can then check all the nodes in the cluster by doing:

kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
strawberry   Ready    control-plane,master   41d   v1.27.6+k3s1
orange       Ready    <none>                 34d   v1.27.6+k3s1
mango        Ready    <none>                 32d   v1.27.6+k3s1
guava        Ready    <none>                 35d   v1.27.6+k3s1

That’s it! We successfully deployed k3s to our PI cluster using Colmena and even sops-nix from our previous posts. In the next post, we will look at how we can add Tailscale to be able to securely connect to our cluster from anywhere.

If we are using colmena, how can we set it up when we deploy a secret, for example when deploying k3s the token? i.e. services.k3s.tokenFile = "/my.token";.

So to do this first, I will assume you already have a colmena config and sops-nix setup in your config. First, let’s set up our hosts, in this case RPIs which already come /etc/ssh/ssh_host_ed25519_key ssh key we can turn to an age key, i.e. in our .sops.yaml.

keys:
  - &users:
    - &haseeb F04F743A24CD81B628A20667CD20E7373D83B71C
  - &hosts:
    - &strawberry age1qng4kav7deqtjmxeqz2vnyxywaqplf8k2lu3q347r2rz4zxdsynq0sf4um
    - &orange age187eesfqwv04gpd2dnfwsjgleevr57v6xvrwujjy8ehhf0ehl338qdnlqlf
    - &guava age10qsd50v2qmvn4vy4l8cjxvjxjuvedkxjc0a72ap9laap9mz6rctqmp3efl
    - &mango age16tskx6gle6v4v0hzhm5fvj0yd29mmn0s47d8q0h3tgcj9wej53uquv98cn

To get this file, we need to log in to our rpi host and run nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'. Then we add our secrets file:

creation_rules:
  - path_regex: hosts/rpis/secrets.ya?ml$
    key_groups:
    - age:
      - *strawberry
      - *orange
      - *guava
      - *mango
      pgp:
      - *haseeb

Then we can create our actual secrets file running sops hosts/rpis/secrets.yaml. Now we can reference these secrets in our colmena config. Let’s add sops to our common config:

{
  defaults = { pkgs, ... }: {
    imports = [
      inputs.hardware.nixosModules.raspberry-pi-4
      inputs.sops-nix.nixosModules.sops
      ./rpis/common.nix
    ];
  };
}

For example, if we take look at common.nix we can use sops like we normally would:

{
sops.secrets.k3s_token = {
    sopsFile = ./secrets.yaml;
  };

  services.k3s.tokenFile = config.sops.secrets.k3s_token.path;

  sops = {
    age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
  };
}

Then, we can run colmena switch like we usually would. Then the secret is made available at /run/secrets/k3s_token on the rpis like it normally would be.

We want to automate this process rather than deploying to each machine manually. I looked at bento, but couldn’t quite work out how to make it work for my use case. Then I found colmena, which worked (is working) to do what I needed.

What does it do?

We push changes from our Desktop and deploy to all our rpis at once, without needing to deploy to each manually. Using one command, colmena apply switch --build-on-target It will deploy our changes to all our RPIs. Where we specify common config shared by all the machines, but then also specific config for each of our RPIs.

Setup

Let’s have a look at how we can set up Colmena, so, in our current Nix flake config. Add colmena as an input:

{
    inputs = {
        colmena.url = "github:zhaofengli/colmena";
    };
}

Then in the outputs, let use specify a few things:

{

  outputs =
    { self
    , nixpkgs
    , colmena
    , ...
    } @ inputs:
    {

      colmena = {
        meta = {
          nixpkgs = import nixpkgs {
            system = "x86_64-linux";
          };
          specialArgs = inputs;
        };

        defaults = { pkgs, ... }: {
          imports = [
            inputs.hardware.nixosModules.raspberry-pi-4
            ./hosts/rpis/common.nix
          ];
        };

        strawberry = {
          imports = [
            ./hosts/rpis/strawberry.nix
          ];

          nixpkgs.system = "aarch64-linux";
          deployment = {
            buildOnTarget = true;
            targetHost = "strawberry";
            targetUser = "strawberry";
            tags = [ "rpi" ];
          };
        };
      };
    };
}

Breaking this file down, first we create a section for colmena, then we specify some meta information i.e. which nixpkgs to use. Then one of the cool bits of colmena we can specify some common config between all of our hosts.

Here we are importing sops-nix for secret management and hardware module for Raspberry Pi 4, so we can get various optimised settings.

common.nix

Our common.nix looks something like this:

{ config, pkgs, ... }: {
  boot = {
    kernelPackages = pkgs.linuxKernel.packages.linux_rpi4;
    kernelParams = [
      "cgroup_memory=1"
      "cgroup_enable=cpuset"
      "cgroup_enable=memory"
    ];

    initrd.availableKernelModules = [ "xhci_pci" "usbhid" "usb_storage" ];
    loader = {
      grub.enable = false;
      generic-extlinux-compatible.enable = true;
    };
  };

  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
      options = [ "noatime" ];
    };
  };

  networking.firewall = {
    allowedTCPPorts = [
      22
      6443
      6444
      9000
    ];
    enable = true;
  };

  programs.fish.enable = true;
  users.users.root.hashedPassword = "!";

  environment.systemPackages = with pkgs; [
    git
    vim
    wget
    curl
    gnupg
  ];

  services.avahi = {
    enable = true;
    nssmdns = true;
    publish = {
      enable = true;
      addresses = true;
      domain = true;
      hinfo = true;
      userServices = true;
      workstation = true;
    };
  };

  services.openssh = {
    enable = true;
    settings.PasswordAuthentication = false;
    settings.KbdInteractiveAuthentication = false;
  };

  security.sudo.wheelNeedsPassword = false;
  hardware.enableRedistributableFirmware = true;
  system.stateVersion = "23.11";
}

This config will be applied to all of our hosts. It includes things like setting up ssh, installing some packages and setting up a default shell (fish). Though specifics don’t matter much, more just you can put common config into a nix module.

Host-Specific Config

{
strawberry = {
  imports = [
    ./hosts/rpis/strawberry.nix
  ];

  nixpkgs.system = "aarch64-linux";
  deployment = {
    buildOnTarget = true;
    targetHost = "strawberry.local";
    targetUser = "strawberry";
    tags = [ "rpi" ];
  };
};
}

We import nix specific config for our host, in this case the host name of the machine is strawberry and gave the same The name of the colmena “resource”. Where my strawberry.nix file looks like:

{ config, pkgs, lib, ... }:

let
  hostname = "strawberry";
in
{
  networking = {
    hostName = hostname;
  };

  nix.settings.trusted-users = [ hostname ];

  users = {
    users."${hostname}" = {
      isNormalUser = true;
      shell = pkgs.fish;
      extraGroups = [ "wheel" ];
      password = hostname;
      openssh.authorizedKeys.keys = [
        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMxe8kDCJa6xcAM9WE8c5amGG+2secXmnof7vlmAq1Da hello@haseebmajid.dev"
      ];
    };
  };
}

Which just includes some specific config like hostname and username, and other ssh keys that can be used to log in to this user. Again, the specifics don’t matter too much, it’s more the idea we can have specific config for our strawberry host.

Then going over the rest of our config:

{
strawberry = {
  imports = [
    ./hosts/rpis/strawberry.nix
  ];

  nixpkgs.system = "aarch64-linux";
  deployment = {
    buildOnTarget = true;
    targetHost = "strawberry.local";
    targetUser = "strawberry";
    tags = [ "rpi" ];
  };
};
}

This specifies things like the host to connect to, i.e. strawberry.local and the user to log in with strawberry. We can also add tags which we can use during deployment to deploy to machines with specific tags.

Then we simply specify our remaining hosts:

{
  colmena = {
    orange = {
      imports = [
        ./hosts/rpis/orange.nix
      ];

      nixpkgs.system = "aarch64-linux";
      deployment = {
        buildOnTarget = true;
        targetHost = "orange.local";
        targetUser = "orange";
        tags = [ "rpi" ];
      };
    };
 # other hosts ...
}

If you set deployment.buildOnTarget = true; for a node, then the actual build process will be initiated on the node itself. Colmena will evaluate the configuration locally before copying the derivations to the target node.

We want to build the nix config on the pis, rather than on my desktop, as they are different architectures, x86 vs arch.

Deploy

Now to deploy to our rpis we can do colmena switch, from my desktop. Where my desktop has connectivity to all my PIs (running on the same local network). That should be it, as long as we can connect to the PIs from our machine we run the colmena command on it will deploy the new config.

In the next article, we will look at how we can manage secrets using sops-nix when deploying using colmena.

Appendix

• My colmena config

Recently, I proceeded to experiment with some Raspberry PIs (RPI) that I had lying around. I wanted to do something with them, so I decided I would turn them into a k8s cluster and put various random tools that might be nice to have on it. Such as a GitLab runner, Jellyfin media server & pi hole for ad blocking.

Hardware

The list below shows the things I used to set up my rpi cluster. None of this is sponsored! I paid for everything listed below:

• Utronics SSD Cluster Case for Raspberry Pi
  • SKU: U6244
• 2x Raspberry Pi 4 - 4GB RAM
• 2x Raspberry Pi 4 - 8GB RAM
• 2x 2TB SSD
• 2x 2TB Hard Disks 2.5"
• 4x SSD to USB 3.0 Cable for Raspberry Pi
• 4x Raspberry Pi PoE+ HAT
• 4x MicroSD Extender Set for Uctronics Cluster Cases
  • Easier to access SD cards in the case
• 4x 16 GB Micro SD Cards
• 4x CAT7 Ethernet Cables 0.5M
  • Could’ve gone a bit smaller (you can see in the photos)
• Powerline Adapter
  • P-Link TL-WPA4220
• TP-Link 5-Port Gigabit Desktop PoE Switch
  • TL-SG1005P
• Double sided Velcro
  • Attaches switch to Utronics case

Extra:

• Micro SD card to USB Adapter
• For flashing the image with NixOS

Setup

Power Over Ethernet

So I learnt I could power my rpis using Ethernet, power over Ethernet (PoE). This saves us needing an extra cable on each pi, I had a USB hub on top of my pi cluster (on top of the switch).

To use POE, we need to have a switch which can be used for POE, we also need a PoE hat on the Raspberry Pi. Then finally an Ethernet cable from the switch to the Raspberry Pi. We still need to power the switch from our mains. Since my rpis will be away from my router, I also use a powerline adapter for a more stable internet connection. So 4 of the ports in the switch connects to the rpis and the 5th connects to a powerline adapter.

SSD to USB

Since the SD cards I got only are 16GB for more persistent storage, I used some of my spares SSDs and hard disks. Once nice thing was the case has a nice slot for the SSDs to be placed (under the pis).

Then we use an adapter to convert the SSDs to USB 3.0 on the pi. So we can still access the SSD, with the PI. For more persistent data storage.

Install NixOS

I followed this tutorial, to setup NixOS on my pi cluster. We will need to repeat this process for each of pis (4x).

Assuming our host is running nix we can do:

nix-shell -p wget zstd

wget https://hydra.nixos.org/build/226381178/download/1/nixos-sd-image-23.11pre541036.63678e9f3d3a-aarch64-linux.img.zst
unzstd -d nixos-sd-image-23.11pre500597.0fbe93c5a7c-aarch64-linux.img.zst
dmesg --follow

Plug in your SD card and your terminal should print what device it got assigned, for example /dev/sdX. Press Ctrl+C to stop dmesg –follow.

Then finally we can do:

sudo dd if=nixos-sd-image-23.11pre541036.63678e9f3d3a-aarch64-linux.img of=/dev/sda bs=4096 conv=fsync status=progres

Then put the SD card in your pi. Before we can start deploying to them we need to get some bare-bones setup we can SSH to the machine.

Then I connect the PI to a keyboard and monitor, so I can see the IP address and set a default password on the nixos user.

passwd nixos

#123456

ifconfig

Then we can ssh to the pi from the same network, i.e. my main NixOS machine. By doing something like:

ssh nixos@192.168.0.75, replace 192.168.0.75 with the IP address of your pi, returned from the ifconfig command. Then create a new config file:

ssh

nix-shell -p git vim
sudo vim /etc/nixos/configuration.nix

With the following contents:

 {config, pkgs, lib, ... }:

let
  hostname = "strawberry";
in
{

  imports = [
    "${builtins.fetchGit { url = "https://github.com/NixOS/nixos-hardware.git"; }}/raspberry-pi/4"
  ];

  boot = {
    kernelPackages = pkgs.linuxKernel.packages.linux_rpi4;
    initrd.availableKernelModules = [ "xhci_pci" "usbhid" "usb_storage" ];
    loader = {
      grub.enable = false;
      generic-extlinux-compatible.enable = true;
    };
  };

  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
      options = [ "noatime" ];
    };
  };

  networking = {
    hostName = hostname;
  };

  environment.systemPackages = with pkgs; [ git vim ];
  nix.settings.trusted-users = [ hostname ];

  services.openssh = {
    enable = true;
    settings.PasswordAuthentication = false;
    settings.KbdInteractiveAuthentication = false;
  };

  users = {
    users."${hostname}" = {
      isNormalUser = true;
      extraGroups = [ "wheel" ];
      # Note: This is not very secure, I only use this password for a little while whilst the PI 
      # can only be accessible from my local network
      password = hostname;
      openssh.authorizedKeys.keys = [ "ssh-ed25519 <YOUR_PUB_KEY>" ];
    };
  };
  security.sudo.wheelNeedsPassword = false;

  services.avahi = {
    enable = true;
    nssmdns = true;
    publish = {
      enable = true;
      addresses = true;
      domain = true;
      hinfo = true;
      userServices = true;
      workstation = true;
    };
  };

  hardware.enableRedistributableFirmware = true;
  system.stateVersion = "23.11";
}

Finally, sudo nixos-rebuild switch to build this new configuration.

This setups a few things on my machine, with the avahi service, we can connect to the pi using the host name instead of the IP address. This saves needing to create a static IP for each pi. So I can now connect to this pi like:

ssh strawberry@strawberry.local

We also don’t need a password as I can use my ssh key to log in

Firmware Update

If we want to update our pi firmware, we can do something like:

nix-shell -p raspberrypi-eeprom
mount /dev/disk/by-label/FIRMWARE /mnt
BOOTFS=/mnt FIRMWARE_RELEASE_STATUS=stable rpi-eeprom-update -d -a

That’s It! We’ve set up our rpi cluster with NixOS, and we can connect to it using the hostname. In the next couple of tutorials we will look at how we can use Nix tools to manage the deployment and deploy k3s to the cluster.