Back in 2017 I wrote about my small Kubernetes cluster in OpenFaaS and Kubernetes and Kubernetes Persistent Volumes and RBD. At the time, the nodes were Debian machines that I installed by hand and initialised with kubeadm, and the volumes came from a small Ceph cluster at home. The git repository behind that cluster is still the same one today, with the first commit “Initial import” from September 2017 and about 2900 commits since. Almost everything else has changed.

In this post I will describe how the lab looks now: the nodes run Talos Linux, everything above the nodes is deployed by ArgoCD from the git repository, and storage comes from a TrueNAS box instead of Ceph. I will also go through what actually runs there and what I would still like to fix.

Why Talos

The nodes of the old cluster were general-purpose Linux machines. Each one had its own packages, its own upgrades and its own small differences, and in 2019 I moved the installation to Kubespray so that Ansible would at least keep them consistent. It worked, but upgrading Kubernetes was still a long Ansible run that I postponed as much as possible.

In March 2021 I replaced this with Talos. Talos is a Linux distribution that only runs Kubernetes: there is no shell, no SSH and no package manager. The whole machine is described by one YAML file and managed through an API with talosctl. For a home lab this is exactly what I wanted: a node is something I can reinstall from a file, not something I need to remember how I configured.

The nodes are virtual machines. They first ran on a Proxmox cluster of three physical machines. Today they run on XCP-ng on a single, beefier host. One host uses less power, but it also means that the hardware is now the single point of failure of the whole lab.

The cluster has three control plane nodes and five workers. Three control plane nodes keep etcd and the API available when one of them is down, so I can reboot or upgrade the nodes one at a time without stopping the cluster.

Bootstrapping ArgoCD

Once Kubernetes is running, there is exactly one thing I install by hand:

$ kubectl apply -k argocd

The argocd folder is a kustomization containing the upstream ArgoCD manifest (currently v3.1.0), a Traefik route for the UI, the SSH key to read the repository, and one Application resource for every folder of the repository:

resources:
  - base/install.yaml # from .../argo-cd/v3.1.0/manifests/install.yaml
  - base/ingress-traefik.yaml
  - base/repo-secrets.sops.yaml
  - base/appgroup-default.yaml
  - base/app-argocd.yaml
  - base/app-cert-manager.yaml
  - base/app-forgejo.yaml
  [...]
  - base/app-woodpecker.yaml

This is the “app of apps” pattern, with a small twist: one of the applications is ArgoCD itself, pointing to the same argocd folder.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'ssh://git@git.example.org:10022/servers/kubernetes-lab.git'
    path: argocd
    targetRevision: HEAD
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: argocd
  syncPolicy:
    automated: {}

After the manual kubectl apply -k, ArgoCD takes over its own installation. Adding a new service means creating a folder, adding an app-<name>.yaml in argocd/base and pushing. Upgrading ArgoCD means replacing install.yaml with the new upstream manifest and pushing.

Most applications use automated sync with prune and selfHeal. A few of the base components (Traefik, MetalLB, the SMB CSI driver) are still synced manually, because a bad change there takes everything else down with it and I prefer to press the button myself.

Before ArgoCD sees anything, a Woodpecker pipeline renders every application the same way ArgoCD would (kustomize, helm or plain directory), validates the result with kubeconform, and checks that no plain kind: Secret was committed by mistake. Secrets are encrypted in the repository with SOPS and an age key, which will get its own article.

Storage

Storage is the part that changed the most, and the history is a bit embarrassing. The Ceph RBD storage class from my 2017 post stayed in use until 2024, later with the Ceph CSI drivers. In 2023 I moved the media files to TrueNAS, and in 2024 I experimented Mayastor on three workers with a dedicated disk each, to have fast replicated volumes for databases. Ceph was removed in June 2024.

Mayastor did the job, but in the end was more complex than needed, and it’s coming from someone that had a Ceph cluster for a few years. In December 2025 I removed it as well.

Today everything comes from a TrueNAS server through two CSI drivers. The SMB CSI driver provides the smb storage class, used for most volumes.

Since SMB is not a good place for databases, I added a freenas-iscsi-csi storage class from democratic-csi, which creates a zvol on TrueNAS for every volume and mounts it over iSCSI as xfs.

The PostgreSQL databases themselves are managed by CloudNativePG.

Traefik, cert-manager and MetalLB

The cluster runs on my home network, so there is no cloud load balancer. MetalLB has been doing that since 2018. It announces addresses from a dedicated range in layer 2 mode:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.0.30.0-10.0.30.255
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - default-pool

Traefik is installed with its Helm chart and gets a fixed LoadBalancer IP from this pool. All the services use Traefik IngressRoute resources.

The certificates come from cert-manager and Let’s Encrypt. Since my DNS is hosted on Route53, I use the DNS-01 challenge, which has the big advantage of working for wildcard certificates and for hosts that are not reachable from the internet:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  acme:
    email: <your-email>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - dns01:
        route53:
          region: eu-central-1
          accessKeyIDSecretRef:
            name: prod-route53-credentials-secret
            key: access-key-id
          secretAccessKeySecretRef:
            name: prod-route53-credentials-secret
            key: secret-access-key

The sync wave makes sure that ArgoCD only creates the issuer once the cert-manager CRDs and webhook are up. The cert-manager application also needs ServerSideApply=true, because its largest CRD is bigger than the annotation size limit of a client-side apply.

In the Traefik values, a wildcard Certificate for *.lab.example.org is created and set as the default certificate, so most routes only need tls: {}:

traefik:
  tlsStore:
    default:
      defaultCertificate:
        secretName: lab-tls
  extraObjects:
    - apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: lab-main
      spec:
        secretName: lab-tls
        dnsNames:
          - '*.lab.example.org'
        issuerRef:
          name: letsencrypt-production
          kind: ClusterIssuer

Services without their own login are protected with the traefik-oidc-auth plugin against Keycloak.

For the few services that must be reachable from the internet, an frp client in the cluster dials out to a small public server and forwards ports 80, 443 and Forgejo’s SSH port to Traefik and Forgejo, so nothing has to be opened on my home router.

What runs there

There are a number of deployed applications, since this is a HomeLab, some are only ever used as deployment experiments.

The ones I use the most:

• Forgejo for my git repositories, including this one, behind Anubis to keep the scrapers away, and Woodpecker for the CI, running each step as a pod.
• Immich for the photos, Vaultwarden for the passwords and Keycloak for the logins.
• A Matrix server with Element for internal chat and chatbots.
• Monitoring with Grafana, Loki, Mimir, Tempo and Alloy.

Backups of the volumes are done with k8up to S3, which will also get its own article.

Conclusion

Compared to the 2017 cluster, the biggest change is not one tool but the fact that the whole lab is now described in one repository. The nodes come from three Talos files, and everything else comes from git through ArgoCD. When something breaks, I look at the diff instead of trying to remember what I typed.

There is still work to do, and that is part of the fun of a home lab. The backups run every night, but nothing checks automatically that they can actually be restored. And now that all the nodes share a single host, I would like the cluster to start workers when it needs them and stop them when it is idle, instead of keeping every virtual machine running all the time.