My homelab from scratch (Part 12): Talos and Kubernetes, the chicken and the egg are back

This is the most ambitious phase of the whole series: building a complete, usable Kubernetes cluster, with Talos as the OS, Cilium for networking, Longhorn for storage and ArgoCD for GitOps, all of it described as code. We start from the Proxmox cluster, the Talos images from part 10 and the resilient services from part 11, and we end up with three Ready nodes, tested persistent storage and a working GitOps loop.

And since this is also the phase where we finally switch over to the pipeline (the first apply triggered from GitLab, as promised), let’s just say things move.

The running theme, this time, is the bootstrap order. Almost every obstacle in this phase comes from the same place: a resource depends on something that doesn’t exist yet at the moment you declare it. The CRD that ArgoCD hasn’t created yet, the CNI that has to come after the cluster but before everything else, the Talos extensions that require an upgrade after the image has been downloaded. Declarative infrastructure-as-code assumes you can describe the final state, but some dependencies still impose a sequence.

Three decisions before we start

#

No Proxmox HA on the Talos VMs

#

Unlike the runner and Dokploy from part 11, the Talos VMs are neither replicated nor declared as HA. That may come as a surprise, given that we’ve just built a whole resilience mechanism. But here it would be not only useless, but dangerous:

1. Proxmox HA restarts a VM from the last replicated snapshot, up to 15 minutes behind. Bringing an etcd member back with a stale view of the cluster risks corrupting the quorum.
2. Kubernetes already does this job: lose a node, and the two remaining etcd members keep the quorum while the pods get rescheduled.
3. Stacking two resilience mechanisms that don’t talk to each other means risking that the lower one (Proxmox) breaks the higher one (Kubernetes).

The consequence is liberating: the Talos VMs are disposable. A lost VM is simply recreated from OpenTofu and rejoins the cluster. That’s the very promise of Talos: an immutable, declarative OS.

We switch over to the pipeline

#

It’s official: the first real apply of this phase is triggered from GitLab, as announced at the end of part 11. The cycle set up in part 10 finally goes all the way: branch → MR → read the plan → merge → manual click on apply. We apply from CI now, not from my machine.

We stay inside the core stack

#

I did consider creating a separate OpenTofu stack for Talos. In the end, no: it would have meant shuttling values from core over to talos (outputs + data sources), for zero benefit. Splitting stacks makes sense when lifecycles diverge, or when several people work in parallel, and neither is the case here.

Step 1: the Talos VMs

#

Three VMs, one per physical node (one etcd member per machine). The list is generated from cluster_nodes:

# tofu/stacks/core/talos_vms.tf
locals {
  talos_nodes = {
    for idx, node in var.cluster_nodes : "talos0${idx + 1}" => {
      pve_node = node
      vm_id    = var.talos_vm_id_base + idx
      ip       = cidrhost(var.talos_subnet, var.talos_ip_offset + idx)
    }
  }
}

4 vCPUs, 8 GB of RAM, a 60 GB disk, IPs .52 to .54, and the image imported from the local node (the local storage isn’t shared, which is why the images were duplicated across all three nodes back in part 10).

Step 2: machine configuration and VIP

#

This is where Talos really starts to make sense. The cluster configuration comes down to five resources from the Talos provider, which can only exist in this order, each one depending on the previous:

1. talos_machine_secrets: generates the cluster PKI and tokens. Everything else depends on it.
2. talos_machine_configuration: produces the machine configuration shared by the three nodes, built from those secrets.
3. talos_machine_configuration_apply: pushes that configuration to each node, with its own patch (IP, network).
4. talos_machine_bootstrap: initialises etcd on a single node. The other two join.
5. talos_cluster_kubeconfig: retrieves the credentials needed to talk to the cluster.

First illustration of the running theme: no configuration without secrets, no bootstrap without an applied configuration, no kubeconfig without a bootstrapped cluster. Here’s the common configuration first:

# tofu/stacks/core/talos_config.tf (common configuration)
resource "talos_machine_secrets" "this" {}

data "talos_machine_configuration" "controlplane" {
  cluster_name     = var.talos_cluster_name
  cluster_endpoint = "https://${var.talos_vip}:6443"
  machine_type     = "controlplane"
  machine_secrets  = talos_machine_secrets.this.machine_secrets

  config_patches = [
    yamlencode({
      machine = {
        install = {
          disk = "/dev/sda"
        }
        # ... (Longhorn disk: see step 4)
      }
      cluster = {
        network = {
          cni = { name = "none" }
        }
        proxy = { disabled = true }
      }
    }),
  ]
}

Then the node-by-node apply, the bootstrap and the credential retrieval:

# tofu/stacks/core/talos_config.tf (per-node patch, bootstrap, credentials)
resource "talos_machine_configuration_apply" "nodes" {
  for_each = local.talos_nodes

  client_configuration        = talos_machine_secrets.this.client_configuration
  machine_configuration_input = data.talos_machine_configuration.controlplane.machine_configuration
  node                        = each.value.ip

  config_patches = [
    yamlencode({
      machine = {
        network = {
          interfaces = [{
            interface = "eth0"
            dhcp      = false
            addresses = ["${each.value.ip}/24"]
            routes    = [{ network = "0.0.0.0/0", gateway = var.runner_gateway }]
            vip       = { ip = var.talos_vip }
          }]
          nameservers = var.lab_dns_servers
        }
      }
    }),
  ]

  depends_on = [proxmox_virtual_environment_vm.talos]
}

resource "talos_machine_bootstrap" "this" {
  client_configuration = talos_machine_secrets.this.client_configuration
  node                 = local.talos_nodes["talos01"].ip

  depends_on = [talos_machine_configuration_apply.nodes]
}

resource "talos_cluster_kubeconfig" "this" {
  client_configuration = talos_machine_secrets.this.client_configuration
  node                 = local.talos_nodes["talos01"].ip

  depends_on = [talos_machine_bootstrap.this]
}

What matters in these two blocks:

• cni = { name = "none" } + proxy = { disabled = true } (common configuration): by default, Talos would install flannel and kube-proxy. We disable both, because Cilium is going to replace them. The accepted consequence: the nodes stay NotReady until the CNI is installed (step 3).
• The .55 VIP (for Virtual IP, a virtual IP address) appears twice. In the common configuration it’s the cluster_endpoint, the address everyone uses to reach the Kubernetes API, Cilium included (we’ll see that in step 3). In the per-node patch, it’s declared on eth0 of every control-plane, and Talos handles the election: only one node carries it at a time, and it migrates automatically if that node fails.
• Only one node bootstrapped (talos01). You never bootstrap more than one node, unless you fancy creating two competing clusters. And it’s the cascading depends_on that locks in the order of the five steps: without them, OpenTofu could kick off the bootstrap before the configuration has been applied.
• The secrets live in the state, encrypted client-side. That’s exactly the use case that justified choosing OpenTofu all the way back in part 5: the cluster’s entire PKI in a state file, unreadable without the passphrase.

Snag 1: Talos merges patches, it doesn’t replace them

#

* [networking.os.device.addresses] "eth0": invalid CIDR address: PLACEHOLDER/24

A design mistake on my part: I assumed the per-node patch would replace the interface list from the base patch. In reality, Talos merges the two, and my placeholder from the base patch survived the merge. The fix is a simple rule: the network configuration must exist in one place only (the per-node patch). That’s why, in the block above, the common configuration contains no network section at all.

Snag 2: the hostname already comes from cloud-init

#

* static hostname is already set in v1alpha1 config

Proxmox automatically generates cloud-init metadata containing the VM name as the hostname. Talos, in nocloud mode, reads it and already sets talos01, while my patch was trying to do the same thing again. The fix is the same as for the previous snag: take the hostname out of my patch and let cloud-init set it. Twice in a row, the same rule: one piece of information, one place.

Result: a 27-second boot

#

service[etcd](Running): Health check successful
rendered new static pod {"id": "kube-apiserver"}
enabled shared IP {"operator": "vip", "ip": "192.168.3.55"}
assigned address {"address": "192.168.3.55/32", "link": "eth0"}
sent gratuitous ARP
boot sequence: done: 27.301258027s

The VIP works: talos01 carries it and announces it over ARP. The cluster is up, but not usable yet: without a CNI, the nodes stay NotReady. That’s the job of the next step.

Step 3: Cilium

#

The nodes are NotReady, so they need a CNI (Container Network Interface). On Talos, Cilium needs a few specific settings:

# tofu/stacks/core/cilium.tf
provider "helm" {
  kubernetes {
    host                   = talos_cluster_kubeconfig.this.kubernetes_client_configuration.host
    client_certificate     = base64decode(talos_cluster_kubeconfig.this.kubernetes_client_configuration.client_certificate)
    client_key             = base64decode(talos_cluster_kubeconfig.this.kubernetes_client_configuration.client_key)
    cluster_ca_certificate = base64decode(talos_cluster_kubeconfig.this.kubernetes_client_configuration.ca_certificate)
  }
}

resource "helm_release" "cilium" {
  name       = "cilium"
  repository = "https://helm.cilium.io"
  chart      = "cilium"
  version    = var.cilium_version
  namespace  = "kube-system"

  values = [yamlencode({
    ipam = { mode = "kubernetes" }

    kubeProxyReplacement = true
    k8sServiceHost       = var.talos_vip
    k8sServicePort       = 6443

    securityContext = {
      capabilities = {
        ciliumAgent      = ["CHOWN", "KILL", "NET_ADMIN", "NET_RAW", "IPC_LOCK", "SYS_ADMIN", "SYS_RESOURCE", "PERFMON", "BPF", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
        cleanCiliumState = ["NET_ADMIN", "SYS_ADMIN", "SYS_RESOURCE"]
      }
    }

    cgroup = {
      autoMount = { enabled = false }
      hostRoot  = "/sys/fs/cgroup"
    }

    hubble = {
      enabled = true
      relay   = { enabled = true }
      ui      = { enabled = true }
    }

  })]

  depends_on = [talos_machine_bootstrap.this]
}

Before even looking at the Helm values, note the provider: it’s configured from the kubeconfig produced earlier. The running theme continues, but one level up: this isn’t a resource depending on another resource, it’s a provider depending on a resource created in the same repository.

The settings that matter:

• kubeProxyReplacement = true plus k8sServiceHost pointing at var.talos_vip: since there’s no kube-proxy left to route the kubernetes service, Cilium has to reach the API directly, through the VIP from step 2. Full circle.
• securityContext.capabilities: Talos grants no implicit privileges, so you have to enumerate the capabilities the agent needs. It’s verbose, but it’s exactly what you’d expect from an OS that locks everything down by default.
• cgroup.autoMount.enabled = false: Talos has already mounted the cgroup, and in any case Cilium couldn’t do it itself on a read-only filesystem.
• Hubble enabled (relay and ui): Cilium’s network observability, which will come in handy in the next part.

Snag: Talos taints control-planes by default

#

The cluster looks healthy… but hubble-relay and hubble-ui stubbornly stay Pending:

0/3 nodes are available: 3 node(s) had untolerated taint(s)
Taints: node-role.kubernetes.io/control-plane:NoSchedule

The pods that were working (cilium, coredns) managed it thanks to explicit tolerations. Hubble has none. Except I’d planned for combined nodes (control-plane + worker on the same machines): that’s the whole point of a 3-node cluster on 3 machines. One setting was missing from the Talos configuration:

# tofu/stacks/core/talos_config.tf (added to the common configuration)
cluster = {
  allowSchedulingOnControlPlanes = true
  # ... (network and proxy unchanged)
}

And as always with Talos, changing the common configuration isn’t enough: you have to reapply it to all three nodes, so back through talos_machine_configuration_apply you go. Every configuration fix walks the whole path again.

Once the configuration has been reapplied, hubble-relay and hubble-ui finally find somewhere to land, and the three nodes flip to Ready:

NAME      STATUS   ROLES           AGE   VERSION
talos01   Ready    control-plane   50d   v1.36.0
talos02   Ready    control-plane   50d   v1.36.0
talos03   Ready    control-plane   50d   v1.36.0

Output taken much later, hence the 50 days of uptime: the cluster has been running without a hiccup ever since. Note that all three nodes carry only the control-plane role: there’s no worker role to display, which is precisely the point of the combined nodes we’ve just unlocked.

This time the cluster really is usable, so we can trust it with storage.

Step 4: preparing the ground for Longhorn

#

A word first on the why. In Kubernetes, a pod is disposable: it can be killed, recreated or rescheduled onto another node at any moment, and everything it had written to its local disk goes with it. Fine for a stateless service, but as soon as an application needs to keep something (a database, files uploaded by users, a tool’s configuration), it needs a volume that outlives the pod and follows it when it moves machines.

That’s the job of a StorageClass and a CSI driver. Having no NAS (for now), I’m going with Longhorn: it turns the nodes’ own disk space into distributed block storage, replicates each volume across several machines and automatically attaches it wherever the pod lands. The storage therefore lives inside the cluster, with no external dependency.

That said, it needs three things the cluster doesn’t have: the iscsi-tools extension (attaching volumes over iSCSI), the util-linux-tools extension, and a writable path (the Talos root filesystem is read-only).

A new Talos schematic

#

Extensions are requested from the Image Factory, as in part 10, but with the enriched list:

curl -sX POST --data-binary @- https://factory.talos.dev/schematics << 'EOF'
customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/qemu-guest-agent
      - siderolabs/iscsi-tools
      - siderolabs/util-linux-tools
EOF
# -> {"id":"53513e54bb39202f35694412577a6bc53d484744d35a126e5d42ef34785c0d83"}

A new identifier, since the extension list has changed: that’s the whole point of a deterministic hash, a different configuration necessarily gives a different id.

The disk dedicated to Longhorn is a second 50 GB disk per VM (scsi1 → /dev/sdb), partitioned and mounted by Talos:

# tofu/stacks/core/talos_config.tf (added to the common configuration, machine section)
disks = [{
  device = "/dev/sdb"
  partitions = [{
    mountpoint = "/var/lib/longhorn"
  }]
}]
kubelet = {
  extraMounts = [{
    destination = "/var/lib/longhorn"
    type        = "bind"
    source      = "/var/lib/longhorn"
    options     = ["bind", "rshared", "rw"]
  }]
}

A choice: a dedicated disk rather than sharing the system disk, because it isolates the data and lets me resize it without touching the OS.

Major snag: OpenTofu destroys before it recreates

#

Changing the schematic changes the image URL → OpenTofu marks the images “must be replaced”. Except the new download failed with a timeout (4.2 GB × 3 nodes, and the Factory generates the image on first request):

timeout while waiting for task ... to complete, and the content type 'import'
is not supported by the Proxmox VE version 8.0.0

(The message is misleading: the “Proxmox VE 8.0.0” mention is a secondary error from the provider. The real cause is the timeout.) The result: the images were deleted from all three nodes without being replaced. The cluster was still running (the VMs already had their disks), but the source images no longer existed.

# The name carries the schematic: two variants can coexist
file_name = "talos-${var.talos_version}-${substr(var.talos_schematic_id, 0, 8)}-nocloud-amd64.raw"

upload_timeout = 3600

lifecycle {
  create_before_destroy = true
}

The Talos upgrade, deliberately kept out of the code

#

A crucial and counter-intuitive point: import_from only applies at creation time. Existing VMs keep the system disk built from the old image. Downloading the new image therefore isn’t enough: you have to upgrade the nodes one by one.

talosctl -n 192.168.3.54 upgrade \
  --image factory.talos.dev/nocloud-installer/53513e54bb39202f35694412577a6bc53d484744d35a126e5d42ef34785c0d83:v1.13.7

Result per node: 1 min 27, with automatic node drained then node uncordoned, because Talos coordinates its upgrade with Kubernetes. And an elegant detail: the schematic is then exposed as an extension, so you can see at a glance which configuration is running on each node.

NAME               VERSION
qemu-guest-agent   11.0.2
iscsi-tools        v0.2.0
util-linux-tools   2.42.2
schematic          53513e54bb39202f35694412577a6bc53d484744d35a126e5d42ef34785c0d83

Step 5: Longhorn

#

Time for the persistent storage itself:

# tofu/stacks/core/longhorn.tf
resource "kubernetes_namespace" "longhorn" {
  metadata {
    name = "longhorn-system"

    labels = {
      "pod-security.kubernetes.io/enforce" = "privileged"
      "pod-security.kubernetes.io/audit"   = "privileged"
      "pod-security.kubernetes.io/warn"    = "privileged"
    }
  }
}

resource "helm_release" "longhorn" {
  name       = "longhorn"
  repository = "https://charts.longhorn.io"
  chart      = "longhorn"
  version    = var.longhorn_version
  namespace  = kubernetes_namespace.longhorn.metadata[0].name

  timeout = 900

  values = [yamlencode({
    defaultSettings = {
      defaultDataPath     = "/var/lib/longhorn"
      defaultReplicaCount = 2
      defaultDataLocality = "best-effort"
    }

    persistence = {
      defaultClass             = true
      defaultClassReplicaCount = 2
    }

    csi = {
      kubeletRootDir = "/var/lib/kubelet"
    }

  })]

  depends_on = [helm_release.cilium]
}

• The privileged label is mandatory, on all three axes (enforce, audit, warn): Talos applies Pod Security Admission by default, and Longhorn needs elevated privileges to manage storage.
• defaultDataPath = "/var/lib/longhorn": that’s exactly the mount point prepared in step 4, on the second disk. The disk, the mount and Longhorn’s path are one and the same.
• defaultReplicaCount = 2 rather than 3: with three nodes, two replicas are enough to survive a failure while saving space.
• persistence.defaultClass = true: Longhorn becomes the cluster’s default StorageClass, so a PVC with no storageClassName lands on it.
• Version 1.11.3 rather than the more recent 1.12.0: Longhorn touches data, it’s the component where a regression costs the most. We tread carefully.

The test that counts

#

One PVC, plus a pod that writes a file into it → Bound, Running, file readable. The whole chain works, from the StorageClass to the actual mount. An observation along the way: the test pod triggers a Pod Security warning (restricted:latest), because Talos applies that profile by default on unlabelled namespaces, here in warn mode (non-blocking). Future deployments will have to comply with it, or live in a labelled namespace.

Step 6: ArgoCD and the app-of-apps pattern

#

The last link: the GitOps loop. The manifests live in the same repository (argocd/apps/), consistent with the monorepo choice: since the secrets are encrypted, read access doesn’t expose them. ArgoCD reaches it through a GitLab deploy token: read-only, limited to this repository, revocable, stored in SOPS and exported as TF_VAR_* by tofu/env.sh.

# tofu/stacks/core/argocd.tf (excerpt)
resource "helm_release" "argocd" {
  name       = "argo-cd"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  version    = var.argocd_version
  namespace  = kubernetes_namespace.argocd.metadata[0].name

  values = [yamlencode({
    global = {
      domain = var.argocd_domain
    }

    configs = {
      params = {
        "server.insecure" = true
      }
      repositories = {
        kentrowlab = {
          url      = var.gitlab_repo_url
          username = var.gitlab_deploy_username
          password = var.gitlab_deploy_token
        }
      }
    }

    dex = {
      enabled = false
    }

  })]

  depends_on = [helm_release.cilium]
}

Two choices that anticipate what’s coming: server.insecure, because TLS will be terminated by the ingress in part 13, and dex disabled for lack of an SSO provider for now. In other words, we avoid standing up a self-signed certificate and a homemade authentication setup that we’d have to tear down two weeks later.

The most structural snag of the phase

#

Error: API did not recognize GroupVersionKind from manifest (CRD may not be installed)
no matches for kind "Application" in group "argoproj.io"

The Application CRD only exists after ArgoCD is installed. And that’s the whole problem: depends_on can do nothing about it, because manifest validation happens before the plan runs.

First attempted workaround: declare the root application through the Helm chart’s additionalApplications option. Silent failure: the option was removed from recent versions. The chart installs, and… no application appears, without the slightest error.

The solution is a two-step bootstrap:

1. First apply → ArgoCD is installed, the Application CRD is created.
2. Second apply → the kubernetes_manifest can finally validate and create the root application.

# tofu/stacks/core/argocd.tf
resource "kubernetes_manifest" "root_app" {
  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"
    metadata = {
      name      = "root"
      namespace = kubernetes_namespace.argocd.metadata[0].name
    }
    spec = {
      project = "default"
      source = {
        repoURL        = var.gitlab_repo_url
        targetRevision = "main"
        path           = "argocd/apps"
      }
      destination = {
        server    = "https://kubernetes.default.svc"
        namespace = kubernetes_namespace.argocd.metadata[0].name
      }
      syncPolicy = {
        automated = {
          prune    = true
          selfHeal = true
        }
      }
    }
  }

  depends_on = [helm_release.argocd]
}

The syncPolicy deserves a word: with prune, whatever disappears from the repository disappears from the cluster, and with selfHeal, any change made by hand is pulled back to whatever the repository says. The repository is always right.

It’s the perfect illustration of the phase’s running theme: you’d like to describe everything in one go, but the bootstrap order has other ideas.

Result

#

NAME   SYNC STATUS   HEALTH STATUS
root   Synced        Healthy

The GitOps loop is up and running. ArgoCD reads the repository, the deploy token works, and above all: adding a file to argocd/apps/ will now be enough to deploy an application, without touching OpenTofu. The boundary between “infra” (OpenTofu) and “applications” (GitOps via ArgoCD) is drawn.

The Taskfile: regenerating the credentials

#

The kubeconfig and the talosconfig were sitting around in /tmp. Two options: store them in SOPS, or regenerate them on demand from the OpenTofu outputs. I went with regeneration:

# Taskfile.yml (excerpt)
tasks:
  kubeconfig:
    desc: Write kubeconfig from the encrypted OpenTofu state
    cmds:
      - mkdir -p {{dir .KUBECONFIG_PATH}}
      - cd tofu/stacks/{{.STACK}} && tofu output -raw kubeconfig > {{.KUBECONFIG_PATH}}
      - chmod 600 {{.KUBECONFIG_PATH}}
      - echo "export KUBECONFIG={{.KUBECONFIG_PATH}}"

  configs:
    desc: Refresh both cluster credentials
    cmds:
      - task: kubeconfig
      - task: talosconfig

One command does it all:

$ task configs
task: [kubeconfig] cd tofu/stacks/core && tofu output -raw kubeconfig > ~/.kube/kentrowlab.yaml
task: [kubeconfig] chmod 600 ~/.kube/kentrowlab.yaml
export KUBECONFIG=~/.kube/kentrowlab.yaml
task: [talosconfig] cd tofu/stacks/core && tofu output -raw talosconfig > ~/.talos/kentrowlab.yaml
task: [talosconfig] chmod 600 ~/.talos/kentrowlab.yaml
export TALOSCONFIG=~/.talos/kentrowlab.yaml

Both files come straight out of the encrypted state, in 600, and the task reminds you of the two export commands to run.

The state is already the source of truth; duplicating client certificates would create two places to revoke in case of a leak. (Minor snag: umask doesn’t exist in Task’s internal shell, mvdan/sh, a partial Go implementation. So it was replaced by an explicit chmod.)

Where things stand after this phase

#

A complete Kubernetes cluster, described in code, with modern networking, verified persistent storage and a GitOps loop. Plus a lesson that goes well beyond Kubernetes: declarative code describes a final state, but it doesn’t excuse you from understanding the order in which things have to come into existence.

What’s brewing for part 13

#

Two decisions are already locked in for what comes next:

• Generalised SSO. Without it, local accounts pile up (Proxmox, Dokploy, ArgoCD, Grafana, Longhorn), each with its own password in SOPS. The OIDC provider is leaning towards Authentik (lighter than Keycloak, and it doubles as an authentication proxy). Proxmox, ArgoCD, Grafana and Netbird all support OIDC natively.
• Remote access through self-hosted Netbird: open source WireGuard, which integrates with the OIDC IdP.

What now?

#

The lab has a Kubernetes cluster worthy of the name. Part 13 tackles exposure and services: Technitium for split-horizon DNS, Let’s Encrypt certificates via ACME DNS-01 (OVH API), Traefik as the ingress, Authentik for SSO, Netbird for remote access, and finally observability (Grafana, Prometheus, Loki). In short, everything that turns a cluster into a genuinely usable platform.

See you soon!