My homelab from scratch (Part 13): DNS, certificates and ingress, finally a green padlock

Part 12 left us with a complete Kubernetes cluster: Talos, Cilium, Longhorn, ArgoCD, three Ready nodes and a working GitOps loop. Nice. Except that in practice, opening the ArgoCD interface meant a kubectl port-forward and an IP address. A cluster that was up and running, but not exactly usable.

The goal of this part fits in one line: type https://argocd.ktw.ovh into a browser and get a green padlock, without exposing anything to the Internet.

A word on what isn’t here

#

If you read the end of part 12, you were expecting something else. There, I announced a hefty programme for this part 13: Technitium, certificates, Traefik, but also Authentik for SSO, Netbird for remote access and the whole observability stack (Prometheus, Grafana, Loki). I owe you an explanation.

Once I laid the list out flat, this phase had accumulated everything postponed from the previous ones: DNS, ACME, ingress, SSO, VPN, metrics, logs, external monitoring, backups. Not one phase, three. And above all, a slightly awkward realisation: I’d built an interesting infrastructure but… it still wasn’t hosting anything. Rolling it all out in one go would have meant weeks more without ever serving a real service.

So I split it up again:

• here: DNS, certificates and ingress, the bare minimum to actually use what already exists
• next: identity and access
• then: observability

The part 12 teaser was, in short, too optimistic.

Four decisions before we start

#

Back to applying locally

#

In part 12, I was congratulating myself on finally applying from CI, “not from my machine”. That didn’t last. During the build, the pipeline turned out to be too slow: 17 minutes on a simple image download, without the slightest intermediate feedback, and debugging through CI logs is far less comfortable than a terminal.

So I backtracked: apply locally while I’m building, pipeline once the code settles down. The pipeline doesn’t go away for all that: the plan on MRs remains a portability test, the secret guardrails stay in place and the manual apply job is kept.

Technitium in a dedicated LXC, outside Kubernetes

#

For internal DNS, I’d picked Technitium back in part 5 for its complete API and native split-horizon. The remaining question was where to run it. Three options: a dedicated LXC, inside the Kubernetes cluster or on Dokploy.

I went with the dedicated LXC (ct:210, on pve03, at .40) for a simple reason: the building blocks you need when things go wrong mustn’t depend on the things that can go wrong. DNS inside the cluster creates a circular dependency: if Kubernetes goes down, you lose name resolution for everything, including the tools you’d use to diagnose the outage. It’s the same reasoning I used in part 5 for the backup server and the monitoring, both planned outside the cluster.

And unlike the Talos VMs from part 12, this container is under HA. No contradiction there: Technitium is a singleton service with no resilience mechanism of its own, not an etcd member that might come back with a stale view of the cluster. Proxmox HA is exactly the right tool, with the pattern already well rehearsed on the runner and Dokploy in part 11.

A Cilium LoadBalancer rather than a hostPort

#

Traefik needs an IP reachable from the LAN. Cilium can assign addresses to LoadBalancer services and announce them over ARP (the protocol that, on a local network, maps an IP to a MAC address): I’m reserving the .60 to .69 range for it. The alternative, a hostPort on the nodes, would have pointed DNS at one specific node and therefore broken the resilience we’ve just spent an entire part building.

Two ingresses, and that’s fine

#

Dokploy already ships its own Traefik, listening on ports 80 and 443 of its VM (we saw that in part 11). Deploying a Traefik inside Kubernetes therefore creates two separate entry points, one per orchestration layer. That’s not a problem: different IPs, separate responsibilities. But you need to keep it in mind when creating the DNS records.

Step 1: Technitium

#

The container

#

Nothing new under the sun on the OpenTofu side: an LXC, its replication to the other two nodes, its HA declaration and its affinity rule. It’s the part 11 pattern, applied for the third time:

# tofu/stacks/core/technitium.tf (excerpt)
resource "proxmox_virtual_environment_container" "technitium" {
  node_name = var.technitium_node
  vm_id     = var.technitium_vm_id

  unprivileged  = true
  start_on_boot = true

  initialization {
    hostname = "dns01"
    # ... (IP, gateway, SSH key)
  }

  # ... (1 core, 1 GB of RAM, 8 GB of disk, Debian template)

  lifecycle {
    ignore_changes = [node_name, started]
  }
}

# ... (replication to the other two nodes, HA declaration)

resource "proxmox_virtual_environment_harule" "technitium" {
  rule = "dns-home"
  type = "node-affinity"

  resources = [proxmox_virtual_environment_haresource.technitium.resource_id]
  nodes     = { (var.technitium_node) = 100 }

  strict  = false
  comment = "Managed by OpenTofu"
}

You’ll recognise the ignore_changes = [node_name, started] and the strict = false: those are the two lessons from part 11, which now apply without debate to anything that goes under HA.

The installation

#

To install Technitium, same reasoning as for Dokploy in part 11: the official script, wrapped by Ansible, which only runs it again if the software is missing. So the role checks for a file before doing anything:

# ansible/roles/technitium/tasks/main.yml (excerpt)
- name: Check whether Technitium is installed
  ansible.builtin.stat:
    path: /opt/technitium/dns/DnsServerApp.dll
  register: technitium_installed

The zone and the records, via the API

#

A second role then authenticates against the Technitium API, and creates the ktw.ovh primary zone and its A records using the uri module. All parameters go in the request body rather than in a hand-built URL:

# ansible/roles/technitium_config/tasks/main.yml (excerpt)
- name: Authenticate
  ansible.builtin.uri:
    url: "{{ technitium_config_api_url }}/user/login"
    method: POST
    body_format: form-urlencoded
    body:
      user: "{{ technitium_config_admin_user }}"
      pass: "{{ technitium_config_admin_password }}"
      includeInfo: "false"
    return_content: true
  register: technitium_config_auth
  changed_when: false
  no_log: true
  failed_when: technitium_config_auth.json.status != "ok"

# ... (forwarders, listing and creating the zone)

- name: Add A records
  ansible.builtin.uri:
    url: "{{ technitium_config_api_url }}/zones/records/add"
    method: POST
    body_format: form-urlencoded
    body:
      token: "{{ technitium_config_token }}"
      domain: "{{ item.name }}.{{ technitium_config_zone }}"
      zone: "{{ technitium_config_zone }}"
      type: A
      ttl: "300"
      overwrite: "true"
      ipAddress: "{{ item.ip }}"
    return_content: true
  loop: "{{ technitium_config_records }}"
  register: technitium_config_record_add
  changed_when: true
  failed_when: technitium_config_record_add.json.status != "ok"

With body_format: form-urlencoded, Ansible takes care of the encoding: a special character in a password can’t break the request, the way it would in a hand-assembled URL.

The records themselves are just a list in the role’s variables:

# ansible/roles/technitium_config/defaults/main.yml (excerpt)
technitium_config_forwarders:
  - 1.1.1.1
  - 9.9.9.9

technitium_config_records:
  - { name: pve01, ip: 192.168.3.11 }
  - { name: pve02, ip: 192.168.3.12 }
  - { name: pve03, ip: 192.168.3.13 }
  - { name: pbs01, ip: 192.168.3.20 }
  - { name: dns01, ip: 192.168.3.40 }
  - { name: runner01, ip: 192.168.3.50 }
  - { name: dokploy, ip: 192.168.3.51 }
  - { name: talos01, ip: 192.168.3.52 }
  - { name: talos02, ip: 192.168.3.53 }
  - { name: talos03, ip: 192.168.3.54 }
  - { name: k8s, ip: 192.168.3.55 }

Split-horizon, in concrete terms: a local primary zone makes Technitium authoritative for ktw.ovh. It answers for that entire domain itself, without ever asking the Internet. There’s a consequence you have to accept: any name under ktw.ovh that isn’t declared locally returns NXDOMAIN, even if it exists publicly. Since the domain is dedicated to the lab, that’s acceptable, but it’s something to keep in mind for the certificates.

Result

#

$ dig @192.168.3.40 pve01.ktw.ovh +short
192.168.3.11
$ dig @192.168.3.40 k8s.ktw.ovh +short
192.168.3.55
$ dig @192.168.3.40 gitlab.com +short
172.65.251.78

Lab names resolved privately, the rest of the Internet through the forwarders. First piece of the foundation in place.

Step 2: the certificates

#

The domain at OVH, the zone at Cloudflare

#

In part 5, I announced Let’s Encrypt certificates via the ACME DNS-01 challenge on the OVH API. A clarification is in order: the ktw.ovh domain was indeed bought from OVH, but its DNS zone is served by Cloudflare:

$ dig +short NS ktw.ovh
curt.ns.cloudflare.com.
khloe.ns.cloudflare.com.

So it’s Cloudflare that cert-manager has to drive, and that’s good news: cert-manager supports it natively. No external webhook to install (which OVH would have required), fewer moving parts, and code maintained by the project itself.

Two objects are needed: a secret holding the Cloudflare API token, and a ClusterIssuer, the certificate issuer usable from any namespace:

# tofu/stacks/core/certmanager.tf (excerpt)
resource "kubernetes_secret" "cloudflare_token" {
  metadata {
    name      = "cloudflare-api-token"
    namespace = kubernetes_namespace.cert_manager.metadata[0].name
  }

  data = {
    "api-token" = var.cloudflare_api_token
  }

  depends_on = [helm_release.cert_manager]
}

resource "kubernetes_manifest" "letsencrypt" {
  manifest = {
    apiVersion = "cert-manager.io/v1"
    kind       = "ClusterIssuer"
    metadata = {
      name = "letsencrypt"
    }
    spec = {
      acme = {
        email  = var.acme_email
        server = "https://acme-v02.api.letsencrypt.org/directory"
        privateKeySecretRef = {
          name = "letsencrypt-account-key"
        }
        solvers = [{
          dns01 = {
            cloudflare = {
              apiTokenSecretRef = {
                name = kubernetes_secret.cloudflare_token.metadata[0].name
                key  = "api-token"
              }
            }
          }
        }]
      }
    }
  }

  depends_on = [kubernetes_secret.cloudflare_token]
}

The Cloudflare token follows the same path as every other secret in the series: stored in SOPS, exported as TF_VAR_cloudflare_api_token by tofu/env.sh, never written in plain text in the code.

Checking propagation against public resolvers

#

Before asking Let’s Encrypt for the certificate, cert-manager checks for itself that the challenge TXT record is actually visible. By default, it does this through the cluster’s DNS resolution. But here, pods resolve ktw.ovh names through Technitium: CoreDNS, Kubernetes’ internal DNS, delegates the zone to it so that services can reach the lab machines.

Remember that sentence from step 1? Any name under ktw.ovh not declared locally returns NXDOMAIN. Technitium knows nothing about the challenge’s ephemeral records, which only exist at Cloudflare. Without any particular tweak, cert-manager would therefore wait forever for a TXT record that, from its point of view, doesn’t exist.

The solution: force cert-manager to check propagation against public resolvers, and only those:

# tofu/stacks/core/certmanager.tf (excerpt)
resource "helm_release" "cert_manager" {
  name       = "cert-manager"
  repository = "https://charts.jetstack.io"
  chart      = "cert-manager"
  version    = var.cert_manager_version
  namespace  = kubernetes_namespace.cert_manager.metadata[0].name

  timeout = 600

  values = [yamlencode({
    crds = { enabled = true }

    dns01RecursiveNameservers     = "1.1.1.1:53,8.8.8.8:53"
    dns01RecursiveNameserversOnly = true
  })]

  depends_on = [helm_release.cilium]
}

Result: a certificate issued in 32 seconds.

subject=CN=*.ktw.ovh
issuer=C=US, O=Let's Encrypt, CN=YR2
notAfter=Nov  1 19:46:53 2026 GMT

A wildcard certificate (*.ktw.ovh), valid for 90 days and automatically renewed 30 days before expiry. And that’s the whole point of this chain: a wildcard can only be obtained via DNS-01, never via HTTP-01. It’s also what makes it possible to have valid certificates without any service being reachable from the Internet, as promised in part 5.

Step 3: the Cilium LoadBalancer

#

For Traefik to get an address on the LAN, Cilium needs to learn two things: which addresses it can hand out and how to announce them. Two Cilium objects, declared in the same stack:

# tofu/stacks/core/cilium_lb.tf
resource "kubernetes_manifest" "cilium_ip_pool" {
  manifest = {
    apiVersion = "cilium.io/v2alpha1"
    kind       = "CiliumLoadBalancerIPPool"
    metadata = {
      name = "lab-pool"
    }
    spec = {
      blocks = [{
        start = var.lb_pool_start
        stop  = var.lb_pool_stop
      }]
    }
  }

  depends_on = [helm_release.cilium]
}

resource "kubernetes_manifest" "cilium_l2_policy" {
  manifest = {
    apiVersion = "cilium.io/v2alpha1"
    kind       = "CiliumL2AnnouncementPolicy"
    metadata = {
      name = "lab-l2"
    }
    spec = {
      interfaces      = ["eth0"]
      externalIPs     = true
      loadBalancerIPs = true
    }
  }

  depends_on = [helm_release.cilium]
}

The pool runs from 192.168.3.60 to 192.168.3.69, and the announcement policy publishes those addresses over ARP on eth0. The feature also has to be enabled in the Cilium chart itself. These lines are added to the values shown in part 12.

# tofu/stacks/core/cilium.tf (addition to the Helm values)
    l2announcements = {
      enabled = true
    }

    k8sClientRateLimit = {
      qps   = 50
      burst = 200
    }

Raising k8sClientRateLimit isn’t just for show: L2 announcements rely on electing the announcing node through Kubernetes leases, which generates a lot of API calls.

Restarting the Cilium pods

#

A subtlety along the way: updating the chart does modify Cilium’s ConfigMap (you’ll find enable-l2-announcements: true in it), but it doesn’t recreate the pods that read it. So L2 announcements only kick in after a restart:

kubectl -n kube-system rollout restart ds/cilium

To test it, a simple nginx exposed as a LoadBalancer: it gets 192.168.3.60, a cilium-l2announce-default-nginx-test lease shows up on talos02 (the node that won the election and carries the announcement) and curl answers 200.

Ping proves nothing

#

A useful clarification: ping 192.168.3.60 doesn’t answer, and that’s normal. No interface actually holds that IP. Cilium announces an ARP entry that draws traffic to a node, then its eBPF load balancer only handles TCP aimed at the exposed ports. ICMP isn’t its business.

Step 4: Traefik

#

A single entry point on .60, the wildcard certificate served by default and a systematic HTTP to HTTPS redirect. First the certificate:

# tofu/stacks/core/traefik.tf (excerpt)
resource "kubernetes_manifest" "traefik_wildcard" {
  manifest = {
    apiVersion = "cert-manager.io/v1"
    kind       = "Certificate"
    metadata = {
      name      = "wildcard"
      namespace = kubernetes_namespace.traefik.metadata[0].name
    }
    spec = {
      secretName = "wildcard-tls"
      issuerRef = {
        name = "letsencrypt"
        kind = "ClusterIssuer"
      }
      commonName = "*.${var.lab_domain}"
      dnsNames = [
        var.lab_domain,
        "*.${var.lab_domain}",
      ]
    }
  }

  depends_on = [kubernetes_manifest.letsencrypt]
}

Kubernetes secrets live in a namespace. Rather than copying the secret for the certificate obtained in step 2 by hand, Traefik declares its own Certificate, which cert-manager issues directly into the traefik namespace. Then the chart:

# tofu/stacks/core/traefik.tf (excerpt)
resource "helm_release" "traefik" {
  name       = "traefik"
  repository = "https://traefik.github.io/charts"
  chart      = "traefik"
  version    = var.traefik_version
  namespace  = kubernetes_namespace.traefik.metadata[0].name

  timeout = 600

  values = [yamlencode({
    service = {
      type = "LoadBalancer"
      annotations = {
        "lbipam.cilium.io/ips" = var.traefik_ip
      }
    }

    ports = {
      web = {
        http = {
          redirections = {
            entryPoint = {
              to     = "websecure"
              scheme = "https"
            }
          }
        }
      }
    }

    tlsStore = {
      default = {
        defaultCertificate = {
          secretName = "wildcard-tls"
        }
      }
    }

    # ... (dashboard disabled, CRD and Ingress providers enabled)
  })]

  depends_on = [
    kubernetes_manifest.traefik_wildcard,
    kubernetes_manifest.cilium_ip_pool,
  ]
}

The lbipam.cilium.io/ips annotation pins the address to 192.168.3.60. Without it, Cilium would pick one at random from the range, and DNS would be pointing at thin air after the first redeployment.

Two chart subtleties in these values: the HTTP to HTTPS redirect is declared under web.http.redirections.entryPoint, and TLS on websecure is enabled by default, no need to ask for it. The chart also validates its values against a JSON schema: a misplaced property is rejected before anything gets deployed, with its exact name.

The DNS wildcard

#

On the Technitium side, a single line added to the list from step 1:

# ansible/roles/technitium_config/defaults/main.yml (addition)
  - { name: "*", ip: 192.168.3.60 }

Twelve records in total, and this is the one that changes everything day to day: adding a Kubernetes service no longer requires any DNS change. You declare the route, the name resolves on its own. And the two ingresses from earlier coexist without a hitch: dokploy.ktw.ovh has its own explicit record pointing to .51, which takes precedence over the wildcard.

The routes

#

Three services to expose for starters: ArgoCD, the Longhorn interface and Hubble. One Traefik IngressRoute for each, generated from a single list:

# tofu/stacks/core/ingress.tf
locals {
  ingress_routes = {
    argocd = {
      namespace = "argocd"
      service   = "argo-cd-argocd-server"
      port      = 80
    }
    longhorn = {
      namespace = "longhorn-system"
      service   = "longhorn-frontend"
      port      = 80
    }
    hubble = {
      namespace = "kube-system"
      service   = "hubble-ui"
      port      = 80
    }
  }
}

resource "kubernetes_manifest" "ingress_routes" {
  for_each = local.ingress_routes

  manifest = {
    apiVersion = "traefik.io/v1alpha1"
    kind       = "IngressRoute"
    metadata = {
      name      = each.key
      namespace = each.value.namespace
    }
    spec = {
      entryPoints = ["websecure"]
      routes = [{
        match = "Host(`${each.key}.${var.lab_domain}`)"
        kind  = "Rule"
        services = [{
          name = each.value.service
          port = each.value.port
        }]
      }]
      tls = {}
    }
  }

  depends_on = [helm_release.traefik]
}

The empty tls = {} isn’t an oversight: it tells Traefik to terminate TLS with the tlsStore’s default certificate, i.e. the wildcard. It’s also where a loop opened in part 12 finally closes: ArgoCD was running with server.insecure, “because TLS will be terminated by the ingress”. Here we are: Traefik presents the certificate and ArgoCD stays on HTTP behind it, on port 80 of its service. Same goes for Hubble, enabled in part 12 and finally reachable some other way than through a port-forward.

The final test

#

curl -s -o /dev/null -w '%{http_code}\n' https://argocd.ktw.ovh

Note the absence of -k. Without that option, curl refuses any certificate it can’t validate. If it accepts the connection, it means the whole chain works: DNS resolves the name to .60, Cilium announces the address, Traefik answers and presents a valid Let’s Encrypt certificate for that name. That’s the real end-to-end test.

Then, just for the pleasure of it, in the browser: https://argocd.ktw.ovh, green padlock, no warnings. Goal for this part achieved.

Where things stand after this phase

#

The cluster’s services are now reached with a name and a padlock, no longer with an IP and a port-forward.

Loose ends

#

• My machine resolves everything through Technitium, hard-coded. If the container goes down (while HA restarts it elsewhere), I lose all name resolution, Internet included. Two options: have the network’s DHCP hand out Technitium first and a public resolver second, or run a second Technitium as a replica.
• One more local account. The Technitium admin password lives in SOPS, until SSO replaces it.
• A minor idempotency debt. The Add A records tasks have changed_when: true hard-coded, so they report a change on every run. No real consequence, since the API works with overwrite=true, but it isn’t idempotent in the strict sense.

What now?

#

Names that resolve and valid certificates: exactly what was missing for what comes next. But first, a break on the hardware side: the next part leaves code behind for plastic, with the 10-inch rack I 3D printed to house this whole little crew. After that comes identity and access: Authentik for SSO (Single Sign-On, one login for every service) over OIDC, then self-hosted Netbird for remote access, finally settling the chicken-and-egg question raised in part 12. This time, I’ll be careful not to promise anything more.

See you soon!