My homelab from scratch (Part 11): high availability, Dokploy and two authorities for one resource

In part 10, we built the base images and adopted a proper branch workflow. Right at the end, one question was left hanging: “if pve01 crashes, does the CI runner restart somewhere else?” The answer was no, and this phase 5 is the one that turns that “no” into “yes, in 1 min 14, proven with a stopwatch in hand”.

Two topics: first making a service genuinely resilient (ZFS replication + high availability, validated by a real-world failover test where we actually power off a node), then deploying Dokploy, the lab’s first real application service.

And the running theme: when a resource comes under HA, two attributes stop belonging to the code: where it runs, and whether it runs. OpenTofu and the HA manager become two authorities over the same resource, and you have to teach them not to step on each other.

High availability isn’t a checkbox

#

Let’s go back to the starting question: why doesn’t the runner survive the loss of pve01? For two compounding reasons:

1. HA isn’t configured. Proxmox only restarts a resource elsewhere if it’s declared in its high-availability manager. Without that, a container (or a VM for that matter) on a dead node stays dead.
2. Even with HA, it wouldn’t be enough. The container’s disk lives in pve01’s local ZFS pool. The other nodes do have a pool with the same name, but it’s empty. Restarting the container on pve02 without its data leads nowhere.

So we’re going to build the chain, link by link, on the runner, which makes an excellent test bench: no critical data, and pipelines that can wait.

Part 1: making the runner resilient

#

ZFS replication

#

First link: the data has to exist elsewhere before the failure. That’s the job of ZFS replication, which regularly sends incremental snapshots of the local pool to the other nodes. In OpenTofu:

# tofu/stacks/core/runner_ha.tf (replication)
locals {
  # Every node except the one hosting the runner
  runner_replica_targets = [for n in var.cluster_nodes : n if n != var.runner_node]
}

resource "proxmox_virtual_environment_replication" "runner" {
  for_each = { for idx, node in local.runner_replica_targets : node => idx }

  id       = "${proxmox_virtual_environment_container.gitlab_runner.vm_id}-${each.value}"
  type     = "local"
  target   = each.key
  schedule = var.replication_schedule
  comment  = "Managed by OpenTofu"
}

Two jobs, one per destination node: the local computes “all nodes except the runner’s” and the for_each creates one job per target. For the runner on pve01, that gives a 200-0 job to pve02 and a 200-1 to pve03. Replication is one-way, so you do need one copy per destination.

The replication_schedule variable defaults to */15 (every 15 minutes): it’s the one that defines the RPO (Recovery Point Objective), worst case you lose 15 minutes of data. It’s a trade-off to make, but it’s more than enough for a CI runner.

tofu providers schema -json \
  | jq '.provider_schemas[].resource_schemas["proxmox_virtual_environment_replication"].block.attributes'

Declaring the service to HA

#

Second link: telling the cluster that this container must be managed under high availability. That’s the job of the haresource resource, the link that, concretely, puts ct:200 under the HA manager’s responsibility:

# tofu/stacks/core/runner_ha.tf (HA declaration)
resource "proxmox_virtual_environment_haresource" "runner" {
  resource_id = "ct:${proxmox_virtual_environment_container.gitlab_runner.vm_id}"
  type        = "ct"
  state       = "started"

  max_restart  = 1
  max_relocate = 1
  comment      = "Managed by OpenTofu"

  depends_on = [proxmox_virtual_environment_replication.runner]
}

The state = "started" says “HA must keep this service running”, and the depends_on guarantees that replication exists before the resource comes under HA: no point declaring a service resilient when its data isn’t anywhere else yet.

Failover test #1: without an affinity rule

#

At this stage, the minimal chain is in place: the data is replicated, the service is declared to HA. But no node preference is set yet. Let’s test this foundation: we power off pve01 for real and watch the CRM logs (the cluster resource manager):

Failover in 1 min 10. Service up on pve02, same IP, same hostname, up 0 min. The chain works.

But: when pve01 comes back up, the container stays on pve02. That’s intentional on the Proxmox side (a failback would be a second interruption), but it raises a real problem on the infrastructure-as-code side. Which one? That’s the next snag.

Two authorities for one resource

#

With the container on pve02 while the code says pve01, a simple tofu plan gives you cold sweats:

# proxmox_virtual_environment_container.gitlab_runner has been deleted
Plan: 2 to add, 0 to change, 1 to destroy

OpenTofu proposes to recreate the container on pve01. Applying would have destroyed the running runner, or created an ID conflict.

The cause is subtle: the bpg provider identifies a resource by the node + vmid pair. The state says “CT 200 on pve01”, the provider looks on pve01, finds nothing… and concludes the resource has been deleted.

The workaround is elegant: a node preference rule (node-affinity), which we’ll set up right after. Since the service will come back to its preferred node on its own, reality converges towards what the code declares. The state drift becomes temporary (just for the duration of the return) instead of permanent. We don’t fix the provider limitation, we organise the infra so it never surfaces.

HA groups no longer exist in PVE 9

#

What’s left is to set this node preference, so the runner comes home (pve01) as soon as it can. First attempt with proxmox_virtual_environment_hagroup… and immediate failure:

HTTP 500 - Reason: cannot create group: ha groups have been migrated to rules

Proxmox VE 9 has replaced HA groups with a rule system. The provider still exposes hagroup for older clusters, but a 9.2 cluster flatly refuses it. The replacement resource is proxmox_virtual_environment_harule, with two types: node-affinity (node preference) and resource-affinity (keeping services together or apart). It’s the first one that replaces the group:

# tofu/stacks/core/runner_ha.tf (affinity rule)
resource "proxmox_virtual_environment_harule" "runner" {
  rule = "runner-home"
  type = "node-affinity"

  resources = [proxmox_virtual_environment_haresource.runner.resource_id]

  nodes = {
    (var.runner_node) = 100
  }

  strict  = false
  comment = "Managed by OpenTofu"
}

Another difference from the old system: no more no_failback. Returning to the preferred node is the default behaviour of affinity rules, exactly what we were after.

Failover test #2: with an affinity rule

#

We replay the failure, affinity rule in place this time.

Failover (pve01 goes down):

→ 1 min 14. Worth noting: the CRM waits 50 seconds between detection and fencing. That’s deliberate: you don’t want to move a service over a fleeting network blip of a few seconds.

Automatic return (pve01 comes back):

→ 2 min 06, of which ~1 min 45 was transfer and restart.

Bottom line: downtime of about 1 min 15, full convergence in about 3 min. For a homelab, that’s excellent, and above all it’s measured, not assumed.

started also belongs to HA

#

One last trap, caught during the ~1 min 45 startup on pve01. A tofu plan run at that precise moment showed:

~ started = false -> true
Plan: 0 to add, 1 to change, 0 to destroy

OpenTofu wanted to start a VM that HA was already starting. An apply at that moment would have put the two authorities into direct conflict. This exact case is the one that the lifecycle block already set on the runner back in part 9 was anticipating: added “just in case” at the time, here it finds its concrete justification, backed by a real incident:

# tofu/stacks/core/runner.tf (runner container lifecycle block, set in part 9)
lifecycle {
  ignore_changes = [node_name, started]
}

Part 2: Dokploy, the first real service

#

With the HA mechanism validated, we can deploy something that matters: Dokploy, an application deployment platform (self-hosted Heroku style), which will serve as the lab’s application foundation.

A VM, not an LXC

#

A decision already settled in part 10: Dokploy runs in a VM. It relies on Docker Swarm, which manipulates overlay networks, iptables and kernel modules, all poorly supported in an unprivileged container. Even with nesting=true, you’d spend your time working around it. LXC for simple services, VM as soon as there’s an orchestration layer inside.

Sizing and cloning

#

4 GB of RAM, 2 vCPUs, 40 GB of disk, IP 192.168.3.51, on pve02, to spread the load (the runner already occupying pve01). The VM is a clone of the Debian 9000 template created earlier:

# tofu/stacks/core/dokploy.tf (excerpt)
clone {
  # Template lives on runner_node; this clone crosses nodes
  node_name = var.runner_node
  vm_id     = proxmox_virtual_environment_vm.debian_template.vm_id
  full      = true
}

full = true: a full clone, not a linked one. A linked clone would permanently depend on template 9000 (which you then couldn’t delete, and which is attached to pve01). The full clone is self-contained, so migratable and replicable: essential for HA.

The agent you wait 15 minutes for

#

Here’s the most baffling snag of the phase. Creating the VM stayed stuck:

proxmox_virtual_environment_vm.dokploy: Still creating... [1m30s elapsed]

…even though the VM was already answering ping and SSH. It was up, but OpenTofu refused to consider the creation finished.

The cause: agent { enabled = true } is a promise made to the provider. It waits for the QEMU agent to respond in order to fetch the VM’s IP, with a 15-minute timeout. But the official cloud images don’t ship qemu-guest-agent. The promise could therefore never be kept. Worse: this hang also affected the refresh, so even a simple tofu plan stayed suspended.

The clean fix: install the agent via cloud-init, which benefits all future VMs:

# tofu/stacks/core/cloud_init.tf
locals {
  vendor_data = <<-EOT
    #cloud-config
    package_update: true
    packages:
      - qemu-guest-agent
    runcmd:
      - systemctl enable --now qemu-guest-agent
  EOT
}

resource "proxmox_virtual_environment_file" "vendor_data" {
  for_each = toset(var.cluster_nodes)

  node_name    = each.value
  content_type = "snippets"
  datastore_id = "local"

  source_raw {
    data      = local.vendor_data
    file_name = "vendor-data-agent.yaml"
  }
}

On the VM side, this snippet is wired in via initialization { vendor_data_file_id = ... }: the Dokploy VM points to the copy of the snippet present on its own node.

Prerequisite, as with import in part 10: enable the snippets content on the storage.

pvesm set local --content iso,vztmpl,backup,import,snippets

Result: recreating the VM, 1 min 45 of creation (the time for cloud-init to install the package), then qemu-guest-agent: active with no intervention. The path is validated from scratch.

Installing Dokploy: the official script, wrapped by Ansible

#

Two possible approaches to installing Dokploy:

• Reimplement each step as Ansible tasks, seemingly “cleaner”.
• Use the official install script (curl | sh), a black box.

I went with the official script, wrapped by Ansible. The role checks the state before acting and only (re)runs the installer if Dokploy is absent from the Swarm:

# ansible/roles/dokploy/tasks/main.yml (excerpt)
- name: Check whether Dokploy is already deployed
  ansible.builtin.command: docker service ls --filter name=dokploy --format '{{ "{{" }}.Name{{ "}}" }}'
  register: dokploy_service
  changed_when: false
  failed_when: false
  when: dokploy_docker.rc == 0

- name: Download installer
  ansible.builtin.get_url:
    url: "{{ dokploy_install_url }}"
    dest: /tmp/dokploy-install.sh
    mode: "0755"
  when: dokploy_service.stdout | default('') is not search('dokploy')

- name: Run installer
  ansible.builtin.command: /tmp/dokploy-install.sh
  when: dokploy_service.stdout | default('') is not search('dokploy')
  changed_when: true

- name: Wait for the web interface
  ansible.builtin.wait_for:
    port: "{{ dokploy_port }}"
    delay: 5
    timeout: 300

Two minor snags along the way:

• The prolonged silence of the command module: Ansible only reports the output at the end of the command, so 5 to 10 minutes with no way of knowing whether the installer is working or stuck. Acceptable for a third-party installer; for your own code, you’d break it into visible steps.
• The Jinja2 / Docker conflict: Docker’s --format '{{ ".Name" }}' (shown above) fails if you write it naively, because Ansible interprets the {{ }} before Docker does. Hence the '{{ "{{" }}.Name{{ "}}" }}' escaping in the role. In an ad-hoc command, it’s better to avoid Docker templates altogether.

The result, once the installer has run:

ID    NAME               MODE         REPLICAS   IMAGE
...   dokploy            replicated   1/1        dokploy/dokploy:v0.29.13
...   dokploy-postgres   replicated   1/1        postgres:16

CONTAINER   IMAGE            STATUS             PORTS
...         traefik:v3.6.7   Up 2 minutes       80->80, 443->443
...         dokploy          Up (healthy)       3000->3000
...         postgres:16      Up 5 minutes       5432

Traefik is already listening on ports 80 and 443, which will come in handy as soon as we get to the certificates phase.

The first deliberate departure from “everything as code”

#

Let’s say it plainly: the Dokploy admin account is created by hand, in the web interface.

A reading trap: available, not used

#

Proxmox was showing 100% RAM used on the Dokploy VM. Panic? No. Seen from the inside:

              total   used   free   buff/cache   available
Mem:          3.8Gi   1.3Gi  176Mi  2.6Gi        2.5Gi

Linux deliberately fills free RAM with disk cache, releasable instantly the moment an application needs it. Proxmox, for its part, sees touched pages without distinguishing cache from the rest, hence the false 100%.

HA on Dokploy: the pattern applies

#

What’s left is putting Dokploy under HA. And here’s the good news: it’s the pattern already validated on the runner, give or take a few parameters: type = "vm", resource_id = "vm:201" and the preferred node pve02.

First replication: 66 seconds per target for a 40 GB VM (~3.3 GB actually used). Subsequent replications only transfer the changed blocks: a few seconds. Sustainable even on 1 GbE. (Minor snag: pvesr takes a global lock during a replication, can't lock file '/var/lock/pvesr.lck'. That’s normal contention, not an error.)

Final state of the two services under HA:

service ct:200 (pve01, started)
service vm:201 (pve02, started)

Two highly available services, on two different nodes, each replicated to the other two.

A common confusion: HA across 3, replication to 2

#

“So we’re only doing HA between 2 hosts?” No, and it’s a classic confusion. These are two distinct mechanisms:

• Replication creates two jobs (one per destination), because they’re one-way copies.
• HA has only one resource, and can restart the service anywhere, since the data is already everywhere.

Only the simultaneous loss of two nodes would be fatal, but the cluster would lose quorum anyway at that point.

A design debt, identified but not yet paid

#

runner_ha.tf and dokploy_ha.tf are now nearly identical: same replication ×2, same haresource, same harule give or take a parameter or two. This is exactly the point where an OpenTofu module would start to make sense: a ha_service module parameterised by type, id and preferred node.

What is the pipeline actually for today?

#

An honest observation to finish: the apply is still done locally, from my machine. The pipeline’s tofu-apply job never has anything to do. So what’s the CI for as it stands?

It brings two real things:

• The plan on a merge request = a portability test. That’s exactly what had revealed all the implicit dependencies on my workstation.
• The guardrails: sops-encrypted and secret_detection, which run every time.

What’s useless as it stands is the apply job, present “just in case”. We’ll fix that in part 12, when we actually move the apply into the pipeline.

Where things stand after this phase

#

Two genuinely resilient services, a failover mechanism that’s measured rather than assumed, and the lab’s first real application service running. But above all, a lesson I won’t forget any time soon: HA isn’t a checkbox, it’s a chain you only truly know once you’ve unplugged a node for real.

What’s next?

#

The lab has solid foundations and a first service. Part 12 tackles the big one: Talos and Kubernetes, three VMs combining control-plane and worker, Cilium for the network, Longhorn for storage, ArgoCD for GitOps. And, as promised, moving the workflow onto the pipeline: from there on, we apply from GitLab, no longer from my machine.

See you soon!