My homelab from scratch (Part 10): golden images, Git workflow, and four errors for a single download

In part 9, we consolidated the foundations: OpenTofu, an encrypted state and a CI that runs with the runner inside the cluster. For this phase, I’d promised golden images built with Packer. Except… I changed my mind along the way, and that’s one of the two running threads of this article.

This phase actually does two distinct but related things. First, I take the opportunity to switch the project over to a real branch workflow, the one that finally makes the CI’s tofu plan useful. Then it builds the base images we’ll need for the upcoming VMs: a Debian 13 cloud-init template and a custom Talos image, available on all three PVEs.

And as usual, it’s the failures that make the article. This time quite a few of them: four successive failures on one and the same resource, a simple image download. No tutorial shows this path: they all present the final .tf as if it had fallen from the sky. In practice, it was a lot more complicated than that.

Part 1: the Git workflow, or the plan that arrived too late

#

The realisation that kicks it all off

#

Until now, I was pushing straight to main, single branch. Pragmatic during the bootstrap: at the start of a project, you want to move forward, not waste time. But the previous phase introduced a pipeline with a tofu plan… and a plan that runs after the merge is useless.

The whole point of a plan is to see what’s going to change before you decide. Pushed to main, it dutifully showed me what I’d already just applied. Might as well be a weather forecast for yesterday.

The chosen workflow: simplified GitLab Flow

#

The fix comes down to two rules:

With:

• main = the state of the infrastructure, protected: no more direct pushes
• One branch per change (feat/golden-images, fix/zfs-mountpoint)
• Fast-forward merge → a strictly linear history

The plan now running on the MR, I read it before merging. The pipeline gets its meaning back.

The manual apply, a deliberate safeguard

#

The point left open at the end of the previous phase, I settle it here: the apply is a manual job, not automatic. A merge can destroy a VM through a simple coding mistake; the click leaves one last chance to read the plan before letting OpenTofu touch anything real.

And the good surprise is that on the pipeline side, there’s almost nothing to change. The rules written in part 9 already anticipated this behaviour: the tofu-plan triggers on merge_request_event, and the tofu-apply is already a manual job on main. So what was missing wasn’t the YAML, it was the usage. As long as I was pushing everything to main, those rules ran for nothing (the plan executed after the fact). By adopting branches and protecting main, the pipeline finally starts doing what it was written for.

The four traps of branch protection

#

It’s while locking down main that the trouble starts. Four hiccups, all genuine, all instructive.

1. The masked CI variable refuses multi-line.

failed to load age identities: no secret keys found

The keys.txt file generated by age-keygen contains three lines: two comments (# created:, # public key:) and the key itself. But GitLab’s Masked variables don’t accept multi-line, so the value arrived truncated, or even empty. The fix: copy only the key line.

2. Protected blocks all working branches.

A CI variable marked Protected is only available on… protected branches. My MR branch not being one, the job had no access to it, so no MR could ever run a plan. The whole workflow emptied of its meaning. The fix: untick Protected (the variable stays Masked in the logs, which protects what matters).

3. Locking yourself out with a wildcard.

An attempt to work around the previous trap: protect * (all branches). Result:

! [remote rejected] feat/golden-images -> feat/golden-images (pre-receive hook declined)

By protecting * with “Allowed to push: No one”, no push was possible anywhere anymore, including on the working branches. I’d locked myself out.

4. Fast-forward demands a rebase.

Merge blocked: Fast forward merge is not possible. Please rebase.
The source branch is 1 commit behind the target branch.

That’s the trade-off of fast-forward mode: a perfectly linear history, in exchange for a rebase as soon as main moves under your feet.

git fetch origin && git rebase origin/main && git push --force-with-lease

Note the --force-with-lease rather than --force: it refuses to push if the remote branch has changed in the meantime. A reflex to keep even when you’re alone: it’s the safety net that stops you overwriting a commit you’d forgotten about.

Alternative if the systematic rebase becomes a pain: the “Merge commit with semi-linear history” mode, which also requires the rebase but keeps an explicit trace of each MR. For now, strictly linear suits me fine.

Part 2: the golden images

#

Decision: I’m abandoning Packer

#

Here’s the moment I go back on my promise from part 9. To build the Debian template, two philosophies were competing:

Option A, Packer + ISO: Packer boots a VM from the netinst ISO, drives the installer via a preseed served over HTTP, installs cloud-init and the agent, then converts the whole thing into a template.

• Pros: full control over what’s “baked” into the image, and very educational
• Cons: 15 to 20 minutes per build and above all a fragile boot_command: Packer literally types on the keyboard in the VNC console to kick off the preseed. That’s the part that breaks most often

Option B, official cloud image + OpenTofu: Debian publishes genericcloud images in qcow2, already cloud-init ready. The bpg provider downloads them and turns them into a template without any Packer at all.

• Pros: a few minutes, official image, one less brick to maintain
• Cons: less control over what’s preinstalled

I chose option B.

A decision made along the way: Dokploy in a VM, not an LXC

#

A question posed for the next phase, but settled here because it determines the kind of image I need: the future application VM (Dokploy) will be a VM, not an LXC container.

The reason is precise: Dokploy relies on Docker Swarm, not just Docker. Swarm manipulates overlay networks, iptables, kernel modules: all things that work poorly or not at all in an unprivileged container. Even with nesting=true, you’d spend your time working around it.

The Debian 13 template

#

Concretely, two OpenTofu resources: the download of the cloud image, then the template VM that builds on it.

# tofu/stacks/core/images.tf (Debian portion)
locals {
  debian_image_url = "https://cloud.debian.org/images/cloud/trixie/${var.debian_image_snapshot}/debian-13-genericcloud-amd64-${var.debian_image_snapshot}.qcow2"
}

resource "proxmox_virtual_environment_download_file" "debian_cloud" {
  for_each = toset(var.cluster_nodes)

  node_name    = each.value
  content_type = "import"
  datastore_id = "local"

  file_name          = "debian-13-genericcloud-amd64-${var.debian_image_snapshot}.qcow2"
  url                = local.debian_image_url
  checksum           = var.debian_image_checksum
  checksum_algorithm = "sha512"
  overwrite          = false
}

resource "proxmox_virtual_environment_vm" "debian_template" {
  node_name = var.runner_node
  vm_id     = var.template_vm_id
  name      = "debian-13-cloudinit"
  template  = true
  started   = false

  agent { enabled = true }
  cpu {
    cores = 2
    type  = "host"        # expose the host CPU
  }
  memory { dedicated = 2048 }

  disk {
    datastore_id = var.storage_pool
    interface    = "scsi0"
    import_from  = proxmox_virtual_environment_download_file.debian_cloud[var.runner_node].id
    size         = 20
    discard      = "on"
    ssd          = true
  }

  network_device { bridge = "vmbr0" }

  # Clones will override, we leave DHCP on the template
  initialization {
    datastore_id = var.storage_pool
    ip_config { ipv4 { address = "dhcp" } }
    user_account {
      username = "debian"
      keys     = [trimspace(var.ssh_public_key)]
    }
  }

  operating_system { type = "l26" }
  serial_device {}

  lifecycle {
    ignore_changes = [disk[0].file_id]
  }
}

Why the checksum is essential. The URL already points to a dated snapshot (var.debian_image_snapshot) rather than latest/, but that’s not enough: at a constant URL, Debian could republish. The SHA512 checksum locks the exact content. Without it, OpenTofu could fetch a different image without batting an eye. With it, the slightest divergence of a single byte makes the download fail.

The other choices deserve a word:

• import_from: the template’s disk is imported directly from the cloud image downloaded on the same node (debian_cloud[var.runner_node].id). No manual conversion, OpenTofu chains download → import.
• serial_device {} + operating_system { type = "l26" }: Debian cloud images expect a serial console and a Linux 2.6+ kernel profile. Without the serial console, no usable output in Proxmox.
• cpu type = "host": exposes the host processor’s instructions. Essential for performance and for Talos/Kubernetes later.
• discard = "on" + ssd = true: TRIM is propagated all the way to the ZFS pool: the space freed inside the VM is actually returned to the storage.
• lifecycle { ignore_changes = [disk[0].file_id] }: avoids an untimely recreation of the template if the source file’s identifier changes.

The Talos image

#

Talos doesn’t download “raw”: you go through Sidero’s Image Factory, which produces a custom image with the requested extensions. Here, qemu-guest-agent, without which Proxmox knows neither the IP nor the real state of the VM.

curl -sX POST --data-binary @- https://factory.talos.dev/schematics << 'EOF'
customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/qemu-guest-agent
EOF
# -> {"id":"53513e54bb39202f35694412577a6bc53d484744d35a126e5d42ef34785c0d83"}

On the OpenTofu side, the resource builds the Image Factory URL from the schematic id and the version, and downloads the image on each node:

# tofu/stacks/core/images.tf (Talos portion)
locals {
  talos_image_url = join("/", [
    "https://factory.talos.dev/image",
    var.talos_schematic_id,
    var.talos_version,
    "nocloud-amd64.raw",
  ])
}

resource "proxmox_virtual_environment_download_file" "talos" {
  for_each = toset(var.cluster_nodes)

  node_name    = each.value
  content_type = "import"
  datastore_id = "local"

  # The name carries the schematic id: two variants can coexist during a switchover
  file_name = "talos-${var.talos_version}-${substr(var.talos_schematic_id, 0, 8)}-nocloud-amd64.raw"
  url       = local.talos_image_url
  overwrite = false

  upload_timeout = 3600        # 4.2 GB to download

  lifecycle {
    create_before_destroy = true
  }
}

Two details that trip you up:

• You have to take the nocloud-amd64 variant: it’s the one that includes the nocloud support the provider needs to inject the machine configuration under Proxmox.
• No VM template for Talos, unlike Debian. The Talos VMs will be created directly from the image, with their machine configuration injected. That’s the mode of operation Talos expects, as it has no notion of post-clone customisation: each node gets its config, full stop.

The images on the three nodes (for_each)

#

The problem is an old friend of this series: the local storage is not shared between nodes. My images only existed on pve01, but the next phase will want one Talos VM per physical node.

Three options:

• A. Download on all three nodes via for_each = toset(var.cluster_nodes). Redundant in space (~13 GB total), but negligible on 91 GB per node.
• B. Create on pve01 then migrate: adds a step, and it’s contrary to the declarative approach.
• C. Shared storage: clean, but I don’t have a NAS, and it adds a dependency. I’m still thinking seriously about it for a coherent evolution of the homelab.

I chose option A. The simplest, the most declarative, for a trivial cost.

tofu state mv 'proxmox_virtual_environment_download_file.talos' \
              'proxmox_virtual_environment_download_file.talos["pve01"]'

To close the determinism thread, here are the targets actually frozen for this phase:

# variables.tf + terraform.tfvars (pinned values)
debian_image_snapshot = "20260810-2566"   # a dated Debian snapshot, not "latest/"
debian_image_checksum = "0ce1f1d6...93dc" # SHA512 of the qcow2 above
talos_version         = "v1.13.7"
talos_schematic_id    = "53513e54...0d83" # deterministic hash (qemu-guest-agent)

The four errors of a single download (the technical heart)

#

Here’s the part promised in the intro. Four successive failures on a single resource: the image download. Frustrating to live through, but it’s exactly what the classic tutorials keep quiet about.

Error 1: PVEAdmin isn’t enough (HTTP 403)

#

HTTP 403 - Reason: Permission check failed
error retrieving URL metadata for "https://cloud.debian.org/..."

Investigation: pveum user permissions tofu@pve shows Sys.Audit, Sys.Console, Sys.Syslog… but not Sys.Modify, precisely the privilege required to download from a URL.

It stings a bit, because in part 9, I’d specifically added PVESysAdmin to the token thinking it brought Sys.Modify for exactly this moment. Big mistake: I hadn’t checked the actual content of the role. The inspection confirms it:

PVESysAdmin │ Sys.Audit,Sys.Console,Sys.Syslog

PVESysAdmin only grants read rights, despite its name. It never brought Sys.Modify: I’d stacked a useless role. The fix is a custom role:

pveum role add TofuExtra --privs 'Sys.Modify'
pveum acl modify / --users tofu@pve --roles TofuExtra

Error 2: iso vs import

#

scsi0: local:iso/...img has wrong type 'iso' - needs to be 'images' or 'import'

I’d switched the content_type from import to iso, “to be safe”. Bad idea: Proxmox distinguishes strictly between these categories, and a file typed iso is treated as a CD image, not a disk. Back to import, and enabling this content type on the storage (it isn’t by default):

pvesm set local --content iso,vztmpl,backup,import

Error 3: the rejected extension

#

HTTP 400 - (filename: invalid filename or wrong extension)

The import type only accepts disk format extensions (.qcow2, .raw, .vmdk), not .img. The .img came from a bad habit: many examples rename it that way because the old iso type refused .qcow2. With import, it’s exactly the opposite. Fix: keep the format’s real extension (.qcow2).

Error 4: no decompression for import

#

TASK ERROR: decompression not supported for import

The Talos image is served as .raw.xz. The bpg provider does expose a decompression_algorithm, but Proxmox only decompresses for the iso and vztmpl types, never for import. Fix: download the image already decompressed: the Image Factory serves both variants, you just have to remove the .xz from the URL.

The cost: 4.2 GB instead of much less in the compressed version, ×3 nodes ≈ 13 GB. Acceptable, and it keeps everything inside OpenTofu rather than adding an Ansible decompression step.

Where things stand after this phase

#

We now have what we need to cleanly build the VMs of the coming phases: a minimal official image, cloud-init for customisation and Ansible behind it for configuration. And a Git workflow that finally does what it was designed for: showing the changes before they touch anything real.

A point raised along the way

#

Along the way, a question forced itself on me: “if pve01 crashes, does the CI runner restart elsewhere?” The answer is no, and for two combined reasons:

1. HA isn’t configured: Proxmox only restarts a resource elsewhere if it’s declared in the high-availability manager.
2. Even with HA, it wouldn’t be enough: the container’s disk lives in pve01’s local ZFS pool. The other nodes have a pool of the same name, but empty. The data has to be there first, which is the role of ZFS replication.

So two bricks are missing, in this order: a pvesr job (replication) → an HA resource. With those, if pve01 is lost, the container would restart on pve02 from the last replicated snapshot.

And now?

#

For a CI runner, the resilience stakes are low (no data, pipelines that can wait). But it’s the perfect textbook case to validate the mechanism before applying it to workloads that actually matter.

Part 11 will therefore tackle resilience: ZFS replication, HA resource, and above all a real failover test, where we actually shut down pve01 and watch whether the service comes back up elsewhere. And right after, we’ll deploy Dokploy, the lab’s first real application service, before part 12 takes on Talos and Kubernetes.

See you soon!