Leaving the Hashistack

Why

For years, the Hashicorp stack was the heart of my container setup. Nomad scheduled the containers, Consul knew where they were running and Vault kept the secrets. I learned a lot with it, and it was fun to experiment with.

But when I took an honest look at what the stack really did for me, the result was a bit sobering:

• Nomad was a better systemd. Nearly every workload was pinned to exactly one host, mostly via attr.unique.hostname. The scheduler had nothing to schedule.
• Consul was service discovery for one customer. Basically only VictoriaMetrics asked Consul which services are around to scrape.
• Vault held one certificate (well, the wildcard certificate plus six real secrets).

For this, I ran three clustered services on three dedicated VMs, with their own updates, backups, unseal procedure, keepalived and monitoring.
And there was the old chicken-and-egg problem: my images come from the GitLab registry, and the registry sits behind Traefik. If Traefik goes down during a Nomad deployment, nothing can be pulled anymore, so I had a "prefetch" workaround in front of every deployment.

So the question I asked on a Monday afternoon was quite simple: "Can I deploy Traefik completely with Ansible?"
Five weeks later, the whole stack was gone.

Starting Point

what	before
container projects	26 Git repos, each builds an image and deploys it as a Nomad job
runtime	Docker on five container hosts
orchestration	Nomad + Consul servers on three VMs, clients on every container host
secrets / certificate	Vault (Raft), rendered to the hosts by consul-template
service discovery	Consul catalog, plus a DNS zone rendered from Consul on my DNS server
deployment code	the same Ansible role code copied 26 times
health	Consul checks with check_restart, Docker itself only marks "unhealthy"

The last line is the reason why I didn't simply go back to Docker. A Docker HEALTHCHECK marks a container as unhealthy...and that's it. The restart policy only reacts to the exit of PID 1, never to a failed health check.

Target

• no Nomad, no Consul, no Vault - completely, not "just a bit less"
• every container as a plain systemd unit via Podman Quadlet, with HealthOnFailure=restart as replacement for Nomad's check_restart
• deployment by Ansible, from the project repo itself
• one place that knows which container runs where, in which version
• secrets in ansible-vault, certificates from an ACME job in GitLab CI
• everything must still start when the registry or GitLab is down

How

Three Layers

The basic idea was to separate three things which were mixed up before:

layer	repo	content
system	my existing Ansible repo	OS, packages, Podman, firewall, network
placement	container-deployment	which unit runs on which host, in which version, with which RAM
project	the 26 project repos	build the image, the Quadlet template, the service config

The placement repo is the new "Consul": one small YAML file per unit. Ports are declared there and checked for collisions per host before anything touches a host, and so are memory limits against the budget of each host. Quadlet has no scheduler, so this has to happen at render time.

The 26 copies of the role code went into an Ansible collection, which every project pins by tag. That was the very first step, and it paid off every single day afterwards: a fix in the collection is a new tag, not 26 commits.

Before, the way from a project repo to a running container went through all three tools:

flowchart LR
    P1[project repo] -->|nomad job run| N[Nomad servers<br/>3 VMs]
    N -->|schedules| C1[Nomad client]
    V[Vault] -->|consul-template| C1
    C1 --> D[Docker container]
    D -->|registers| K[Consul catalog]
    K --> M1[VictoriaMetrics]
    K --> Z[DNS zone]

After, Ansible writes a file and systemd does the rest:

flowchart LR
    P2[project repo] -->|./run.py deploy| A[Ansible + collection]
    PL[placement repo] --> A
    AV[ansible-vault] --> A
    AC[ACME job] -->|certificates| A
    A -->|renders| Q[Quadlet unit]
    Q -->|systemd| PC[Podman container]
    PL -->|scrape targets| M2[VictoriaMetrics]

Small Steps, Each One Reversible

I didn't want a big bang, so the path was split into steps that could be rolled back on their own:

1. Move the data. All Docker volumes moved to host paths under one directory per service, still under Docker and Nomad.
2. Docker to Podman, Nomad stays. Instead of a new Nomad driver, Nomad's existing Docker driver simply talks to Podman's Docker-compatible socket. Not a single job file had to be touched. All five hosts switched in one day.
3. Nomad to Quadlet, one service at a time. Traefik first, as blueprint, then service by service until the last one (GitLab, of course).
4. Replace what the stack did on the side: scrape targets without Consul, certificates without Vault, a dashboard instead of the Nomad UI, image cleanup instead of Nomad's garbage collection.
5. Switch off the stack. The Ansible roles for Nomad, Consul and Vault were not deleted; each got a switch, and "off" removes everything the role once installed.

Every change ran on a throwaway test VM first, always in check mode before the real run, and a second run had to show changed=0.

One Service, Before and After

To get an idea of what changed for a single service, here is my small fuel price exporter. Before, the Nomad job as Ansible rendered it for its last deployment under Nomad (the {{ ... }} parts were resolved by Nomad at runtime, from Vault and Consul); after, the Quadlet unit as Ansible renders it on the host today:

Before: Nomad jobAfter: Quadlet unit

job "station-prices" {
  region = "home"
  datacenters = ["underverse"]
  type = "service"
  priority = 25
  update {
    max_parallel = 1
    health_check = "checks"
    min_healthy_time = "45s"
    healthy_deadline = "300s"
    auto_revert = true
  }
  group "stations" {
    constraint {
      attribute = "${node.class}"
      value = "external"
    }
    count = 1
    disconnect {
      lost_after = "12h"
      reconcile = "keep_original"
    }
    restart {
      attempts = 3
      interval = "10m"
      delay = "60s"
      mode = "fail"
    }
    network {
      port "http" {
        to = 8080
      }
    }
    task "station-prices" {
      consul {}
      vault {
        env = false
        disable_file = true
      }
      template {
        data = "{{ with secret \"underverse/data/station-prices\" }}{{ .Data.data.tankerkoenig_api_key }}{{ end }}"
        destination = "${NOMAD_SECRETS_DIR}/tankerkoenig_api_key.env"
        env = true
      }
      template {
        data = "{{ key \"configs/station-prices/station_prices.conf\" }}"
        destination = "local/station_prices.conf"
      }
      driver = "docker"
      config {
        image = "registry.underverse.net/underverse/container-projects/station-prices:1.7.2-1"
        hostname = "station-prices"
        command = "/usr/bin/station_prices.py"
        args = [
          "-d",
          "-c",
          "/local/station_prices.conf",
        ]
        ports = [
          "http"
        ]
        labels {
          cpu_shares = "100"
          container_hostname = "station-prices"
        }
      }
      resources {
        cpu = 100 # MHz
        memory = 64 # MB
      }
      service {
        name = "station-prices"
        tags = [
          "${node.class}",
          "monitoring=enabled"
        ]
        port = "http"
        check {
          name = "station-prices"
          type = "http"
          method = "HEAD"
          path = "/"
          interval = "20s"
          timeout = "5s"
          check_restart {
            limit = 3
            grace = "10s"
          }
        }
      }
    }
  }
}

# Ansible managed -- Quelle: station-prices/roles/station_prices/templates/station_prices.container.j2
[Unit]
Description=Tankerkoenig fuel price exporter (station-prices)
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=station-prices
Image=registry.underverse.net/underverse/container-projects/station-prices:1.7.3-1
HostName=station-prices
Pull=never
RunInit=true
PublishPort=10.153.200.5:8080:8080
Volume=/etc/opt/underverse/services/station-prices/station_prices.conf:/etc/station_prices/station_prices.conf:ro
Exec=/usr/local/bin/station_prices.py -d -c /etc/station_prices/station_prices.conf
EnvironmentFile=/etc/opt/underverse/services/station-prices/station_prices.env
Memory=64M
PodmanArgs=--memory-swap=64M
Label=container_hostname=station-prices
Label=cpu_shares=250
HealthCmd=curl -fsS -m 5 -o /dev/null -I http://127.0.0.1:8080/ || exit 1
HealthInterval=20s
HealthTimeout=5s
HealthStartPeriod=180s
HealthRetries=3
HealthOnFailure=restart

[Service]
Restart=always
CPUWeight=250
OOMScoreAdjust=250
MemoryMax=64M

[Install]
WantedBy=multi-user.target

The unit is not shorter by accident: everything that Nomad, Consul and Vault did in between is either gone or happens before the unit is written. The template behind it is mostly a handful of macro calls from the collection.

Pitfalls

Oh, there were a few. Here are my favorites, mostly the silent ones.

Two Runtimes Don't Share a Host

Docker sets the iptables FORWARD policy to DROP, Podman's netavark writes its rules to nftables. Under nftables both are evaluated, and Docker's policy kills the bridge traffic of Podman. Result: Podman containers had no network at all.
Lesson: remove Docker and install Podman in the same maintenance window, not in two.

Docker Survives Its Own Uninstall

The docker0 bridge, Docker's netfilter tables and its data directories (several GB) were still there after removing the package. Only a reboot or a rebuild shows this kind of thing.

Restarting the Podman Socket Breaks Every Container Using It

A restart of podman.socket creates a new socket file. Every container which had mounted the socket before (Traefik, the GitLab runner) still points to the old inode and gets Connection refused, although the file is there and writable.
Since then: never restart the socket from a handler.

Podman Has Fewer Capabilities Than Docker

Same image, other runtime, and suddenly Telegraf's ping input complains about CAP_NET_RAW. Podman's default set lacks NET_RAW, MKNOD and AUDIT_WRITE.

Unit Names Live in the Package Namespace

A Quadlet unit traefik.service silently shadows a packaged traefik.service, the generator wins. So every container unit got the suffix -cntr.
Nice, but a certificate hook in another repo still restarted traefik.service - which no longer existed, and || true swallowed the error. Nobody would have noticed until the certificate expired. Found it by accident four days later, while working on something else.

Quadlet Reads Values Like systemd

Label= values are split at spaces, ' is a quote, and a label containing \. is dropped completely - only a line in the generator log tells you. A Traefik rule with a regex like HostRegexp(...\.net$) has both...so every label now goes through a small macro, and every route is checked in the Traefik API after a deployment.

The Journal Made the Logs Nine Times Bigger

With Podman, every container logs into the journal and from there into VictoriaLogs. Great...until I saw 6.2 million lines per day instead of 660,000.
Two sources: podman.service logged every call to its own API socket, and Traefik logged every upload of a log line - which produced a new log line to upload. One --log-level=warn and one Traefik label later, I was back to the level before the migration.
Today it's about 2 million again, but for a good reason: since GitLab moved, its own logs land in VictoriaLogs as well (1.3 million lines per day). Without GitLab, it's still the old level - and the chatter of Consul and the Vault unseal helper (170,000 lines per day) is gone for good.

And honestly, even if the volume is higher: having the logs of all containers in one place is worth it. In Nomad, I had to click into every single allocation to read its logs, and every now and then (for whatever reason) they simply didn't show up at all. Now it's one query in VictoriaLogs, across all hosts and services.

A Firewall Rule That Was Too Friendly

My new nftables template had a rule "allow published container ports". It did exactly that - on every interface, including the public one. It showed up while moving MariaDB and Nextcloud, and the access log showed that someone else had found it before me. Fixed, and since then every change of what is reachable from outside gets an explicit negative test from outside.

The Obvious Image Cleanup Is a Time Bomb

podman image prune thinks an image is unused if no container references it. A stopped Quadlet unit has no container. With Pull=never (which protects me from the registry chicken-and-egg problem) the unit would never start again after the next reboot. The keep list now comes from the placement repo, not from the runtime.

Zombies

CrowdSec restarted every ~11 hours. The reason: the health check used busybox wget for HTTPS, which starts an ssl_client child; CrowdSec as PID 1 never reaped it. 2048 zombies later, the PID limit was reached and the health check could no longer fork. Under Nomad, the check ran from outside via Consul, so this was new. RunInit=true and done.

Effort and Duration

date	milestone
08.09.2026	the question: "Can I deploy Traefik completely with Ansible?"
09.09.2026	the collection replaces 26 copies of the role code
14.09.2026	all five hosts switched from Docker to Podman (still under Nomad)
16.09.2026	the first host is free of Nomad
23.09.2026	the big services (VictoriaMetrics, Grafana, MariaDB, Nextcloud) moved
26.09.2026	GitLab, the last service, moved
28.09.2026	Nomad has no running job anymore
01.10.2026	Nomad, Consul and Vault are switched off on all hosts
11.10.2026	the first certificate renewal through the new ACME chain
13.10.2026	project closed

The number of containers under Nomad and under systemd tells the same story. The plateau from 16.09. to 22.09. was the week where Traefik and the first services ran under Quadlet while I sorted out the pitfalls above, and then it went fast:

That's a bit more than three weeks until the stack was off, and five weeks including the cleanup afterwards.
In numbers: more than 700 commits in about 30 repos, 27 releases of the collection, and a German archive of measurements and decisions with more than 12,000 lines. That last one sounds crazy, but whenever something broke again, there was a place to look up what we already knew.

"We", because I did this together with Claude Code. It read the code, measured on the hosts, wrote the plan and kept the archive. All decisions, commits, builds and production rollouts stayed with me, and a good part of the pitfalls above were found because I asked one more question when something looked too easy.
I've recently started using LLMs in various contexts, especially @work. And I have to admit, even though I didn't expect this and, of course, a lot of oversight is still needed (I would currently always verify the results manually and not just blindly rely on "it'll be fine"), implementing such projects would take much longer, involve a lot more trial and error, and likely result in significantly more downtime.
So yes, it definitely makes sense to use these tools, as long as you keep track of things and occasionally point out that there's surely an even "simpler" way to do it.

Final State

what	after
runtime	Podman, one Quadlet unit per container, 30 units on five hosts
orchestration	none - systemd starts, restarts and orders the units
placement	one YAML file per unit, checked before every deployment
deployment	./run.py deploy from each project repo, Ansible with a shared collection
secrets	ansible-vault in the project repos
certificates	ACME job on a GitLab shell runner, distribution in the same pipeline
service discovery	Traefik reads the container labels, scrape targets come from the placement
health	every unit has a check of the service itself, HealthOnFailure=restart
registry / GitLab out	units start from the local image store, tested with both blocked
firewall	nftables from one template on all hosts, nothing open that it doesn't list

The replacement for the Nomad UI is a Grafana dashboard built from Telegraf metrics and the journal: which unit runs where, its health, restarts, CPU, RAM against its limit and IO - plus, per host, how much RAM is still free for further limits.

On the host itself, it's just systemd:

systemctl list-units --no-pager '*-cntr.service'
  UNIT                            LOAD   ACTIVE SUB     DESCRIPTION
  alertmanager-cntr.service       loaded active running Prometheus Alertmanager (alertmanager)
  archpkg-cntr.service            loaded active running Apache serving the underverse arch package repository (archpkg)
  archpkg-deploy-cntr.service     loaded active running sshd accepting package releases for the arch repository (archpkg-deploy)
  bind9-cntr.service              loaded active running BIND secondary nameserver (bind9)
  bind9-monitoring-cntr.service   loaded active running Telegraf (bind9-monitoring)
  central-monitoring-cntr.service loaded active running Telegraf (central-monitoring)
  gitlab-runner-cntr.service      loaded active running GitLab Runner (gitlab-runner)
  grafana-cntr.service            loaded active running Grafana (grafana)
  station-prices-cntr.service     loaded active running Tankerkoenig fuel price exporter (station-prices)
  traefik-cntr.service            loaded active running Traefik ingress (traefik)
  victoriametrics-cntr.service    loaded active running VictoriaMetrics time series database (victoriametrics)
  vlagent-external-cntr.service   loaded active running vlagent accepting journald for the external region (vlagent-external)
  vmagent-external-cntr.service   loaded active running vmagent scraping one node class (vmagent-external)
  vmalert-cntr.service            loaded active running vmalert evaluating the alerting rules (vmalert)

14 loaded units listed. Pass --all to see loaded but inactive units, too.

The monitoring went from 50 to 37 scrape targets - the 13 missing ones were Nomad, Consul and Vault themselves, switched off on 01.10.:

Three VMs fewer to update, no unseal after a reboot, no consul-template and a deployment that I can read from top to bottom.

Conclusion

Would I recommend the Hashistack for a home lab? For learning: absolutely, I don't regret a single year with it. For running a handful of pinned containers: no. systemd can do most of what I used Nomad for, and Quadlet makes it nice to write.
What I didn't expect was how much the exit taught me about my own setup. A good number of the problems above were not caused by the migration, they were just hidden before - and some of them for years.