My Server Setup, 2026 Edition

Back in 2023 I wrote a five-part series describing the infrastructure that hosts my websites and projects. Almost none of it is true any more. The workloads are mostly the same; nearly everything underneath them has been replaced. This is the update.

Then and now

The scheduler survived: it’s still Consul and Nomad, and I still think that’s the right amount of orchestration for this scale. Everything else went.

From one cluster to two

This is the biggest architectural change, and it didn’t happen in one move.

The first evolution was actually in the opposite direction: I brought the whole cluster in-home to save on hosting costs. Scaleway went away, the workloads moved onto hardware I already owned, and the publicly accessible parts were served through a Cloudflare tunnel: cloudflared runs inside the cluster, dials out to Cloudflare, and nothing needs an inbound port or a public IP. That change also quietly killed the old certificate story: with Cloudflare terminating TLS at the edge, Aleff and its Let’s Encrypt automation had nothing left to do.

Most of the static sites, this blog included, left the cluster entirely and now live on Cloudflare Pages, built straight from their repos. They were a whole section of the 2023 series; now they’re mostly not my problem at all.

Running everything at home was cheap, and it worked, right up until the outages started. Internet connection issues, a NAS failure, and building work that kept interrupting the electricity supply made the point repeatedly: home is fine for things that only matter at home, and the wrong place for anything the outside world should be able to reach. That forced a rethink for the publicly accessible stuff, which led to the next two steps.

First, the control plane moved off-home. The original cluster’s brain lived on the least reliable site I own: residential power, residential internet, hardware within reach of children. The fix was to invert the topology: put the Consul/Nomad servers on reliable cloud VPSes and let home become disposable.

Then that design turned out to be wrong too. The assumption behind it was “a house that’s offline has little to serve,” and that stopped being true: home now runs things that are deliberately local (HomeAssistant, MQTT, electricity-meter monitoring) that must keep scheduling, healing, and finding each other with the internet down. A home cluster whose control plane is on the far side of a WAN link can’t promise that.

So the current shape is two fully independent clusters. Each has its own Consul + Nomad quorum, its own deployer, its own ingress. Nothing crosses between the clusters; other than the deployment configurations, they’re not aware of each other. Both reconcile from the same monorepo, and each service’s deployment configuration says which clusters it lands on. The default is home, so nothing ever reaches the cloud cluster without an explicit opt-in. Forget to opt in and the failure mode is “it stayed where it was,” not “it vanished.”

The cloud cluster

Four Hetzner Cloud VPSes in Falkenstein/Nuremberg:

• prod-01..03 are hybrids: each runs a Consul/Nomad server (three-way raft, tolerates one node down) and schedules workloads.
• prod-db-01 is a client only. It stays out of the quorum, carries a cloud-db node class, and only jobs that ask for that class land on it. It runs Postgres and NATS on a dedicated /data partition; everything else keeps its state in a CSI volume or treats the disk as disposable.

Postgres and NATS were chosen partly because hosted options exist: if either outgrows self-management, there’s a managed service to migrate to without changing the applications.

The cluster also runs the Hetzner CSI driver, so jobs that need persistent storage get dynamically provisioned Hetzner volumes rather than being pinned to a node’s disk.

Cluster traffic (gossip, RPC) rides Hetzner’s private network, never a public IP. The firewall is default-deny, and public traffic arrives the same way it does at home: through cloudflared, dialling out. No inbound ports.

The home cluster

The home lab is four mini PCs of varying specs (three collected over the years, one bought specifically for the cluster) running as one server and three clients from its own NixOS repo. Hostnames are handed out by the router, so nodes can join and leave without config changes; Consul leaves the cluster and Nomad drains the node on graceful shutdown. The two repos are deliberately separate so that a push to one can never touch the other, but they’re very similar, so changes can be tested in the home cluster before being rolled out to the cloud.

From Ansible to NixOS

The 2023 setup managed Ubuntu boxes with Ansible and Nomad jobs with Terraform. It worked, but the servers were pets: state accumulated, and the answer to “what is running on this box?” was “whatever Ansible has done to it over the years.”

Now every node, on both clusters, is NixOS, and every node is cattle. A cloud machine is installed with nixos-anywhere and disko: point it at a fresh Hetzner Debian box and it wipes the disk and installs the exact configuration in the repo. The only per-node state worth keeping is the SSH host key, which is pre-generated, kept outside the repo, injected at install time, and doubles as the node’s sops decryption key. Lose the box, reinstall, done. Durable state lives in S3, and the Consul/Nomad raft directories rebuild from quorum.

There’s no per-host configuration drift because there’s no per-host configuration: one flake output per node, identical bar an ordinal that picks the hostname, private IP, and a slot in the auto-upgrade schedule. Config management is now git push.

Nightly auto-updates

Every node updates itself overnight, unattended. Each one pulls the latest configuration from its repo, rebuilds, and switches to the new system; if the kernel or anything else reboot-worthy changed, it reboots too. The whole thing reports to Slack, including a “host has returned” message after an upgrade-driven reboot, so a node that doesn’t come back is noticed.

The schedule is staggered: each node gets its own hour, offset by its ordinal, with a random delay on top, so no two nodes are ever upgrading or rebooting in the same window. On the cloud cluster that’s what keeps the Raft quorum intact: with three servers taking turns, two are always up while the third restarts. Updates roll through the cluster every night and nothing goes down.

Tailscale replaced the bastion

The 2023 setup had a hardened bastion host for SSH. Now admin access is Tailscale: SSH binds to the tailnet, root login is off, and the bastion doesn’t exist. It deletes a whole category of maintenance (no bastion to patch, no single point of access failure) in exchange for trusting a mesh I was already running.

Tailscale is deliberately only the admin fabric. Cluster traffic stays on each site’s private network; the tailnet is for humans.

sops replaced Vault

Vault was the most operationally expensive component of the old cluster relative to what it did for me, which was mostly “store a handful of secrets.” Secrets are now sops-encrypted files in the repo, decrypted on the node at activation time using the node’s SSH host key. The secrets travel with the config, the git history is the audit log, and there’s no unseal ceremony after a power cut.

cd replaced Noman, DockerHub, and the webhook chain

The 2023 deploy pipeline was a Rube Goldberg machine I was quite proud of: GitHub Action tags the repo, DockerHub sees the tag and builds, a webhook tells a custom component (Noman) about the new image, Noman updates a key in Consul, and Nomad notices. It worked, and every hop was a place it could silently stop working.

The replacement is a single home-grown reconciler called cd. Everything lives in one monorepo containing multiple projects, both platform infrastructure and production workloads, and every service carries a deployment configuration declaring what it is and which clusters it belongs to. Each cluster’s cd instance receives webhooks from GitHub, each of which triggers a reconcile that converges its subset: builds, submits, and deletes, the part the old system never had. Noman could only ever push new images; it couldn’t remove a decommissioned job or notice drift. cd stamps every job it manages and may only deregister jobs carrying its stamp, so its destructive path is fenced by ownership rather than naming conventions. Deploys and failures report to Slack.

Infra-level jobs (the proxy, MQTT, the registry, and friends) used to have their own blind resubmit-on-a-cron loop; they’re now just another project kind under cd, with the same lifecycle semantics as everything else.

What I’d tell 2023 me

• The pets-to-cattle switch was worth more than any single tool choice. Being able to reinstall a node from scratch without dreading it changes how you treat everything above it. And it’s not just recovery: scaling the number of nodes up or down is now trivial too.
• Every hop in a deploy pipeline is a failure mode. The DockerHub webhook chain didn’t fail loudly; it failed by nothing happening. A reconciler that converges on desired state can tell you it’s stuck. A chain of triggers can only go quiet.

The 2023 series ends with a section called “Eliminating the Downtime.” The 2026 version of that goal is less about downtime and more about recovery being boring: any node, on either cluster, can be rebuilt from the repo in minutes. That, it turns out, was the property I actually wanted all along.

What’s next?

• Scaling down the home cluster and repurposing the now-excessive hardware.
• Building actual products for the platform to host, rather than continuing to build out the platform itself.
• Continuing to monitor Hetzner for cheap VPS and dedicated nodes to add as needed.