narc: GitHub Action Runners on Nomad

I ran out of GitHub Actions minutes. Public repositories get hosted runners for free, but private ones draw from a monthly allowance, and between the platform and a handful of projects I was burning through it well before the month was out. The obvious fix was to pay for more. The less obvious fix was sitting right there: a Nomad cluster that spends most of its time with CPU and memory to spare.

Minutes on self-hosted runners don’t come out of the allowance, so the question became how to turn that spare capacity into runners without making a mess. The answer is narc, the Nomad Action Runners Coordinator.

Why not just run a runner?

You can register a self-hosted runner in about five minutes. Put it in a container, point it at your org, done. I didn’t want that, for two reasons.

First, a long-lived runner is a pet. Every job leaves something behind: Docker images, caches, half-deleted build directories, a tool someone installed with sudo. Eventually a build passes or fails because of what the previous build did, and you spend an evening working out which.

Second, it’s a security problem. A runner executes whatever the workflow says. If that runner lives on a cluster node, or in a container on the cluster network, every workflow has a shell somewhere it can see Nomad, Consul and everything else I run.

What I actually wanted was what GitHub-hosted runners give you: a fresh machine for every job, with root and a real Docker daemon, thrown away afterwards. On Kubernetes that’s actions-runner-controller. On Nomad, the options I found were either single-host or Docker-centric, and I wanted a VM per job.

Less work than it sounds

“Port actions-runner-controller to Nomad” sounds like a big job, but most of ARC is Kubernetes operator plumbing: custom resources, reconcilers, Helm charts. The part that talks to GitHub is available on its own as actions/scaleset, an MIT-licensed Go client for the runner scale set API, published specifically so people can build autoscalers for platforms other than Kubernetes.

That makes narc a small Go service: scale set listener in, Nomad dispatch out. It’s about 1,200 lines of Go, plus a reference runner job, ACL policies, a CNI network config, firewall rules, and a NixOS module that sets up the nodes.

How it works

narc registers a scale set with GitHub for each runner shape you configure. A workflow picks one by name:

jobs:
  build:
    runs-on: nomad-linux

narc long-polls GitHub for demand. When a job is waiting, it asks GitHub for a single-use just-in-time (JIT) runner config and dispatches a Nomad parameterized job with it. The reference job boots a stock Ubuntu cloud image under QEMU/KVM, cloud-init installs Docker and the latest runner, the runner takes exactly one job, and the VM powers off. The allocation ends and the disk goes with it.

Every runner is a dispatched child of the runner job, so the job’s page in the Nomad UI doubles as a record of every CI job narc has run:

A few properties fall out of that design that I’m particularly happy with:

• No public ingress. The scale set API is a long poll from narc to GitHub, so there are no webhooks to receive. Like everything else on the platform, narc dials out.
• Nomad does the scheduling. Each scale set maps to an ordinary Nomad job, so placement, constraints and resource limits are just jobspec. narc doesn’t need its own schema for any of it.
• Stateless. On startup narc adopts any runners that are still alive, cleans up anything orphaned, and reconciles against GitHub. While it’s down, jobs queue at GitHub and running jobs carry on regardless.
• Warm pools. Each scale set can keep a number of runners booted and idle, so a job doesn’t have to wait for a VM to start. narc keeps min(max, assigned jobs + warm) runners alive and never scales down: runners exit when their job is done.

Keeping CI jobs away from the cluster

Running arbitrary workflow code on the same machines as production needs more care than “it’s in a VM”. The VM boundary handles the node itself: the workload gets root inside the guest and nothing on the host, and its copy-on-write disk is deleted with the allocation, so nothing persists between jobs.

The network needs its own treatment. Runner VMs sit on a dedicated CNI bridge, and nftables rules on each node drop anything from that bridge to private ranges, the tailnet, link-local and metadata addresses, the node itself, and every node’s public addresses. That last one matters: a service bound to 0.0.0.0 is reachable on the node’s public IP, even if it was only ever meant for the private network. CI jobs get the internet and nothing else.

The JIT config deserves care too, because whoever reads it first gets the job, secrets included. It never appears in dispatch metadata (anyone who can read jobs can see that), only in a Nomad Variable that the runner job’s template can read and narc itself can’t. And narc’s own Nomad policy deliberately leaves out submit-job: it can dispatch the runner jobs you’ve registered, but a compromised narc can’t register new ones.

The price of a fresh VM

Isolation isn’t free. The main downside of a fresh VM per job is startup time. Every runner boots a stock Ubuntu cloud image, and then cloud-init does all the setup, every time: it updates the package lists, installs Docker and a build toolchain, downloads the latest runner release and installs its dependencies. Only then does the runner register and take the job. And it all happens again for the next job, because the whole point is that nothing survives from the last one.

Warm pools hide most of that for a steady trickle of jobs, but not for a burst. When a push kicks off several workflows at once, the warm runners are taken immediately and every other job waits for a VM to boot from scratch. While that’s happening, GitHub’s UI can make it look as though there are no runners available at all: they exist, they’re just still installing packages.

The fix is a custom image with everything except the JIT config already baked in, so a VM only has to boot, write its config and start the runner. narc already supports this: the runner job takes any image URL and SHA-256, and only the first job on each node pays to download it. Building that image is next on my list. The catch is the runner version: GitHub refuses runners that fall too far behind its latest release, so baking the runner in means rebuilding the image regularly instead of picking up the latest release on every boot.

Keeping an eye on it

narc exposes Prometheus metrics on /metrics:

• live runners, split into starting, idle and busy
• GitHub’s own counts of assigned and running jobs
• dispatch failures
• runners whose VM died before finishing its job, which narc removes from GitHub
• time from dispatch to job start: the boot cost described above

There’s also a /healthz endpoint that only reports healthy once every scale set has an active session with GitHub.

The repo includes an example Grafana dashboard built on those metrics, filterable by target and scale set. Plotting GitHub’s assigned jobs against live runners is the quickest way to see when demand has outrun a scale set’s max.

What production taught me in an afternoon

The first version landed in the early hours. By teatime it had picked up a dozen fixes, most of them for things I’d only ever have found by running real CI on real nodes. The commit log is honest about it, and a few are worth repeating.

Disks fill up fast. A dead allocation’s directory stays on disk until the Nomad client garbage-collects it, and each runner’s overlay holds everything its job wrote. About 30 finished runners at roughly 6.5 GiB each took 160 GB of a node’s disk in one afternoon. The runner job now has a poststop task that deletes the overlay as soon as the VM exits.

QEMU needs memory too. Nomad’s QEMU driver gives the guest the task’s full memory allocation, which is also the cgroup limit. Add QEMU’s own overhead and the process goes over, so the kernel OOM-killed VMs under compile load, and GitHub reported that the runner had “lost communication”. The guest now gets 1 GB less than the task.

Nomad agents re-render templates on restart. narc originally deleted the JIT config variable as soon as the runner started, which felt tidy. But when a Nomad agent restarts it restores running tasks and renders their templates again, and the template then blocked forever waiting for a variable that no longer existed. Nomad lost track of the VM, and narc counted the runner as alive until it hit the six-hour limit. The obvious fix, only rendering the variable if it exists, was worse: the template re-rendered the file without the config, and because QEMU’s FAT seed disk reads files lazily, fresh VMs could see the emptied file and boot without a usable config. The variable now lives until the allocation ends.

IDs are only unique where they say they are. I run two targets, an org and a personal repo, sharing one runner job. GitHub’s scale set IDs are only unique per target, so after a restart both scalers adopted every live runner. One of them counted the other’s busy runners as its own idle ones, decided it had enough capacity, and let jobs queue. Runners now carry their target and scale set in their dispatch metadata.

GitHub doesn’t tell you which runners are online. A VM whose runner never registers (bad cloud-init, a failed download) looks exactly like an idle runner waiting for work, so narc happily kept it until the six-hour limit while jobs queued behind it. narc now infers it: if GitHub has had more jobs assigned than there are busy runners for fifteen minutes, idle runners older than that are stopped and replaced.

None of these were exotic. They’re all the kind of thing that only shows up under real load, on a real cluster, with real agent restarts, which is a good argument for getting something minimal into production early and letting it tell you what’s wrong with it.

What’s next

First up is the custom image. The other big gap is garbage collection for the per-node image cache: when the base image changes, the old one is never removed, so node disks slowly fill. An image that gets rebuilt regularly will make that worse, so the two go together. Each image’s timestamp is touched on use, so the fix is a sweep that deletes anything not used for a while. After that, maybe a Podman runner job for trusted workloads that don’t need Docker.

Meanwhile my CI runs on hardware I already pay for, the monthly allowance has stopped being something I think about, and the cluster’s spare capacity finally has a job. narc is MIT-licensed and on GitHub; if you run Nomad on machines with KVM and pay for CI minutes, give it a try. Issues and pull requests are welcome.