How we run GPU infrastructure with agents, from meeting to merge

On the GPU and AI infrastructure we run, a decision starts as a recorded meeting and reaches production as a merged pull request. That holds for the platforms we operate for customers and for our own GPU servers. In between, LLMs and agents do most of the writing: meeting summaries, deployment intents, Kubernetes manifests, scripts and patch plans. People make the decisions and approve every change.

What makes that safe is a trail of written records that people and agents both read. On a GPU cluster, a driver pin or a kernel argument usually exists because something broke once, and our lessons file keeps the reason beside it. If that reason lives only in someone's head, the next engineer or the next agent will undo it. So every step from meeting to merge leaves a file in git.

Every conversation becomes a record

Every meeting is recorded and transcribed by Granola, which also writes an AI summary. When a meeting decides or defers something, that summary becomes a session record committed to the design repo. The record lists what was discussed, what was decided and what was explicitly deferred, and it links back to the source transcript. Once committed, nobody edits it. Every decision gets a record within one working day, and without a record there is no ADR (Architecture Decision Record).

Plenty of decisions land in a Teams thread, a Linear issue or a document instead. That is what AKMS, the knowledge system we built to index work where it already lives, is for. People and agents query it through MCP (Model Context Protocol) for the context behind a piece of work and the tasks attached to it. An agent reads the same history a person would, with a link to every source.

Decisions harden before anything is built

A session record is the narrative. The ADR is the decision. We write ADRs in the Nygard form: context, decision, alternatives considered, consequences. Each one opens as Proposed and points back at its session record. Drafting is AI-assisted. Only the architecture owner can move an ADR to Accepted.

Accepted ADRs never change. When we changed AI gateways, a new ADR superseded the old one, and the old one got an updated status line with its body left alone. A decision log you can rewrite tells the next reviewer nothing. Security review also starts at the first conversation, not at the end.

An accepted ADR then flows into the living design spec. No spec may contradict an accepted ADR, and superseded wording is deleted rather than annotated. That rule matters more with agents in the loop. An agent reading the spec later cannot tell which paragraph is stale, so it will carry whatever it finds downstream. If an answer cannot be derived from the design, the design is incomplete, and we fix it there instead of in a manifest.

The intent file carries the design into the deploy repo

The deploy repo has one folder per service, and every folder has an intent.md. It says why the service exists and what properties it must have. It never contains a command. It has four required sections:

• Platform requirement: the problem the service solves, with links to the ADRs.
• Design goals: the properties the solution must have.
• Non-goals: what is explicitly out of scope.
• Open questions: decisions still to make, closed as they are settled.

Its history stays readable without a diff. A revised goal gets a short inline note with the date and the reason. A settled question is struck through with the decision beside it. This is one from the intent of our certificate service:

- ~~Wildcard vs per-name SAN growth.~~ DECIDED 2026-08-03: one
  wildcard certificate for *.example.org. Single label only,
  no apex, no per-name certs. DNS-01 supports per-name issuance
  if an auditor ever demands it; that would be a new decision.

Claude Code writes the intent. Today we prompt it by hand, and we are moving that prompt into a skill so every run starts from the same instructions. Its inputs are the accepted ADRs, the relevant session records, the design repo's git history, an existing intent as a format reference and our lessons file. It must not contradict an accepted ADR without raising the conflict in the pull request. Values nobody has decided yet stay as named environment variables, so the checks that depend on them report SKIP instead of passing on a guess.

A person reviews every intent, and the reviewer has to be someone who could have written it. Generation is what makes this fast. Review is what makes it safe.

The agent writes the package, and test earns production

From the intent, Claude Code writes the rest of the service package: Kubernetes manifests and Helm values, one Argo CD Application and four scripts. There is no generator program. No template engine or CI job turns intent into YAML.

The scripts follow one contract. preflight.sh is read-only and checks the prerequisites. deploy.sh is idempotent and always ends by calling verify.sh. teardown.sh removes everything the deploy created. An operator can run any of them without an agent. Every check in spec/checks.md traces to a goal in the intent or an entry in the lessons file. A check with nothing behind it is noise, and noise teaches people to ignore a red result.

A service is done only when teardown.sh plus deploy.sh rebuilds it from the repo alone. In test, that single rule exposed every piece of unmanaged state before anything depended on it, from models registered only through an admin API to hand-applied NodePort patches.

Every change runs in the test environment first. What we learn there goes into LESSONS-LEARNED.md in four buckets: binding decisions, one-way doors, recurring gotchas and things never to copy from test. Production is then re-authored from the test intent plus those lessons. It is never copied, and every difference between the two has an owner who chose it.

The one-way doors bucket exists because of pain. In test, the identity server picked its own UID range. A botched attempt to fix that broke its certificate authority and forced a full reinstall, with every node re-enrolled. In production the range is set in the intent before the first user exists, because the provisioner, node logins, storage and file ownership all hard-code it.

Merge is the deploy, until you reach the operating system

Every change is a pull request. gitleaks has run in CI on every push and pull request since the first commit. Secrets come from a cloud parameter store at deploy time and never enter git. A second engineer who did not write the change approves it. Before a significant cluster change lands, we snapshot etcd and its encryption key, per-node host config, a full database dump and the identity realm export.

For manifests and Helm values, the merge is the deploy. A root Argo CD Application watches one folder in the repo, and merging an Application file into it is the only way a service goes live. Sync is automated with selfHeal: true and prune: false, and rollback is git revert.

selfHeal is on because of what we saw in test. The largest recurring failure there was live state drifting away from git, such as a kubectl patch silently reverted by the next helm upgrade. With selfHeal on, a hand edit simply does not stick. Prune is off for the opposite reason. Deleting a file must never delete a live service, so production removals stay staged and human: scale to zero, soak for at least 48 hours, then delete.

Below GitOps the rules change. The OS baseline, kernel, GPU drivers and cluster bootstrap never get an Argo CD Application. They are scripted and started by a person, because a merge must never quietly reconfigure a node. Rolling back there means restoring a snapshot.

A task is not finished until the as-deployed inventory and changelog are updated. Then the test environment is redeployed from the production baseline, so the next change starts from what production actually runs.

Agents run the routine, people press go

Routine operations run through skills checked into the deploy repo. The patch-round skill takes OS, kernel and security patching across the fleet. It first re-checks the vulnerability scan against the live hosts, because a scan is often old. It writes the plan as a pull request with status Proposed. Then it runs one window at a time, with a snapshot, a drain and a verify per window, and waits for a person's explicit go before each one.

Some gates only a person can meet, such as out-of-band access to a GPU server. The skill asks for them and never assumes them, and an approval from one round does not carry into the next. GPU nodes get a trial boot: the old kernel stays the GRUB default and the new one boots once, so a panic lands back on the known-good kernel. Old kernels stay installed for at least 48 hours after the round closes, because the old kernel is the rollback.

The skill was distilled from the record of a real patching round, lessons included. Restarting an API server re-fetches the OIDC (OpenID Connect) issuer, and our issuer's name resolved to a public edge the nodes cannot reach. OIDC logins failed. It went unnoticed for two reboots, because every check went through the cluster's virtual IP, which stayed on the one API server that had not restarted yet. The skill now proves OIDC against each API server directly after every reboot.

The skill ships with evals, and one assertion reads plainly: drain the node before running the package install. The second skill is a read-only health check that runs from the nodes up to every live inference engine in about a minute. Its exit code is the number of failures. It never restarts a pod or syncs an app, so fixes go back through the repo like any other change.

Lessons go back to the start

Anything surprising in test or production owes something back to the design layer: a new session record, and a new ADR if a decision moved. When verify finds drift, the fix and the lessons entry ship together. Otherwise the same drift turns up in the next environment. That is how the OIDC lesson ended up in a skill instead of in one engineer's memory.

This is why I am comfortable letting the agent move fast. Any mistake it makes above GitOps is one revert away, and anything below GitOps waits for a person.

Next, the intent prompt becomes a skill, so the first draft of every service starts from the same written instructions our patch rounds already do.