MLOps below the model: AI agents build a tested replica of the AI factory from git

MLOps usually covers models: tracking experiments, versioning them and testing each release. The models run on an AI factory platform of GPU drivers, schedulers, inference engines, gateways, identity and storage. Each part is updated on its own schedule, and a bad update to any of them can stop a model from serving.

I devised and created a method to test that platform the way software teams test code, and it is the method I use to build and test production replicas. The method has three rules:

Production's git repository is the single source of truth.
A replica of production is built from that repository alone, in a separate test environment.
Only fixes proven in the replica go back to production, through a reviewed pull request.

The whole process is driven by an AI agent. Claude Code is the one I use, but the method doesn't depend on a particular agent, because the instructions, scripts and checks all live in the repository. The agent writes and fixes the scripts, builds the replica from empty VMs one phase at a time, and prepares the fixes that go back to production. Today a person approves every phase and every change that reaches production. The next step is to remove that gate and let the agent run the whole process on its own.

The platform in this example provides GPU inference, an AI gateway, notebook workspaces, single sign-on and shared storage, about 30 services in all. Network addresses, VMs and storage come from other teams. Everything else comes from the repository.

Diagram titled The platform, layer by layer: about 30 service packages, each with an intent file and deploy, verify and teardown scripts. Layers from top to bottom. User tools: AI chat with agents, Coder for notebook and IDE workspaces with an optional GPU, and MLflow for experiment tracking and a model registry and Kubeflow for ML pipelines and training jobs, both marked as in use but not yet in the repo. AI serving: the AI inference gateway, with API keys, rate limits and one OpenAI-compatible endpoint, and vLLM, the inference engine on the GPU servers. AI chat, Coder, the AI inference gateway and vLLM are highlighted as the path of a model request. GPU: the GPU Operator for the driver, toolkit, device plugin and DCGM; KAI Scheduler for GPU queues and quotas per project; GPU fractions so several workloads share one GPU; and GPUDirect Storage. Identity, as a chain: an external identity provider, then Keycloak for single sign-on, a provisioner that creates the user on first login, FreeIPA for UIDs and GIDs, and SSSD for names on every node. Platform services: Argo CD, ingress with MetalLB and NGINX Gateway Fabric, cert-manager for public TLS certificates, DNS naming, Harbor container registry, one HA PostgreSQL cluster, Kyverno admission policies and an admin portal. Observability and security: Prometheus with alerts and a GPU exporter, Grafana, metrics-server, Headlamp, a platform health sweep and vulnerability scan agents on every host. Kubernetes and storage: a three-node HA control plane built with kubeadm, kube-vip for the API address, Calico for the pod network, and shared storage for home directories, model weights and metrics. Operating system: an Ubuntu baseline with patches, kernel settings and RDMA and GPU host tuning. At the bottom, in an amber dashed box, the hardware built by other teams and not in the repo: control-plane VMs, CPU worker VMs, bare-metal GPU servers with RDMA networking, a FreeIPA VM and the storage cluster. — Everything above the hardware is built from the repo. MLflow and Kubeflow are in use but not in the repo yet, so a build from the repo would leave them out.

How the repository is organised

Each service in the repository is a package with the same layout. The single sign-on service looks like this:

services/keycloak/
├── intent.md        # why it exists, what it must and won't do
├── README.md        # how to run it day to day
├── manifests/       # the Kubernetes files it applies
└── scripts/
    ├── deploy.sh    # installs the service
    ├── verify.sh    # checks that it works
    └── teardown.sh  # removes it

The scripts are safe to run twice, and a person can run them without any AI. An earlier post covers intent files in more detail.

An AI agent writes the scripts from each intent file, and a person reviews every change. One rule applies to every package: a service isn't finished until it can be removed and installed again from the repository alone. Building a replica tests that rule across the whole platform at once.

The repository also holds a lessons document. It records which decisions can't be undone and which mistakes tend to come back.

How this maps to CI/CD

The method maps onto the stages of a CI/CD pipeline:

Build: each package's deploy script.
Test: each package's verify script.
Deliver: Argo CD, which keeps production in step with the repository's main branch. This is called GitOps.
Release check: a reviewed pull request for every change that reaches production.

Not all of it is automated yet. Today, building the replica isn't a pipeline: the agent runs it step by step, and a person approves each step. The only job that runs automatically on every change is a scan for leaked secrets. Making the whole build automatic is the next step, covered at the end of this post.

The replica has its own repository

The replica, called dev, lives in a separate git repository that starts from production's. A script swaps production's addresses, hostnames and secret locations for dev ones. Settings that differ between the two are variables: production's value is the default, and dev sets its own values in one file.

Every change made in dev gets one of two labels:

Environment-neutral: safe for production, so it can be sent back.
Dev-only: never leaves dev.

Fixes go back through a promotion script. It copies them onto production's repository, stops if any dev setting slipped in, and opens a draft pull request. The pull request asks two questions: what was tested in dev, and what dev can't test. The dev results are evidence for the reviewer, and a person still reviews and merges every change.

Dev is updated from production at least once a week to keep the two close.

A dev branch inside production's repository would be riskier. Argo CD watches production's main branch and applies changes automatically, so one wrong merge from a dev branch would put dev settings into production.

Diagram titled Production stays upstream of the test environment. Top row, left to right: the production repo, the source of truth, where Argo CD syncs main automatically and reverts drift; a retarget script that rewrites production addresses, hostnames and secret paths to dev values; and the dev repo, where every commit is labelled environment-neutral or dev-only. An arrow from the dev repo leads to the dev environment: fresh VMs, its own storage cluster and no route to production data. Bottom row, the way back: environment-neutral commits go to the promotion script, which cherry-picks them, fails if any dev value came along and opens a draft pull request; a person reviews and merges it into the production repo. Dev-only commits never leave the dev repo. An amber dashed box lists the guards set up before the build: production hosts removed from every node list, ssh and scp wrappers that refuse production addresses even behind an alias, and two guard checks for production storage paths and hosts that must print nothing after every upstream merge. — Changes flow down from production through a rewrite script. Fixes go back up only through the promotion script and a reviewed pull request.

Guards go in before the agent touches a VM:

Production servers are removed from every list of machines.
ssh and scp are wrapped so they refuse any production address, even one hidden behind a nickname.
Two checks look for production storage paths and production hosts after every update from production. Both must find nothing.
Dev has its own storage cluster, so it can't reach production data at all.

Today: an AI agent does the work, and a person approves each step

The instructions to the agent are mostly limits: never connect to production, never read production secrets and never push to production's repository. The build is split into six phases, from the operating system up to the GPUs. After each phase, the agent stops, shows what the verify scripts report and waits for approval.

Some settings can't be changed later without starting again. On the identity server, FreeIPA, these are the realm name and the range of user IDs. A person decides them and writes them into the instructions, and the agent asks for confirmation before installing. The earlier post shows what goes wrong when the ID range is left to the installer.

Between approvals, the agent runs each package's scripts, reads the results, adjusts to what it finds and fixes what breaks. Fixes that would also help production go into their own commits, ready to send back. The person's job is to read the results and say yes or no.

Diagram titled Six phases, with a person at every gate. After each phase the agent stops, shows the verify output and waits for an OK. A safety step comes first: production hosts out of every node list, ssh and scp guards in place, guard checks clean. Then six phases in order, each ending at a gate: 1 OS baseline, patched Ubuntu on every node; 2 Kubernetes, a highly available cluster built from empty VMs; 3 FreeIPA and node logins, marked as a step that can't be undone, where a person sets the realm name and ID range in the prompt and confirms them before the installer runs; 4 Argo CD, synced from the replica's repo; 5 Platform services, package by package; 6 GPU stack, the GPU Operator and drivers. Under the phases, who does what. The agent runs each package's deploy and verify scripts in order, adapts to what it finds on the nodes, fixes what a fresh build breaks and puts environment-neutral fixes in their own commits. The person sets the boundaries in the prompt, makes the decisions that can't be undone, reads each verify output and says yes or no at every gate. — How the build runs today. The agent does the work between gates, and a person makes the decisions that can't be undone and gives every go-ahead.

What a replica build catches

A production platform drifts away from its own scripts. Fixes go into the running system, and nobody builds the platform from empty machines again until a disaster forces it. A replica build runs every step from empty machines and finds failures such as these:

Timing races. Calico, the cluster's network plugin, creates its own resource definitions (CRDs) only once it is running. A script that applies them too early logs a warning and carries on without a pod network.
Teardown that leaves things behind. A virtual IP address still bound, or network routes still in place, stop the next install from starting.
Stale version pins. A pin that names a release since replaced by a patch round fails its own pre-install check.

In production, these steps only run during a rebuild after a disaster. A replica build finds them in a test instead.

What the replica can't test

A replica is usually smaller than production. It may have one GPU in a VM instead of bare-metal GPU servers with fast RDMA networking, and some production services may be left out. These gaps go into the "what dev can't test" part of each pull request, so the reviewer knows which changes production will test first.

Some gaps aren't in any script, such as a setting on a storage system that another team owns. The build lists them: network addresses, VM builds and storage settings that still live in people's heads or other teams' tools. Once the list is written down, someone new can rebuild the platform without help from the original team.

Next: no person at the gates

The next step is to automate the whole process, so a person no longer has to act as gatekeeper. Most of the checks already decide pass or fail on their own: the verify scripts, the guards against touching production, and the promotion script's check for dev settings. In a fully automated run, the agent builds the replica, runs every check, fixes what fails and opens the pull request without waiting for anyone, with the automated checks acting as the gates.

A verify script that misses a failure would then let it through unnoticed, so the checks need the same review a person gives each phase today.

Upgrades take the same path

Upgrades go through the replica just like fixes. A new GPU driver, inference engine or gateway version is tried in dev first, and reaches production only through a reviewed pull request.

Run this with your agent

The method is written down so that an AI agent can follow it. To set it up for your own platform, give Claude Code or another coding agent access to your platform's repository and paste this prompt:

Read https://myvirtualcloud.net/mlops-production-replica-from-the-repo/ and apply its method to this repository.

1. List the services this platform runs and propose one package per service under services/<name>/, each with intent.md, README.md, manifests/ and scripts/deploy.sh, verify.sh and teardown.sh. Show me the list before writing anything.
2. For each package, write intent.md first, then the three scripts. The scripts must be safe to run twice and must work without an AI agent.
3. Turn every value that differs between environments into a variable, with the production value as the default.
4. Create a separate replica repository that starts from this one. Add a script that rewrites production addresses, hostnames and secret paths to replica values, and label every replica commit as either environment-neutral or replica-only.
5. Write a promotion script that copies environment-neutral commits onto this repository, fails if any replica value is present, and opens a draft pull request that states what was tested in the replica and what the replica can't test.
6. Before touching any machine, add the guards: remove production hosts from every node list, wrap ssh and scp so they refuse production addresses, and add checks for production hosts and storage paths that must find nothing.
7. Build the replica in phases, from the operating system up. After each phase, stop, show the verify output and wait for my approval.
8. Never connect to production and never read production secrets. Changes reach this repository only through the promotion script's pull requests, never by pushing to its main branch.
9. Ask me before setting anything that can't be changed later without starting over.
10. Keep a list of everything the build needs that isn't in the repository.

The rules of the method, as a checklist for you or your agent:

Production's repository is the single source of truth.
Every service is a package with an intent file, a README, its manifests and three scripts: deploy, verify and teardown.
Scripts are safe to run twice, and a person can run them without an AI agent.
A service is finished only when teardown followed by deploy rebuilds it from the repository alone.
Values that differ between environments are variables with production's value as the default. The replica sets its own values in one file.
The replica lives in its own repository and is updated from production at least once a week.
Every replica commit is either environment-neutral or replica-only, never a mix of both.
Fixes reach production only through the promotion script and a reviewed pull request that states what was tested and what wasn't.
Guards against touching production are in place before the agent touches any machine.
The build runs in phases, and each phase ends with its verify scripts. Today a person approves each phase. In a fully automated run, the checks are the gates.
A person makes every decision that can't be undone.
Upgrades take the same path as fixes.
Anything the build needs that isn't in the repository is written down.

If you run an AI platform, build a replica from your repository on a few spare VMs and send your next GPU upgrade through it before it reaches production. If you want to compare notes on intent files or agent-driven builds, get in touch.