Resolving the Harbor uid 10000 vs FreeIPA admin collision on Kubernetes

getent passwd 10000 on a Kubernetes node that resolves FreeIPA through sssd returns admin. Harbor's Helm chart (1.19.2, app v2.15.2) hard-codes runAsUser and fsGroup to 10000 for every component except its Valkey cache. FreeIPA installed with --idstart=10000 hands that exact number to the first account it creates, the built-in admin, and gid 10000 to the admins group. Two sensible defaults, and a registry that would write every image layer to shared NFS storage as the directory administrator.

I hit this deploying Harbor on a Kubernetes research AI platform. This covers the two ways out I evaluated, why the less obvious one won, and what broke on deploy day. It is not a Harbor tutorial.

Why you cannot just change the uid

There is no values knob. The harbor-helm project has had a request open since 2020 to make the security context configurable, with follow-up issues in later years, and the chart still ships fixed uids. OpenShift users cope by running the images under random arbitrary uids, which tells you the images tolerate it, but "tolerates" is not the same as "tested upstream".

The collision was avoidable on paper. The platform had a written uid scheme that put every service uid below 10000 precisely so it could never meet a FreeIPA account. It even had a rule: image-fixed uids always win, so check the image before you allocate. Grafana at 472 and CloudNativePG at 26 were already recorded as exceptions. Harbor slipped through because an earlier design note said "10000, accepted and bounded" without anyone running getent on a node that resolves FreeIPA through sssd. The FreeIPA install notes warned that admin takes the first DNA value. Two correct documents, never read together.

A uid is a number on disk. NFSv3 passes it straight through, and whoever holds it in your directory owns the files. Kubernetes RBAC does not change that.

Two ways out, one built and thrown away

Option A was a Kyverno mutating admission policy: rewrite runAsUser and fsGroup from 10000 to a platform-allocated 3008 as the pods are created, and leave FreeIPA alone. I built it, proved it offline against synthesised pods with the Kyverno CLI (seven mutated, none missed, one Valkey pod correctly untouched), and wired preflight and verify checks around it.

I withdrew it anyway. A mutating webhook now sat in the registry's pod-creation path. If Kyverno was down, Harbor pods would pend. If I set the failure policy to ignore, pods would silently land as 10000 and only a later check would notice. And I would be betting that every Photon-based Harbor image runs cleanly as an arbitrary uid, which others have shown but I had not.

Option B was to move admin and the admins group off 10000 and let Harbor keep the number the chart wants. One ipa user-mod and one ipa group-mod, no webhook, no arbitrary-uid bet, and the chart runs exactly as upstream tests it. The catch is obvious: it is surgery on a production identity. Red Hat's IdM documentation says deleting the built-in admin breaks things, but changing its uidNumber is allowed, with the warning that the server does not validate uniqueness and that duplicate IDs break SID generation. So the work became proving the change was safe rather than avoiding it.

What made moving the admin safe

Three facts, all of which I verified live before committing the change. First, nothing on the platform consumed admin's number. HBAC rules, RBAC and the provisioner's protected-users list all reference the name, Kerberos principals do not carry a uid, and a filesystem scan found nothing owned by 10000. Second, the just-in-time provisioner, which turns Keycloak users into FreeIPA POSIX accounts, never picks a uid itself. It calls user_add and lets FreeIPA's Distributed Numeric Assignment (DNA) plugin hand one out. Third, and this was the decisive one: DNA is a monotonic counter. The 389 Directory Server design describes dnaNextValue as the next free value, and the live counter sat at 10013 with uids 10001 and 10002 vacant from deleted early users and never reissued. Vacate 10000 and it stays vacant, without touching the provisioner.

So admin moved to 19998, inside a platform-service block I had already reserved next to the provisioner's own service account. I recorded two consequences rather than hiding them. Admin keeps the SID that FreeIPA minted from 10000 at creation, which is inert without an Active Directory trust and becomes a task if one is ever added. And a future DNA range reset or reinstall with --idstart=10000 would re-collide, so the uid scheme now has a row saying exactly that.

Deploy day: the sssd cache does not care about your restart

I scripted the change as a phase in the FreeIPA deploy tooling: backup, uniqueness check, user-mod and group-mod, flush sssd on the IPA host and all six nodes, then a per-node proof that admin resolves to 19998 and 10000 resolves to nobody. Never a UI click. Three things went wrong, none of them in LDAP.

• grep -c returns exit code 1 on a zero count, which under set -e aborted the script before it had taken the backup. Trivial, and exactly the kind of thing that only surfaces on a live run.
• A plain sssd restart does not drop cache_<domain>.ldb. One management node had resolved admin minutes earlier and kept serving 10000 after the change. The nodes do not have sss_cache installed, so the flush became stop, delete the cache and timestamp files, start.
• A cold sssd needs a few seconds before its first lookup succeeds. A two-second sleep became a thirty-second poll.

The script ran three times. The LDAP change happened exactly once, because the phase was idempotent and every rerun re-proved the end state. Harbor's own preflight refused to proceed until every node agreed, and verify finished at 32 of 32 checks including a push, pull and Trivy scan round-trip.

What to check before you hit this

Run getent passwd <uid> on a node that resolves your directory before you accept any image-fixed uid, not after the first file shows up owned by nobody. Treat your uid scheme as a registry with one row per exception and a reason on each. When a fix needs a mutating webhook in the path of something you need at 3am, ask whether the problem can be moved somewhere quieter instead. And if you change a production identity, script it, back up first, and prove the result on every node rather than the one you happen to be logged into.

Harbor remains a standing exception to the platform's own uid rule. It is written down, guarded by two checks, and that is a better position than a webhook that needs explaining during an outage. If you have hit the same collision and solved it differently, I would like to hear how.