AI Dataset Authoritative Copy Governance and Shadow-Copy Risks

Ask an AI team where the authoritative version of a dataset lives and you will usually get a tour: the original in the data lake, the cleaned copy on the training cluster, the snapshot a scientist exported last quarter, the frozen subset under a compliance hold. Every stop on the tour has a reason; none of them is authoritative; and the moment two training runs consumed different stops, the lineage record became fiction. Data-versioning tools solved the mechanics years ago — the missing piece is the governance concept the tools assume.

Definition: What the Authoritative Copy Must Guarantee

The authoritative copy of a training dataset is the single store whose identity, contents, and history the organization treats as definitive — the dataset's system of record. It must guarantee three things: identity (a named dataset resolves to exactly one canonical location and one version registry), per-version immutability (a version, once published, never changes under it), and lineage (each version records its sources, transformations, and the training runs that consumed it). Copies lacking any guarantee are not backups or mirrors — they are shadow copies, whatever they are called.

Enterprise data guidance draws the underlying distinction carefully — a system of record is authoritative for one domain, a source of truth harmonizes across domains — and training datasets need exactly the former: the registry entry that says what this dataset is, which no other store may contradict. The guarantees are organizational commitments that tooling enforces; buying a versioning tool without making the commitments produces tracked shadows instead of untracked ones.

Enterprise Reality: Why Shadow Copies Form

Shadow copies form through legitimate forces, not carelessness: the training cluster caches a dataset locally because the authoritative store is slow; a data scientist exports a snapshot for offline analysis; a compliance hold freezes a subset; a migration copies rather than references. Each copy starts life with a reason and drifts as the authoritative version advances — and the moment a training run consumes a shadow, the lineage record lies, results become irreproducible, and deletion obligations (a subject's data removed upstream) silently miss every copy nobody tracked. The governance work is not banning copies — caches are how fast training works — but classifying them: derived, tracked, and disposable, versus authoritative and singular.

The audit that follows is administrative, not forensic: list every location answering the flagship dataset's name, mark which one the registry points to, and classify the rest — and run it per estate, because the registry that governs a single-tenant enclave from OneSource Cloud is a different audit surface than a shared platform's object store. The discipline pays off in the two moments that matter — the reproduction question (which exact version trained this model?) and the deletion obligation (remove this subject's data everywhere) — both of which are unanswerable precisely to the degree shadows exist.

Boundary: Versioning Patterns and the Storage Contract

Two versioning patterns dominate: pointer-based versioning (DVC-style — Git tracks hashes, storage holds objects) suits teams whose datasets fit existing object stores and whose scale is dataset-at-a-time; repository-based versioning (lakeFS-style — Git semantics over the store itself, with branches and atomic commits) suits multi-team estates needing isolated experiment branches on large data. The boundary both patterns share is the storage contract underneath: the authoritative store must deliver throughput that keeps training fed (parallel NVMe-oF tiers, not cold object storage on the hot path), durability with integrity verification, and access logging that makes the lineage guarantee auditable — the tier design delivered fabrics document for training enclaves.

Whichever pattern wins, the storage contract is the same document: throughput for the training hot path, durability with verification, and access logs that make lineage auditable rather than asserted — exactly the properties OneSource Cloud contracts for its storage tiers. That is the tier structure OneSource Cloud AI Storage Architecture delivers in dedicated enclaves — NVMe-oF parallel tiers with GPUDirect Storage paths — so the versioning layer above it inherits auditability instead of fighting the storage for it.

FAQ

Can there be more than one authoritative copy of a dataset?

One authoritative copy per dataset, by definition; multi-region estates run one authoritative origin with tracked, verifiable replicas that carry the same guarantees — anything that can drift independently of the registry is a shadow, and the test is administrative: delete the registry entry, and see how many stores still claim the dataset.

What storage should the authoritative copy live on?

On tiers that honor the contract: parallel NVMe-oF storage with GPUDirect paths for the hot training path, durable lower tiers for archived versions, and access logging throughout — the tier structure OneSource Cloud delivers in dedicated enclaves, so the versioning tool above it inherits throughput and auditability rather than fighting the storage for both.