Customer-Managed Keys for GPU Training Security Controls

Customer-managed keys for GPU training are encryption keys the customer controls so volumes, checkpoints, and object stores stay unreadable unless a named unwrap path runs, and the useful control is whether the GPU provider can perform that unwrap without you. A checkbox labeled CMK is not the control. The unwrap path is.

Training jobs must decrypt data to put tensors in GPU memory. That moment is where a provider can become able to read weights, or can be designed not to. Ask where the KMS or HSM lives, who can call decrypt, and what happens to core dumps. The rest of this article is that sequence.

What CMK, BYOK, and HYOK actually change

GPU memory while a job runs is plaintext for the kernels. CMK does not encrypt tensors in HBM. It encrypts disks, snapshots, and object checkpoints. If the threat is a stolen volume, CMK matters. If the threat is a live support session on the node, you need session policy, not only a key ID.

Questions that expose unwrap

Can the provider’s hypervisor or storage plane decrypt without a customer-approved grant? Do training workers cache unwrapped data keys on disk? Do crash dumps include decrypted pages? Can you revoke and prove in-flight jobs die? If the answers are vague, you bought a label.

Do not assume encryption always slows training. The usual cost is at read of checkpoints and datasets, not at matmul. If someone claims CMK makes H100s unusable, ask for the I/O path, not the slogan. Slow checkpoint restore is a storage and key-unwrap latency issue, which you can measure.

Where exclusive infrastructure helps

Fewer extra admins means a shorter unwrap audience. Exclusive U.S. nodes still need a documented KMS. Private GPUs without key custody are just quieter shared risk. Demand the diagram before PHI or model weights land.

OneSource Cloud’s private AI infrastructure is exclusive capacity where that diagram can be customer-specific. AI storage is where checkpoints and keys meet. Healthcare workloads still need the healthcare AI path and HIPAA-ready language, not a claim that CMK equals compliance. OnePlus, OneSource Cloud’s AI orchestration platform, should schedule jobs as identities so unwrap grants are per workspace, not a shared admin cloud.

FAQ

Can a GPU provider unwrap customer-managed keys during training?

They can if unwrap runs in their KMS with a key policy that includes their service roles. They cannot if unwrap requires your external HSM grant and you can refuse it. Ask which of those you bought. Training must decrypt data in GPU memory either way. The question is who else can request decrypt.

Does CMK encrypt GPU memory?

No. GPU HBM holds plaintext tensors while kernels run. CMK protects volumes, snapshots, and objects at rest. Live-memory and dump policy is a separate control. Treat them as a pair. Buying only CMK leaves dumps as a hole.

Will customer-managed keys slow GPU training?

They can add latency when opening encrypted checkpoints or datasets. They should not change kernel throughput once data is on device. Measure restore time with and without unwrap on your storage path. If restore dominates step time, fix checkpoint I/O, not the GPU SKU first.

How do we revoke provider access?

Disable the grant, rotate data keys, and kill running jobs that already unwrapped. Revoke that does not stop in-memory work is incomplete. Practice the drill. A key policy PDF without a job-kill path is not revocation.

Is HYOK required for regulated training?

Not always. Some programs accept CMK in a regionally constrained KMS with tight key policy. Others require customer HSM. Match the threat: stolen disks versus provider administrators. Counsel and the CISO set that bar. Infrastructure shows the unwrap path.

Summary

CMK for GPU training is an unwrap-path problem: disks and checkpoints, not HBM magic. If the provider can decrypt without you, the checkbox failed. For exclusive U.S. training pools where key custody can be designed, evaluate OneSource Cloud private AI infrastructure and put jobs on named workspaces via OnePlus.