Writing GPU Scheduling Policy for Research and Production Clusters

Scheduling research has produced the mechanisms — fair-share, preemption, elasticity, time budgets — and operators have published the practice, yet the artifact an organization actually needs is neither: it is a written policy that its scheduler can enforce and its researchers will accept. Between those requirements sits real design work: which classes exist, what floors protect them, when research jobs may be preempted and with what notice, and who decides when the rules conflict. This page is the writing procedure.

Prerequisites: What the Policy Must Decide

A mixed-cluster policy must decide five things up front: the workload classes (production serving, production training, interactive research, batch research), the protection floor each class guarantees, the sharing budget for everything above floors, the preemption relationship between classes, and who arbitrates when the rules produce a conflict. The scheduling literature and operator practice supply the mechanism vocabulary — fairness, elasticity, priorities, time budgets — but the policy work is deciding your organization's answers to the five, because a scheduler can enforce any rules and cannot invent them.

The five decisions deliberately precede mechanism choice because they are organizational facts, not technical ones: the classes reflect how the organization actually produces value, the floors reflect what failure costs, and the arbitration reflects who holds trust. Mechanisms — the scheduler features — implement the answers; teams that start from mechanisms end up with fair-share policies that are perfectly fair about the wrong things.

Writing the Rules: Classes, Budgets, Preemption, Reuse

Write the rules in four parts. Classes and floors: production serving gets a hard reservation floor; production training gets a scheduling window class; research gets the remainder-plus-reuse. Budgets: apply time-budget fair-share within research — operator practice shows budgets with fair-share allocation and time-slicing keep research equitable where raw queues breed politics, and time-aware fair-share is now native in Kubernetes GPU tooling. Preemption: research jobs may borrow above floors but are preemptible by production with a checkpoint-notice window — preemption policy with checkpoint terms protects long runs from silent loss. Reuse: idle production capacity flows to research automatically and reclaims automatically, which is the rule that makes sharing cheaper than separation.

The survey literature confirms the dimensions these rules move along — fairness, placement, elasticity, priorities — and operator practice at research organizations shows the time-budget form working at scale: budgets with fair-share allocation and time-slicing prioritize high-impact work without starving the bench. What neither supplies is your numbers, and the writing procedure exists to force them onto paper: floors stated in GPUs, budgets in hours, notice windows in minutes, and the reclaim behavior named so nobody is surprised by it.

Verification: Does the Policy Hold Under Conflict?

Verify the policy the way you verify infrastructure — under conflict, not under agreement: stage a production burst against a running research borrow and confirm the checkpoint-notice preemption fires and the research job resumes from checkpoint rather than restarting; drain the cluster and confirm floors hold; audit a week of allocations against the stated budgets. Then verify the governance: name the arbiter for rule conflicts, publish the allocation report to both constituencies monthly, and schedule a policy review each quarter — a scheduling policy without published accounting decays into folklore that everyone remembers differently.

Enforcement belongs to the scheduler, and the mechanism set this policy assumes — fine-grained quotas, preemption priority queues, topology-aware gang placement, anti-fragmentation scheduling — is exactly the delivered capability set of the OnePlus orchestration platform on OneSource Cloud enclaves: the written floors and budgets become engine-enforced behavior rather than convention, and topology awareness keeps the placement decisions the hierarchy rewards.

FAQ

Should research and production share one GPU cluster at all?

Usually yes, with rules: sharing converts research from a competing silo into a borrower of idle production capacity, and the floors-plus-reuse pattern protects production while giving research more usable hours than a separate smaller cluster — separate only when compliance boundaries demand physical separation, which dedicated enclaves solve differently.

What enforces the policy once it is written?

The scheduler's real mechanisms: quotas, priority queues with preemption, and topology-aware placement — capabilities the OnePlus platform delivers on OneSource Cloud enclaves with fine-grained quota management and anti-fragmentation scheduling, so the written floors, budgets, and checkpoint terms are enforced by the engine rather than by convention.