AI Orchestration Platform: How It Works on a Private GPU Cluster

An AI orchestration platform is a software layer that schedules GPU resources, manages training jobs and model deployments, and gives multiple teams governed self-service access to shared infrastructure. On a private GPU cluster, it decides which team's job runs on which GPUs, what happens when demand exceeds capacity, and how idle GPUs are lent out and reclaimed. This page explains the mechanisms behind those decisions, from quota borrowing and gang scheduling to MIG partitioning and utilization metrics, so teams can evaluate platforms on how they work rather than on feature lists.

Why Private GPU Clusters Need Orchestration

Clusters without orchestration show the same symptoms in almost every organization. Each symptom maps to a specific platform capability.

Foundations: Kubernetes, Slurm, or Both

Orchestration platforms build on one or both of two schedulers. Understanding their defaults explains why a platform layer is needed.

Many organizations run Kubernetes for inference and development and Slurm for large training, or run training on Kubernetes with a batch scheduling add-on. The platform layer gives users one entry point and applies policy consistently across both.

Batch Scheduling on Kubernetes: Kueue, Volcano, and KAI Scheduler

Three open-source projects are commonly used to add AI-oriented scheduling to Kubernetes.

Each project solves part of the problem. Teams still need identity integration, developer workspaces, model serving, monitoring, and usage reporting on top, which is what a complete orchestration platform provides.

How Quotas and Borrowing Work: A 64-GPU Example

Consider a 64-GPU H100 cluster shared by four groups. Static allocation would split it four ways and leave idle GPUs stranded. A quota model with borrowing works differently.

Here is how a busy afternoon plays out. Research submits a 32-GPU training job. It has 24 GPUs reserved, so the scheduler checks whether 8 more can be borrowed from idle Applied ML capacity. Because the job needs all 32 GPUs at once, gang scheduling either starts every pod together or keeps the job queued, so no GPUs are held idle while it waits. Later, Applied ML submits work that needs its reserved GPUs back. The scheduler preempts the borrowed portion, and the research job resumes from its last checkpoint when capacity frees up.

Production inference sits outside the borrowing group. Its 16 GPUs are never lent, which protects latency targets at the cost of some idle capacity overnight. The 8-GPU development pool is partitioned into MIG slices so many users share it.

This model only works if training jobs checkpoint frequently and restart cleanly. Preemption without good checkpointing wastes the compute it was meant to save.

GPU Sharing Options Compared

Not every workload needs a full GPU. Four mechanisms let several workloads share one.

For a multitenant development pool, MIG is usually the right default because it gives each user predictable memory and keeps one user's crash from affecting others. Time-slicing is simpler to set up but should be limited to workloads from the same team.

Measuring Utilization Correctly

Orchestration decisions depend on accurate utilization data. The most common metric is misleading on its own.

A notebook that shows 100% GPU utilization but 2% Tensor Core activity is a candidate for a MIG slice. A training job with low SM activity often points to a data loading or network bottleneck rather than a scheduling problem.

Chargeback: Turning Usage Into Cost

A platform that meters GPU-hours by team makes chargeback straightforward. A common internal rate is:

Internal rate per GPU-hour = total monthly cluster cost ÷ (GPU count × hours in month × target utilization)

Total monthly cost should include hardware or contract cost, operations, storage, networking, and facility. Dividing by target utilization rather than raw capacity means teams pay for the cluster's real cost even when some capacity sits idle. Many organizations charge reserved quota in full and charge borrowed GPU-hours at a lower rate, which encourages teams to use idle capacity without reserving more than they need.

Where the OnePlus Platform Fits

The OnePlus Platform is OneSource Cloud's AI orchestration platform for private GPU clusters. It brings the layers above into one product: an infrastructure portal for unified monitoring and management, a PaaS studio for Kubernetes lifecycle management, and a developer hub with Jupyter notebooks, Kubeflow, inference services, and GitHub or GitLab integration. Because it runs on dedicated infrastructure designed and operated by OneSource Cloud, scheduling policy, MIG configuration, and hardware operations have a single point of accountability. Details are on the OnePlus Platform page.

Evaluation Scorecard

Score each platform from 1 to 5 against your own priorities. These questions test real behavior rather than feature checkboxes.

FAQ

What is the difference between an AI orchestration platform and an MLOps platform?

An AI orchestration platform manages infrastructure: scheduling GPUs, enforcing quotas, and running jobs and endpoints. An MLOps platform manages the model lifecycle: experiments, registries, and pipelines. They overlap at model deployment, and many organizations run both.

Should we use Kubernetes or Slurm for AI training?

Slurm is a mature choice for large batch training and is familiar to HPC teams. Kubernetes is stronger for inference, pipelines, and container-native workflows, and adds batch features through Kueue, Volcano, or KAI Scheduler. Many organizations use both behind one platform.

What is gang scheduling and why does distributed training need it?

Gang scheduling starts all pods of a job together or not at all. Without it, a 32-GPU job can start some pods, hold those GPUs, and wait indefinitely for the rest, blocking other work.

Is MIG better than time-slicing for sharing GPUs?

For multitenant use, usually yes. MIG gives each instance dedicated memory and compute, so one workload cannot exhaust another's memory. Time-slicing is simpler but offers no isolation, so it suits trusted workloads from one team.

How do I know if my GPU cluster is underused?

Look beyond the basic GPU utilization counter. Low SM activity or Tensor Core activity on GPUs that show as busy means allocated GPUs are doing little work, often because of oversized allocations, idle notebooks, or data bottlenecks.

Summary

An AI orchestration platform turns a private GPU cluster into a shared, governed service. The mechanisms that matter most are gang scheduling for distributed jobs, quotas with borrowing and reclaim, GPU sharing through MIG, accurate utilization metrics, and GPU-hour metering for chargeback. Open-source projects such as Kueue, Volcano, and KAI Scheduler provide parts of this. A complete platform adds identity, developer workspaces, model serving, and operations. Evaluate options with real test scenarios, and decide early who will operate the platform over time.

To see these mechanisms running on dedicated GPU infrastructure, explore the OnePlus Platform.