MIG vs Time-Slicing: GPU Sharing Overhead for Inference

Quick Verdict: Use MIG when workloads need predictable latency and hard memory isolation, because it partitions the GPU in hardware. Use time-slicing for development, notebooks, and bursty low-priority jobs, where sharing capacity matters more than guaranteed performance. Use MPS when several processes from the same trusted workload need concurrent execution with low switching overhead.

GPU sharing is the practice of running more than one workload on a single accelerator, either by partitioning it into isolated slices or by interleaving execution across processes. Each mechanism reduces idle capacity, and each imposes a different kind of overhead — some in throughput, some in latency variance, some in operational rigidity.

The mistake teams make is treating these as interchangeable knobs for "fractional GPU." They provide fundamentally different guarantees, and the wrong choice produces either wasted capacity or unexplained latency spikes on a production endpoint.

GPU Sharing Mechanisms Compared

MIG: Hardware Partitioning

Multi-Instance GPU divides a supported data center GPU into separate instances, each receiving its own slice of streaming multiprocessors, L2 cache, memory, and memory bandwidth. A workload placed in one instance cannot consume another instance's memory or crowd out its bandwidth, which is why MIG is the only option here that provides a real performance guarantee.

The overhead is not in execution; it is in allocation granularity. MIG instances come from a fixed set of profiles, so a model that needs slightly more memory than a given profile provides must move to the next profile up, leaving the difference unused. Across a fleet, that stranded capacity is the true cost of hardware isolation.

The second constraint is operational rigidity. Changing the partition layout of a GPU generally requires that no workloads are running on it, so profile changes become drain-and-reconfigure events rather than live adjustments. Clusters that reconfigure frequently should standardize on a small number of layouts and route workloads to nodes that already match, rather than reshaping GPUs on demand.

Choose MIG when: multiple teams or tenants share hardware, a latency commitment exists, memory exhaustion by a neighbor is unacceptable, or individual models are small relative to full GPU memory.

Time-Slicing: Scheduler Interleaving

Time-slicing lets several processes take turns on the full GPU. In Kubernetes environments this is typically exposed by advertising a GPU as multiple schedulable units, so several pods land on the same physical device.

There are two distinct costs. The first is context switching: each swap between processes carries a cost that grows with the amount of state involved, and frequent switching between memory-heavy workloads is measurably worse than between light ones. The second, and more damaging in production, is latency variance. A request arriving while another process holds the GPU waits for its turn, so tail latency degrades in a way that averages hide.

The absence of memory protection is the sharper operational risk. Processes share the memory pool, so one workload allocating more than expected can cause another to fail on allocation. Because Kubernetes advertised "replicas" of a GPU do not correspond to reserved memory, the scheduler will happily place workloads that cannot coexist.

Choose time-slicing when: workloads are interactive-but-tolerant, such as notebooks and experiments; when utilization would otherwise be very low; and when no consumer on that GPU has a latency commitment. Avoid it for production inference endpoints.

MPS: Concurrent Execution in a Shared Context

Multi-Process Service allows kernels from multiple processes to execute concurrently on the GPU rather than being serialized. For workloads made up of many small kernels that individually cannot saturate the device, this recovers substantial throughput that time-slicing would leave on the table, and it avoids most of the context-switch cost.

The tradeoff is isolation. Processes operate within a shared context, so the separation between them is weaker than either MIG or standard process isolation. Contention is also unmanaged: a process issuing large kernels can crowd out a latency-sensitive neighbor, with no partition to enforce fairness.

Choose MPS when: the co-resident processes belong to the same team and trust domain, individual processes underutilize the GPU, and throughput matters more than per-process guarantees.

Where the Overhead Actually Shows Up

Teams evaluating GPU sharing usually measure average throughput and conclude that sharing is nearly free. Averages are the wrong instrument. Overhead concentrates in three places that only appear under realistic conditions.

• Tail latency under concurrent load: Time-sliced workloads show acceptable median latency and badly degraded p95 and p99 once neighbors are active. Measure percentiles with the neighbor workload running, not in isolation.
• Memory headroom collapse: Shared memory pools mean an increase in one workload's batch size can push another into allocation failure. This surfaces as intermittent errors rather than slow responses.
• Stranded capacity in fixed partitions: MIG's overhead is invisible in latency and visible in the fleet-wide ratio of allocated to usable memory. Track that ratio, or the isolation benefit gets paid for twice.

A practical evaluation runs the target production workload against a deliberately noisy neighbor and records p99 latency, allocation failures, and effective utilization over at least an hour. Anything shorter misses the interaction patterns that cause incidents.

Applying Sharing Policy Across Teams

The mechanism is only half the decision; the other half is which workload class gets which treatment. A workable default policy separates three tiers.

1. Production inference endpoints get a dedicated GPU or a dedicated MIG instance. Latency commitments and hardware isolation belong together.
2. Internal services and batch scoring get MIG instances sized to their memory profile, accepting some stranded capacity in exchange for predictable behavior.
3. Development, notebooks, and experiments get time-sliced access with idle reclamation, because utilization matters more than guarantees and the cost of a slow cell is low.

Enforcing that policy requires a scheduling layer that understands the distinction, tracks which nodes carry which partition layout, and reclaims idle allocations. The OnePlus Platform, OneSource Cloud's AI orchestration platform, handles GPU allocation and quota enforcement across teams so sharing policy is applied consistently rather than negotiated per project.

Sharing policy also depends on knowing exactly what hardware you have. Partition profiles differ by GPU model, and clusters where instance types change between deployments cannot maintain a stable policy. Fixed hardware inventory on private AI infrastructure makes partition layouts durable rather than something the platform team re-derives each quarter.

FAQ

Does MIG reduce total GPU performance?

Aggregate throughput across instances remains close to the full device for workloads that fit their partitions, because the partitioning is in hardware rather than in scheduling. The practical loss comes from allocation granularity: capacity inside a profile that a workload does not use cannot be lent to another instance.

Can time-slicing be used for production inference?

It can run, but it should not carry a latency commitment. Without memory protection or bandwidth partitioning, a neighboring workload can degrade tail latency or trigger allocation failures. Production endpoints belong on dedicated devices or MIG instances.

How many workloads can share one GPU?

With MIG, the answer is bounded by the profile layout the GPU supports and the memory each workload needs. With time-slicing or MPS there is no hard limit, which is precisely the problem: the scheduler will oversubscribe past the point where the workloads can coexist reliably.

Do we need MIG if our models already fit on one GPU?

If one model saturates the device, sharing adds risk without recovering capacity. MIG earns its keep when models are small relative to GPU memory and several tenants need concurrent, protected access to the same hardware.

Can MIG configuration be changed without disrupting workloads?

Changing a GPU's partition layout generally requires that the device be free of running work, so treat it as a drain-and-reconfigure operation. Clusters that need flexibility usually maintain node pools with different fixed layouts and schedule workloads to the matching pool.

Summary

MIG, time-slicing, and MPS solve different problems. MIG buys hardware-enforced isolation and pays for it in stranded capacity and reconfiguration friction. Time-slicing buys utilization on low-priority work and pays for it in tail latency and missing memory protection. MPS buys concurrency for same-trust processes and pays for it in weak separation. Match the mechanism to the workload tier, and validate the choice with p99 latency and allocation failures measured against a noisy neighbor, not with average throughput.

Consistent sharing policy depends on stable hardware and a scheduler that enforces it. OneSource Cloud delivers dedicated GPU capacity with managed AI infrastructure operations and multi-team orchestration, so utilization improves without exposing production endpoints to neighbor interference. Request an architecture review to design a sharing policy for your workload mix.