LLM Request Scheduling Policies: Latency vs Fairness in Production Serving

In high-throughput enterprise LLM serving platforms, hundreds of concurrent users submit heterogeneous requests: short, latency-sensitive conversational queries intermingle with massive 100-page document summarizations and batch synthetic data jobs. If the inference gateway relies on naive First-Come, First-Served (FCFS) scheduling, a burst of heavy document processing requests will saturate GPU tensor cores and monopolize KV cache memory, causing Time-to-First-Token (TTFT) for interactive chat users to degrade from 40ms to several seconds. Conversely, aggressively prioritizing short requests risks starving complex analytical queries indefinitely. Designing production LLM request scheduling policies requires balancing latency guarantees against multi-tenant fairness across continuous batching engines. On OneSource Cloud bare-metal GPU clusters, dedicated compute eliminates virtualization jitter and enables fine-grained scheduling optimization.

Scheduling Policy Comparison: Latency vs Fairness Trade-Offs

Mathematical Model of Fair-Share Scheduling Delay

Under Weighted Fair Queuing, the virtual completion time ($V_i(t)$) for request $i$ belonging to tenant $k$ with assigned weight $w_k$ and token length $L_i$ is governed by:

$$V_i(t) = \max\left( V_{i-1}, \text{CurrentVirtualTime}(t) \right) + \frac{L_{i,\text{prompt}} \times C_{\text{prefill}} + L_{i,\text{gen}} \times C_{\text{decode}}}{w_k}$$

Where $C_{\text{prefill}}$ and $C_{\text{decode}}$ represent relative computational cost coefficients. By calibrating weights such that interactive user classes receive $w_{\text{interactive}} = 4.0$ and batch pipelines receive $w_{\text{batch}} = 1.0$, interactive users experience immediate responsiveness while batch jobs are guaranteed 20% of continuous GPU compute capacity. Deploying these schedulers on OneSource Cloud bare-metal instances ensures that scheduling decisions are executed without operating system hypervisor preemption noise.

Production Priority Scheduler with Anti-Starvation Aging

The following Python scheduler demonstrates multi-queue request dispatching with dynamic priority aging, ensuring short requests are prioritized without starving long-running batch jobs:

import time
import heapq
from typing import Optional, Dict, Any
class LLMRequestScheduler:
    def __init__(self, starvation_threshold_seconds: float = 10.0):
        self.starvation_threshold = starvation_threshold_seconds
        self.queue = []  // Stores (effective_priority, arrival_time, request_dict)
        self.counter = 0
    def submit_request(self, req_id: str, prompt_tokens: int, base_priority: int, tenant_id: str):
        // Base priority: 1 = Interactive/High, 5 = Batch/Low
        arrival = time.time()
        heapq.heappush(self.queue, (base_priority, arrival, self.counter, {
            "req_id": req_id,
            "tokens": prompt_tokens,
            "base_priority": base_priority,
            "tenant_id": tenant_id
        }))
        self.counter += 1
    def dispatch_next_batch(self, max_tokens: int = 4096) -> list:
        now = time.time()
        reordered_queue = []
        // Dynamic Priority Aging: Prevent starvation of low-priority batch jobs
        while self.queue:
            prio, arrival, count, req = heapq.heappop(self.queue)
            wait_time = now - arrival
            // Promote request if wait time exceeds starvation threshold
            adjusted_prio = prio
            if wait_time > self.starvation_threshold and prio > 1:
                adjusted_prio = 1
                req["promoted"] = True
            reordered_queue.append((adjusted_prio, arrival, count, req))
        heapq.heapify(reordered_queue)
        // Assemble batch within token budget
        batch = []
        accumulated_tokens = 0
        remaining_queue = []
        while reordered_queue:
            prio, arrival, count, req = heapq.heappop(reordered_queue)
            if accumulated_tokens + req["tokens"] <= max_tokens:
                batch.append(req)
                accumulated_tokens += req["tokens"]
            else:
                remaining_queue.append((prio, arrival, count, req))
        // Restore remaining requests to queue
        for item in remaining_queue:
            heapq.heappush(self.queue, item)
        return batch
// Example Run
scheduler = LLMRequestScheduler(starvation_threshold_seconds=5.0)
scheduler.submit_request("batch-job-1", 2048, 5, "analytics_team")
scheduler.submit_request("chat-user-1", 128, 1, "vip_customer")
dispatched = scheduler.dispatch_next_batch(max_tokens=4096)
print(f"Dispatched Batch ({len(dispatched)} requests): {[r['req_id'] for r in dispatched]}")

Operationalizing Chunked Prefill in Continuous Batching

Modern inference engines (such as vLLM with --enable-chunked-prefill) decouple prompt prefill from decode cycles. Rather than processing a 16,000-token prompt in a single massive kernel execution that blocks ongoing token generation for 250ms, the engine splits the prompt into 512-token increments. In each iteration, 512 prefill tokens execute alongside ongoing token decoding for active chat sessions. This flattens latency variance and ensures consistent token delivery. Operating on OneSource Cloud bare-metal GPU servers provides maximum HBM3 memory bandwidth, ensuring that multi-tenant KV caches remain resident without costly page swapping.

Frequently Asked Questions

What is the difference between Time-to-First-Token (TTFT) and Time-per-Output-Token (TPOT)?

TTFT is the latency required to ingest the prompt and generate the first token (compute-bound by prefill). TPOT is the time required to generate each subsequent token (memory-bandwidth bound by decode). Schedulers optimize TTFT by prioritizing prefill and TPOT by batching decodes.

Why does strict priority scheduling cause problems in enterprise APIs?

Strict priority allows high-priority streams to consume 100% of GPU compute during traffic spikes. If customer-facing chat traffic rises, background batch jobs (e.g., invoice processing, reporting) stall completely, creating massive operational backlogs.

How does OneSource Cloud enhance LLM serving performance?

OneSource Cloud delivers dedicated bare-metal GPU servers with non-virtualized PCIe Gen5 and NVLink fabrics. This delivers peak raw memory bandwidth and compute performance, enabling schedulers to pack higher batch sizes without latency spikes.