The Acceptance Rate That Should Disable Speculative Decoding

Speculative decoding is not a feature you turn on for the cluster. It is a bet, per route, that a small draft model will propose tokens the target model would have emitted anyway. The bet pays in fewer target-model passes. It stops paying when too many of those proposals are thrown away, because you still paid the draft. The number that says which side you are on is the acceptance rate, and a launch-day average across all traffic will lie to you within a month.

Acceptance Is the Only Speedup That Survives Production

Acceptance rate is drafted tokens that the target kept, divided by drafted tokens the small model proposed. Quality does not move with it. Under a correct verifier the discarded tokens never reach the user; the target distribution is what ships. Cost and latency do move. Practitioner measurements on 70B-class targets, not a lab abstract, sort roughly like this:

The line near 40% is a production observation, not a constant in the paper. Below it, teams report the draft's extra compute wiping out the target-model savings. Recompute the line against your own latency budget: a route that is already waiting on retrieval will not feel a decode win, and a route at high batch is often throughput-bound already, so the draft adds overhead to a GPU that was not waiting on verification.

Memory is the other gate, and it is separate from acceptance. Draft plus target have to stay resident. A neighbor's batch that evicts the draft turns a good ratio into a load stall. That is a placement problem. On a dedicated serving node from OneSource Cloud the device is not shared, so the draft is not paged out by someone else's model. The ratio itself does not improve because the box is dedicated. A bad draft is still a bad draft.

Read the Counters, Then Cut by Route

Do this on production traffic for each route, not on a single canned prompt.

1. Find the engine's accepted-token counter and its drafted-token counter. On vLLM-class builds these are exposed as speculative accepted and drafted totals. Names move between versions; the ratio does not.
2. Divide, per route or per model pair, over a window long enough to include the ugly traffic. A lunchtime demo is not a window.
3. Disable speculation on any route whose class sits under the cutoff you computed in the table, even if the cluster-wide average looks fine. Code routes can stay on while the creative route goes off.
4. Record p50 and p99 for that route for a week with the flag off. If p99 does not worsen, the draft was not earning its memory.

What not to average

A single acceptance number for "the API" mixes a code assistant at 75% with a writing route at 40% and then enables the flag globally because the mean cleared 55%. The writing route pays for the code route's average. Cut by route.

Three Ways a Healthy Ratio Goes Stale

The failure mode that hurts is a ratio that was real in April and is fiction in October, with no error and no quality regression anyone filed.

• Draft drift. The target gets a finetune. The draft does not. Acceptance falls, throughput falls, and eval still passes because the verifier is doing its job. Pin the draft to the target's revision and alert on the ratio, not on eval score.
• Tokenizer mismatch. A draft from another family can look plausible and still be rejected on every other token. Same family, same tokenizer, or do not bother.
• Memory contention. If the draft and the target fight for one device, the win disappears even at a high ratio. High batch makes this worse: verification was never the bottleneck, and the draft's overhead is pure cost.

Add the ratio to the same dashboard as time to first token. A throughput drop with a flat error rate is the signature. Waiting for a user complaint measures the wrong thing, late.

Questions that decide the flag

Does a low acceptance rate change the tokens users see?

No. Tokens the target rejects are dropped, and the sampler continues from the target model's distribution. A low rate changes how much compute you spend and how long the request takes. It is not a quality setting. If outputs differ with the flag off, the implementation is wrong; do not explain that away as "the draft's style."

Does OneSource Cloud raise speculative-decoding acceptance?

No. Acceptance is fixed by the draft, the tokenizer, and the traffic on that route. OneSource Cloud only stops a neighboring workload from evicting the draft out of GPU memory on a shared device. You still measure the ratio, and you still disable the routes that fall through.