Does FP8 Quantization Hurt LLM Inference Accuracy

Quick Answer: FP8 can reduce LLM inference accuracy on some tasks and leave others inside your tolerance. The effect is not a single yes or no. It depends on the format, the layers you convert, the eval you trust, and whether tool calls or long context sit in the product path.

FP8 quantization is an 8-bit floating-point conversion that stores weights or activations with less precision so inference uses less memory and more tensor throughput on GPUs that implement those formats. Hopper-class parts commonly expose E4M3 and E5M2. That is a numeric format choice, not a purchasing guide.

Serving owners should treat FP8 as a release candidate with a quality gate. This page explains when accuracy moves and how to measure it. It does not quote tokens per dollar or rank SKUs.

What actually changes when you switch to FP8?

Do not compare “FP8” across vendors without naming which of those rows you changed. A weight-only checkpoint and a full FP8 TensorRT engine are not the same experiment. Write the recipe next to the eval scores.

Which tasks usually move, and which often do not?

Closed-book factual quizzes sometimes stay close if the model was already well calibrated. Structured outputs, tool-calling argument names, and numeric extraction move more often because small logit noise flips a bracket or a digit. If your product is an API that must return valid JSON, that is the eval that matters.

Long-context retrieval inside the prompt is another common mover when KV or activations drop to FP8. The model may still “sound fluent” while it drops the clause you needed. Fluency is not the gate. Citation or exact-span checks are.

Classification and short classification-like routing heads can be surprisingly brittle or surprisingly fine. Do not generalize from a blog perplexity number. Perplexity can improve while the business task fails. Keep both, trust the task.

How should you measure quality before you pin FP8?

Freeze a prompt set that includes the production mix: short chat, long documents, tools, and multilingual if you serve it. Score exact match, schema validity, and a side-by-side rubric. Run BF16 or FP16 as the reference on the same prompts and the same tokenizer files.

Report deltas by slice, not one average. A 0.2% average drop can hide a 15% drop on invoice totals. The slice that pays the bills is the gate. If you do not have slices, you are not ready to change dtype.

Re-run after every engine, tokenizer, or template change. FP8 drift plus a chat-template edit will be blamed on whichever change landed last. Pin the recipe. OneSource Cloud does not publish FP8 accuracy claims; measure on your models. Dedicated serving on private AI infrastructure only makes that replay easier because the SKU and image stay still.

What is out of scope for this definition?

This page does not tell you whether to buy H100, H200, or L40S. Native FP8 support varies by GPU generation and software stack. Confirm the SKU, driver, and engine path in your own lab. L40S and similar inference cards may not be the right assumption for Hopper FP8 recipes.

This page does not compare quantization with fine-tuning as a cost tactic. That is a different article. FP8 can sit beside a fine-tuned model. It does not replace task data. If quality is already failing in BF16, shrink precision later, not first.

FAQ

Is FP8 the same as INT8 quantization?

No. INT8 is integer quantization with a different scaling story. FP8 keeps a floating-point exponent, which is why many LLM stacks prefer it on GPUs that implement it. Accuracy transfer is not automatic either way. Compare them only with the same eval slices.

Does a higher acceptance rate on speculative decoding prove FP8 is safe?

No. Speculative decoding measures draft agreement, not task correctness. You can accept more draft tokens and still emit a wrong tool name. Keep the quality gate on the product task.

Can we FP8 some layers and leave others in BF16?

Often yes, and many production recipes do exactly that for sensitive layers. Treat mixed precision as part of the recipe you pin. A “mostly FP8” engine is still a new artifact. Eval it as a new release.

Will FP8 always save GPU memory?

Weights usually shrink. KV and activation savings depend on what you actually converted. Fragmentation and kernel workspace can eat the headline. Measure resident memory on the serving image, not on a parameter-count spreadsheet.

Should we change dtype in the same week we change the serving engine?

Avoid it. You will not know which change moved accuracy. Change one production variable, keep the prompt set, and retain the previous engine until the slices pass. Speed of rollout is not a quality method.

Summary

FP8 can hurt LLM inference accuracy on tools, numbers, and long context while leaving casual chat looking fine. Name the format, the layers, and the eval slices. Pin a reference precision and do not promote an engine on perplexity alone.

If you need a still serving path to replay those slices, keep GPU SKU and image under change control. Precision remains your measurement problem, not a vendor score.