Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference
[Uczenie maszynowe] Autorzy: Gaoyuan Du, Anam Nawaz Khan, Rex Zhou i 4 więcej
Greedy decoding from large language models is commonly treated as deterministic. We show it is not precision-invariant: the same model, prompt, and decoding algorithm produce different outputs in BF16 versus FP16 on identical hardware. Across our evaluations of six models (1.1B-7B parameters, four families; divergence additionally characterised at 12B) and three benchmarks, 49-100\% of prompts diverge; a single token flip often cascades into trajectory-level divergence. We develop an empirical error-propagation analysis and find that 22 layers of accumulated body error do not distinguish flipping from non-flipping steps; the outcome depends primarily on the top-two logit margin at the LM head relative to the directional perturbation between the top-two candidates. The analysis makes five testable predictions about intervention outcomes, including that applying more FP32 compute (broader…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe