01The Problem: Same Model, Different Answers
We tested a deployment of Gemma 4 using the same GGUF model file and the exact same llama.cpp build. We configured it with greedy decoding and temperature zero. These settings should produce identical outputs every single time. Yet, when we ran the same grade-school math question on two different NVIDIA GPUs, we got different answers. A GTX 1650 Ti on a laptop returned $20. A Tesla T4 in a cloud instance returned 2000.
Each card was consistent with itself. The laptop consistently gave $20, and the Tesla T4 consistently gave 2000. They disagreed on the answer to the same question, using the same model, under identical conditions. Identical configurations do not always mean identical results.
02Floating Point Arithmetic Across Hardware
- This behavior stems from how floating-point numbers are handled across different hardware.
llama.cppsplits large matrix multiplications into smaller chunks, assigning each chunk to a processing unit on the GPU. Our GTX 1650 Ti has 16 such units, while the Tesla T4 has 40. When these chunks are processed and summed up, tiny differences in the order of operations and the resulting rounding offloat32numbers accumulate.
This is analogous to adding a long list of numbers. Grouping them differently changes the final sum by minuscule variations in the last decimal places. Across the many layers of an LLM, these discrepancies snowball. In our Gemma 4 test, the models ran for 124 tokens identically. At a fork between 'spent' and 'saved', the accumulated floating-point differences caused the two GPUs to diverge. Every subsequent token generated was a consequence of that initial divergence.
- This is not unique to Gemma or
llama.cpp. Numerical computations performed on parallel hardware vary based on architecture, core counts, and library implementations like cuBLAS. We observed similar divergence with different quantization levels, where Q8_0 showed more variation in wording than BF16.
03Production Implications
For some applications, output variation is negligible. For others, it introduces failures. A financial analysis LLM might provide different profit margin calculations depending on whether it runs on an on-premise server or a cloud GPU instance. A medical diagnostic assistant could alter its probability assessment based on the underlying hardware.
Hardware drift introduces a significant challenge for debugging. Establishing definitive pass/fail test criteria becomes difficult when the hardware introduces variability. A test suite can pass on a development machine and fail intermittently in production, pointing to the silicon rather than the code base.
04Mitigation Strategies
We changed our LLM deployment process to account for hardware variance.
Parallel Inference and Comparison
We run critical inference tasks on multiple hardware configurations in parallel during testing. We test on local machines alongside representative cloud instances, such as T4 and L4 GPUs, before deploying to production. We collect and compare outputs from each environment.
Tolerance Thresholds
We define acceptable tolerance levels for variations. For numerical outputs, we check if the difference between results falls within a defined epsilon. For text generation, we use semantic similarity checks to ensure key entities and sentiments remain consistent. We test for functional equivalence rather than exact token-for-token matches.
Deterministic Settings
We evaluate options to increase determinism, accepting performance trade-offs where necessary. Setting -ub 1 in llama.cpp forced the two test cards to agree on every token, though this specific setting incurred a 14% to 15% performance penalty. We tune inference engine flags to control variance.
Production Monitoring
Continuous monitoring detects drift in output quality over time. We periodically re-evaluate a subset of inputs against known ground truth and use statistical methods to identify anomalies in generated tokens.
05Sources
- [One Word, Two GPUs: Why Gemma 4 Answers the Same Question Differently on a Laptop and a Tesla T4](https://dev.to/gde/one-word-two-gpus-why-gemma-4-answers-the-same-question-differently-on-a-laptop-and-a-tesla-t4-1dj2) (dev.to)
