Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of using independently implemented training stacks as differential oracles for fine-tuning language models, specifically through a case study of a LoRA adaptation of Qwen3-0.6B on clinical data. By defining a trajectory-level protocol that includes shared specifications and cross-check points across different frameworks, the authors demonstrate that two distinct implementations yield remarkably similar performance metrics, with an average difference in held-out cross-entropy of just 0.134%. Notably, the comparison revealed 17 previously undetected faults, with the most impactful being a rendering mismatch in clinical text that significantly affected model performance, underscoring the importance of implementation diversity in neural network training.
Implementation diversity can uncover critical faults in language model training that single-stack approaches might miss, with one error impacting performance 500 times more than arithmetic issues.
Neural network training has an oracle problem: a run can converge normally and yield a usable model while the software beneath it computes something other than specified. Almost all such work runs on one stack, so there is rarely anything independent to check against. We study whether independently implemented training stacks can serve as differential oracles for a whole fine-tuning pipeline, rather than the operators and inference paths that prior differential testing targets. We define a trajectory-level protocol -- a shared specification, cross-check points spanning arithmetic, model loading, data rendering and the learning trajectory, and a separation of independence of the stack, the orchestration and the language runtime -- and apply it to a LoRA adaptation of Qwen3-0.6B over 168,574 clinical question-answer pairs under PyTorch and under numbat, an independent framework written in Zig, driven natively and through its C interface from six languages. Across 42 paired evaluations spanning a full epoch the two stacks'held-out cross-entropy differs by 0.134% on average, and four implementations end the epoch within 0.15% of one another. The comparison exposed 17 faults that single-implementation development had missed, two of them notable for software engineering. The fault with the largest effect on the trained model lay outside the numerical kernels: a mismatch in how clinical text was rendered moved held-out loss 0.15, some 500 times more than the arithmetic faults found beside it. And four faults were reachable only from a language whose memory model differs from the first two implementations: a scheduler migrating work across threads, a collector blind to device memory, an ownership discipline needing a primitive the interface lacked. Implementation diversity has several axes, and the runtime is one.
