Search papers, labs, and topics across Lattice.
This study systematically investigates small language models' abstract reasoning capabilities on the ARC-TGI benchmark, differentiating between true rule acquisition and distribution-specific fitting. Across diverse model families and fine-tuning regimes, results show that while in-distribution accuracy is achievable, it is highly sensitive to optimization and unevenly distributed across task families, with significant performance degradation outside the training distribution. Attention diagnostics reveal some behavioral differences but do not establish general causal mechanisms for rule induction.
Small language models achieve high abstract reasoning scores by memorizing task distributions rather than learning generalizable rules, failing even simple out-of-distribution shifts.
Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-specific regularities. We study this distinction in small language models on the ARC-TGI benchmark, which organizes abstract grid transformations into controllable task families and supports resampling, spatial shifts, and cross-benchmark transfer. Across more than 1,000 runs, we profile decoder-only, encoder--decoder, and mixture-of-experts model families under supervised fine-tuning. We examine the efficiency and stability of skill acquisition, robustness beyond the training distribution, interactions with model family and task formulation, and layer-wise attention signatures that accompany behavioral differences. Substantial in-distribution accuracy is attainable, but acquisition is sensitive to optimization and unevenly distributed across task families. Performance deteriorates sharply outside the training distribution, including when the rule is retained but grid scale changes. Greater training-set depth and breadth yield uneven gains, while the effect of additional in-context examples depends on model family. Executable-rule induction also yields correct solutions not observed under direct grid generation. On selected tasks, attention diagnostics show distinct concentration and context-dependence profiles, but do not establish general causal mechanisms. Overall, abstract-reasoning scores are conditional on the model, adaptation regime, evaluation distribution, and response format.
