Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces CircuitProbe, a method to efficiently identify reasoning circuits in transformer language models by analyzing activation statistics. CircuitProbe detects "stability circuits" in early layers using the derivative of representation change and "magnitude circuits" in later layers via anomaly scoring. Validated across diverse models, CircuitProbe achieves 3-4 orders of magnitude speedup compared to brute-force search and reveals that layer duplication is beneficial for smaller models (under 3B parameters) but detrimental to larger ones.
Forget expensive brute-force searches: pinpoint transformer reasoning circuits in minutes using activation stats, and discover that duplicating these circuits boosts smaller models but hurts larger ones.
Transformer language models contain localized reasoning circuits, contiguous layer blocks that improve reasoning when duplicated at inference time. Finding these circuits currently requires brute-force sweeps costing 25 GPU hours per model. We propose CircuitProbe, which predicts circuit locations from activation statistics in under 5 minutes on CPU, providing a speedup of three to four orders of magnitude. We find that reasoning circuits come in two types: stability circuits in early layers, detected through the derivative of representation change, and magnitude circuits in late layers, detected through anomaly scoring. We validate across 9 models spanning 6 architectures, including 2025 models, confirming that CircuitProbe top predictions match or are within 2 layers of the optimal circuit in all validated cases. A scaling experiment across the Qwen 2.5 family reveals that layer duplication consistently benefits models under 3B parameters but degrades performance in 7B+ models, making this a practical scaling technique for small language models. CircuitProbe requires as few as 10 calibration examples and its predictions are stable across English, Hindi, Chinese, and French.