Search papers, labs, and topics across Lattice.
LayerTracer, a novel architecture-agnostic framework, is introduced to analyze task knowledge formation and robustness across diverse LLM architectures. It identifies "task particles" (layers where target token probability significantly increases) and "vulnerable layers" (layers most sensitive to perturbation via JS divergence). Experiments reveal that task particles reside in deeper layers, and larger models exhibit greater hierarchical robustness, providing insights into LLM architecture optimization.
Forget hand-tuning layer configurations: LayerTracer reveals the precise layers where LLMs learn and break, paving the way for automated architecture optimization.
Currently, Large Language Models (LLMs) feature a diversified architectural landscape, including traditional Transformer, GateDeltaNet, and Mamba. However, the evolutionary laws of hierarchical representations, task knowledge formation positions, and network robustness bottleneck mechanisms in various LLM architectures remain unclear, posing core challenges for hybrid architecture design and model optimization. This paper proposes LayerTracer, an architecture-agnostic end-to-end analysis framework compatible with any LLM architecture. By extracting hidden states layer-by-layer and mapping them to vocabulary probability distributions, it achieves joint analysis of task particle localization and layer vulnerability quantification. We define the task particle as the key layer where the target token probability first rises significantly, representing the model's task execution starting point, and the vulnerable layer is defined as the layer with the maximum Jensen-Shannon (JS) divergence between output distributions before and after mask perturbation, reflecting its sensitivity to disturbances. Experiments on models of different parameter scales show that task particles mainly appear in the deep layers of the model regardless of parameter size, while larger-parameter models exhibit stronger hierarchical robustness. LayerTracer provides a scientific basis for layer division, module ratio, and gating switching of hybrid architectures, effectively optimizing model performance. It accurately locates task-effective layers and stability bottlenecks, offering universal support for LLM structure design and interpretability research.
