Search papers, labs, and topics across Lattice.
This paper introduces ProbScale, a novel framework that integrates insights from neural scaling laws and probing analysis to optimize the selection of parameter-efficient subnetworks within small language models (SLMs). By mathematically quantifying the relevance of each layer for specific downstream tasks, ProbScale enables the identification of subnetworks that significantly reduce parameters while preserving high performance. Experiments with models like RoBERTa-Large and T5-Base reveal that ProbScale can achieve parameter reductions of 5 to 10 times, maintaining 95% to 98% of the original models' performance on targeted tasks, thereby outperforming existing heuristic approaches.
Achieving up to 10x parameter reduction in small language models without sacrificing performance could redefine efficiency benchmarks in model deployment.
Small Language Models (SLMs) offer a balance between capability and computational feasibility. Neural scaling laws inform their optimal training, suggesting that they possess rich internal representations that scale with their size. However, deploying even these SLMs can be challenging under strict resource constraints. Language model probing provides methods for analyzing the linguistic knowledge encoded in a model's internals. We propose ProbScale, a framework that unifies insights from scaling laws and probing to identify parameter-efficient subnetworks within pre-trained SLMs. ProbScale utilizes the high-quality representations of well-scaled SLMs and uses task-specific probes to mathematically quantify the relevance of each layer for target downstream capabilities. This allows selecting subnetworks that optimally trade off performance against parameter size. We formulate the subnetwork selection as finding a layer subset maximizing aggregated, task-weighted probe performance under a parameter budget. Experiments on representative SLMs such as RoBERTa-Large and T5-Base demonstrate that ProbScale identifies subnetworks achieving significant parameter reduction, from 5 to 10 times, while maintaining high performance (95% to 98% of the original SLMs) on targeted tasks, outperforming heuristic baselines.
