Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This review synthesizes research on compute-optimal training and parameter efficiency in LLMs, moving beyond scale maximization to explore resource-constrained environments. It highlights a shift towards deliberate allocation of parameters, tokens, compute, and hardware, examining techniques like data pruning, efficient architectures, quantization, and LoRA. The paper argues future progress must evaluate LLM efficiency holistically, considering performance relative to parameter count, compute, token allocation, and hardware constraints.
The pursuit of LLM performance is shifting from raw scale to strategic resource allocation, revealing a critical need to re-evaluate scaling laws for parameter-efficient, compute-optimal training in constrained environments.
Large language models (LLMs) have achieved substantial performance gains through increases in model size, training data, and computational resources. However, traditional scaling approaches produce diminishing returns, rising financial and environmental costs, and barriers to participation for researchers operating outside large industrial laboratories. This review examines the evolution of LLM scaling theory from empirical scaling laws to compute-optimal training, with particular emphasis on parameter efficiency, token utilization, data efficiency, and resource-constrained environments. Foundational work on scaling laws is synthesized alongside later research on compute-optimal training, data pruning, efficient architectures, quantization, low-rank adaptation, and edge-oriented optimization. The literature indicates a shift from scale maximization toward more deliberate allocation of parameters, tokens, compute, and hardware resources. At the same time, important empirical, theoretical, and methodological gaps remain regarding whether scaling principles established on enterprise-grade infrastructure generalize to smaller models and constrained computing environments. This review organizes these developments into a unified framework for resource-efficient LLM training and argues that future progress should evaluate efficiency not solely through model performance, but through the relationship among performance, parameter count, computational cost, token allocation, and hardware constraints.