Search papers, labs, and topics across Lattice.
This paper introduces TACTICL, a novel framework for compressing tabular in-context learning models by automating the pruning of transformer layers and integrating lightweight adapters tailored for specific tasks. The approach allows for the reduction of model size and inference costs while preserving in-context adaptability, achieving a remarkable 85% layer reduction without significant performance loss across 47 benchmark datasets. Additionally, TACTICL demonstrates resilience to data shifts, ensuring that the model's in-context capabilities remain intact, thus providing a robust solution for optimizing tabular foundation models.
Compressing tabular models by 85% without sacrificing performance could revolutionize how we deploy foundation models in resource-constrained environments.
The strong performance of foundation models for tabular tasks comes at substantial inference costs. Distilling models into task-specific architectures reduces model size and computational demands but also sacrifices in-context adaptability. Here we introduce TACTICL, an automated task-aware compression framework for tabular in-context learning models that jointly prunes transformer layers and replaces them with lightweight adapters trained on downstream tasks, thus blending in-context with in-weight learning. We study TACTICL on 47 benchmark datasets and show that we can substitute up to 85% of layers without substantial performance drop on a given downstream task. We further show that TACTICL maintains robustness to data shifts, leaving its in-context ability intact. Overall, TACTICL provides a robust framework for exploiting the depth-wise redundancy of tabular foundation models by combining task-specific adaptation and structured compression. We provide the code at: https://github.com/Hebog/tfm_compression
