Search papers, labs, and topics across Lattice.
The paper identifies limitations of attention-based importance filtering for edge-cloud hybrid inference under strict uplink bandwidth constraints. It demonstrates that semantic diversity in transmitted embeddings is crucial for server accuracy, even surpassing the importance of individual high-attention units. They propose SAGE, a training-free method that combines attention-based filtering with embedding-diversity sampling, achieving 93% of server ceiling accuracy while transmitting fewer than half the evidence units on ImageNet-1K.
Naive attention-based filtering for edge-cloud inference is suboptimal under tight bandwidth constraints; prioritizing semantic diversity in transmitted embeddings yields surprisingly large accuracy gains.
Edge-cloud hybrid inference offloads difficult inputs to a powerful remote model, but the uplink channel imposes hard per-request constraints on the number of bits that can be transmitted. We show that selecting transmitted content based solely on attention-based importance, the standard approach in collaborative inference, is inherently limited under hard budgets. Two findings support this claim. First, replacing high-importance units with low-importance but complementary ones improves server accuracy. This shows that what matters is not individual importance but how well the transmitted set covers diverse aspects of the input. Second, spatially uniform selection without any content information achieves competitive accuracy at moderate budgets. This confirms that spatial coverage alone carries independent value. Based on this analysis, we propose SAGE (Semantic Attention-Guided Evidence), a principled, training-free method that combines importance filtering with embedding-diversity sampling. SAGE achieves 93% of the server ceiling in offloaded accuracy while transmitting fewer than half of the available evidence units on ImageNet-1K, substantially outperforming importance-only composition.
