Search papers, labs, and topics across Lattice.
This paper introduces a budget-adaptive routing approach for edge-cloud inference collaborations that optimally decides whether to offload frames from weak models to stronger cloud models based on varying offload budgets. By implementing a competitive weak-skipping estimator that operates directly on raw pixels, the authors achieve significant reductions in computational load while outperforming traditional methods. The proposed routing mechanism not only enhances efficiency but also occasionally surpasses the performance of the strong model itself, achieving a notable increase in mean Average Precision (mAP) at specific operating points.
Budget-adaptive routing can outperform strong models while cutting per-frame latency by nearly 30%鈥攁 game changer for edge-cloud inference efficiency.
Edge-cloud inference collaborations are often designed with a routing estimator that decides whether to offload each frame from weak models at the edge to stronger models in the cloud. Existing systems place the routing estimator after the weak detector, so the weak forward pass still runs even on frames that are later offloaded. In this paper, we argue that this weak-conditioned design can be suboptimal when the offload budget varies. First, we present a competitive weak-skipping estimator (0.153 GFLOPs, about 29x lighter than the weak detector at 4.49 GFLOPs) that extracts routing signal from raw pixels, outperforming the common after-weak placement weak-conditioned baselines. Second, we show that neither weak-skipping nor weak-conditioned placement dominates across the full operating curve, and we propose budget-adaptive routing, which selects between them by offload budget via two offline-tuned thresholds. On PASCAL VOC, our budget-adaptive router traces the upper accuracy envelope of both fixed placements across the operating range. Our method reduces per-frame latency by up to 19.1 ms (about 30% lower at rho = 0.9). Besides outperforming SOTA methods, it is surprisingly stronger than the strong model (+1.7 pp over the strong model's peak mAP) at some operating points with far less compute. Artifacts are available at https://github.com/ViGeng/bgt-ada
