Search papers, labs, and topics across Lattice.
This paper introduces a token-oriented semantic communication framework that enhances collaborative inference in resource-constrained edge systems by selectively transmitting compressed image latents based on token-level task relevance. By leveraging a modular architecture that integrates a lightweight client-side vision transformer, a learned image compression model, and a large server-side vision transformer, the approach circumvents the challenges of high communication costs and interoperability issues associated with direct token embedding transmission. Experimental results on ImageNet demonstrate that this framework achieves superior rate-accuracy trade-offs compared to existing semantic communication methods and traditional codecs.
Selective transmission of task-relevant image latents can significantly reduce communication costs while maintaining high accuracy in edge AI applications.
Token communications realize the semantic communication principle at the granularity of transformer tokens, providing a promising direction for client--server collaborative inference in resource-constrained edge systems. However, directly transmitting token embeddings presents two practical challenges: substantial communication cost and limited interoperability across model-specific token embedding spaces. To address these challenges, we propose a \emph{token-oriented} semantic communication framework. In this framework, token-level task relevance determines which compressed image latents are transmitted, enabling token-granular transmission without directly transmitting token embeddings. The framework is modular, coordinating three pretrained components---a lightweight client-side vision transformer (ViT), a learned image compression (LIC) model, and a large server-side ViT---without end-to-end training. The key enabler is the one-to-one spatial alignment between ViT patch tokens and the LIC latent vectors, which allows token-level task relevance to directly determine which latent vectors are transmitted. Building on this alignment, token-aligned LIC selectively transmits task-relevant latents, layer-selective attention rollout estimates token relevance from a selected range of attention layers in a single forward pass, and surrogate token substitution adapts the frozen server model by optimizing a single learnable token. Experiments on ImageNet show that the proposed framework achieves a more favorable rate--accuracy trade-off than recent semantic communication schemes, hand-crafted codecs, and task-agnostic LIC models.
