Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces CoSense-LLM, an edge-first framework that converts multimodal sensor data into semantic tokens and coordinates with LLMs while considering latency, energy, bandwidth, and privacy constraints. CoSense-LLM employs a lightweight encoder (SenseFusion), edge-based retrieval (Edge-RAG), cost-aware prompt routing, and secure execution to minimize data transmission and ensure privacy. Experiments across diverse environments demonstrate that CoSense-LLM achieves sub-second latency, reduces bandwidth costs through local retrieval, and preserves privacy by transmitting only discrete codes.
Achieve sub-second latency and enhanced privacy in LLM deployments by pushing semantics to the edge, using a novel framework that fuses multimodal sensor data and selectively escalates to the cloud based on cost and uncertainty.
We present CoSense-LLM, an edge-first framework that turns continuous multimodal sensor streams (for example Wi-Fi CSI, IMU, audio, RFID, and lightweight vision) into compact, verifiable semantic tokens and coordinates with large language models under explicit latency, energy, bandwidth, and privacy constraints. CoSense-LLM has four parts: (i) SenseFusion, a lightweight encoder that aligns sensor embeddings with language and compresses them into short discrete code sequences; (ii) Edge-RAG, a local hybrid retrieval layer that grounds generation in site specific policies and notes; (iii) PromptRouter, a cost and uncertainty aware policy that selects edge only generation, edge plus retrieval, or compact cloud escalation; and (iv) Secure Execution, an auditable redaction path that enforces data minimization so raw waveforms never leave the device. The system works with modern serving optimizations, including paged or streaming KV caches, FlashAttention style kernels, speculative decoding, and quantized LoRA adapters, and supports on device personalization and federated updates under non IID drift. Across home, office, and clinic deployments, CoSense-LLM delivers grounded explanations while meeting tight service level objectives: it sustains sub second (p95) end to end latency on edge dominant paths, reduces inter tier token and bandwidth costs by preferring local retrieval grounded responses, and preserves privacy by transmitting only discrete codes and redacted metadata. Ablations show that Edge-RAG improves factual consistency and reduces contradictions, calibrated uncertainty enables selective abstention and controlled escalations, and KV plus decoding accelerators lower energy per decision. The results support an edge first design that treats semantics, privacy, and predictable latency as co equal goals for large model deployments in interference prone environments.