Search papers, labs, and topics across Lattice.
VisionAId is an innovative Android application designed to assist individuals with visual impairments by transforming a standard smartphone into a real-time visual assistant. The system employs six on-device deep learning models for tasks such as depth estimation, instance segmentation, and personalized object retrieval, while also leveraging a cloud-based language model for narrative descriptions. Key results include significant latency reduction through INT8 quantization and high accuracy in object detection, enabling users to locate personal belongings with augmented-reality guidance and multimodal feedback.
A smartphone app that empowers visually impaired users to independently locate personal objects with remarkable accuracy and speed.
Over 285 million people worldwide live with a visual impairment, for whom everyday tasks such as avoiding obstacles, locating personal belongings, recognizing familiar faces, or handling cash remain persistent obstacles to personal autonomy. Existing assistive applications are typically limited to recognizing predefined categories, depend heavily on cloud connectivity, or require dedicated hardware. We present VisionAId, an Android application that turns a commodity smartphone into a real-time visual assistant. The system integrates six on-device deep learning models (metric monocular depth estimation, instance segmentation, visual and facial embeddings, face detection, and a custom banknote detector) running entirely through ONNX Runtime, with an optional cloud large language model (Google Gemini Flash) used only for narrative scene description and automatic object labeling. A distinctive contribution is a few-shot pipeline for personal objects: the user photographs an object from several angles, and the system later locates that specific instance in the environment, guiding the user toward it with augmented-reality markers, spatial audio, and distance-proportional haptics. All feedback is multimodal (Romanian speech synthesis, voice commands, vibration). On a reference device (Samsung Galaxy S21 Ultra), INT8 quantization reduces depth latency from ~1200 ms to ~491 ms, the custom banknote detector reaches an mAP@50 of 0.986, and metric depth is calibrated to below 1 cm of error within 3 m.
