Search papers, labs, and topics across Lattice.
BehaviorVLM is introduced as a finetuning-free vision-language framework that leverages VLMs and LLMs to address pose estimation and behavioral understanding in freely moving animals. It uses a multi-stage pipeline with temporal, spatial, and cross-view reasoning for pose estimation, reducing annotation effort and exposing low-confidence labels. For behavior understanding, it integrates deep embedded clustering, VLM-based video captioning, and LLM-based reasoning for scalable and interpretable analysis directly from visual information.
Unlock scalable, interpretable animal behavior analysis without task-specific finetuning by guiding VLMs and LLMs through explicit reasoning steps.
Understanding freely moving animal behavior is central to neuroscience, where pose estimation and behavioral understanding form the foundation for linking neural activity to natural actions. Yet both tasks still depend heavily on human annotation or unstable unsupervised pipelines, limiting scalability and reproducibility. We present BehaviorVLM, a unified vision-language framework for pose estimation and behavioral understanding that requires no task-specific finetuning and minimal human labeling by guiding pretrained Vision-Language Models (VLMs) through detailed, explicit, and verifiable reasoning steps. For pose estimation, we leverage quantum-dot-grounded behavioral data and propose a multi-stage pipeline that integrates temporal, spatial, and cross-view reasoning. This design greatly reduces human annotation effort, exposes low-confidence labels through geometric checks such as reprojection error, and produces labels that can later be filtered, corrected, or used to fine-tune downstream pose models. For behavioral understanding, we propose a pipeline that integrates deep embedded clustering for over-segmented behavior discovery, VLM-based per-clip video captioning, and LLM-based reasoning to merge and semantically label behavioral segments. The behavioral pipeline can operate directly from visual information and does not require keypoints to segment behavior. Together, these components enable scalable, interpretable, and label-light analysis of multi-animal behavior.
