Search papers, labs, and topics across Lattice.
This paper introduces Molmo2Fish, an interactive tool that leverages a multimodal large language model to enhance fish tracking by allowing users to correct imperfect predictions through natural language guidance. The approach addresses the challenges of multi-object tracking in ecological datasets by facilitating a human-in-the-loop correction process, demonstrating high performance in both guided and unguided tasks. Key findings reveal that while Molmo2Fish performs well in tracking and correction, there remains significant potential for improvement in its integration of natural language inputs.
Interactive correction of fish tracking predictions via natural language guidance shows promise, yet highlights the need for better integration of user input.
Computer vision is increasingly used to automate recognition tasks in large ecological datasets, but more complex tasks such as multi-object tracking continue to pose challenges. As researchers seek to incorporate vision models in ecology workflows, various lines of research have explored how to make imperfect predictions useful through human-in-the-loop processes. We propose a new approach to working with imperfect tracking predictions through an interactive prediction correction workflow taking place as a conversation with a multimodal large language model, which we tailor to a sonar fish tracking dataset as an initial proof of concept. We investigate the performance of the tool, Molmo2Fish, across guided and unguided tasks, correcting its own predicted tracks and external tracks. We find that Molmo2Fish achieves high performance on fish tracking and track correction tasks, but there is still much room to improve on incorporating natural language guidance. The code and data are publicly available at https://github.com/tidalove/molmo2fish.
