Search papers, labs, and topics across Lattice.
This paper revisits human-in-the-loop object retrieval, formulating it as a binary classification task and leveraging pre-trained Vision Transformer (ViT) representations to iteratively retrieve images containing objects of interest based on user feedback. The study explores different representation strategies, annotation formats, and active learning techniques to optimize retrieval performance in complex, multi-object datasets. Results demonstrate the trade-offs between global context and fine-grained local object details, providing insights for designing effective interactive retrieval pipelines.
Pre-trained Vision Transformers can power effective human-in-the-loop object retrieval, but the devil's in the details: instance selection, annotation format, and active learning strategy all significantly impact performance.
Building on existing approaches, we revisit Human-in-the-Loop Object Retrieval, a task that consists of iteratively retrieving images containing objects of a class-of-interest, specified by a user-provided query. Starting from a large unlabeled image collection, the aim is to rapidly identify diverse instances of an object category relying solely on the initial query and the user's Relevance Feedback, with no prior labels. The retrieval process is formulated as a binary classification task, where the system continuously learns to distinguish between relevant and non-relevant images to the query, through iterative user interaction. This interaction is guided by an Active Learning loop: at each iteration, the system selects informative samples for user annotation, thereby refining the retrieval performance. This task is particularly challenging in multi-object datasets, where the object of interest may occupy only a small region of the image within a complex, cluttered scene. Unlike object-centered settings where global descriptors often suffice, multi-object images require more adapted, localized descriptors. In this work, we formulate and revisit the Human-in-the-Loop Object Retrieval task by leveraging pre-trained ViT representations, and addressing key design questions, including which object instances to consider in an image, what form the annotations should take, how Active Selection should be applied, and which representation strategies best capture the object's features. We compare several representation strategies across multi-object datasets highlighting trade-offs between capturing the global context and focusing on fine-grained local object details. Our results offer practical insights for the design of effective interactive retrieval pipelines based on Active Learning for object class retrieval.
