Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To overcome the blind spots of static external embeddings and naive rarity heuristics, this work selects unlabeled data by mapping an object detector's evolving internal representations directly to predicted errors and their anticipated downstream performance impact. Grounding active data selection in the target model's internal state ensures acquired annotations directly address impactful model-specific failure modes rather than non-critical edge cases. Across multiple object detection benchmarks, this approach outperformed rarity-based selection—even when rarity heuristics captured a higher raw error count—and ranked first among six competitive data-selection methods under extended retraining.
Collecting more errors doesn't guarantee better models: targeting only the high-impact failure modes signaled by a network's own internal representations outperforms standard rarity-based active learning while saving annotation budget.
With limited annotation budgets, choosing which images to label determines how much a model improves. Data-selection methods that use features from a separately trained model, or scene descriptions written by vision-language models, have been successful, but those signals do not directly capture changes in the model being improved. The target model's own internal features reflect what it has learned so far and change with retraining, making them a natural cue for choosing the next training data. However, feature rarity alone does not reveal the errors that matter for performance. Here we link internal features to prediction errors and their expected impact on performance and select images for labeling and retraining without using labels for candidate images. We evaluated the method with an object detector on two datasets and two pairs of random seeds. Adding internal features improved the identification of prediction errors in 15 of 16 conditions. When performance was averaged over successive labeling rounds, the method outperformed selection based only on feature rarity in all four evaluation settings and ranked among the top two of six methods. With other conditions held fixed, performance after retraining was again higher than with rarity-based selection, even though the latter collected more errors. With longer retraining, the proposed method ranked first among six methods. These results suggest that linking a model's internal features to its errors and their effects on performance may help select training images that improve performance, thereby allowing the model's current state to guide which images are labeled next.