Search papers, labs, and topics across Lattice.
This paper introduces a human-in-the-loop learning framework that goes beyond simple labeling by incorporating item ranking and exemplar selection queries to train binary classifiers. The authors model human responses to these queries probabilistically, based on the relationship between perceived item scores and distance to the classifier. They then develop active learning algorithms that leverage these rich queries, along with a variational approximation for computational efficiency, and demonstrate significant reductions in sample complexity and learning time compared to label-only active learning, achieving a 57% reduction in word sentiment classification.
Stop treating humans as mere labelers: ranking and selection queries can slash learning time by over 57% compared to traditional active learning.
Integrating human expertise into machine learning systems often reduces the role of experts to labeling oracles, a paradigm that limits the amount of information exchanged and fails to capture the nuances of human judgment. We address this challenge by developing a human-in-the-loop framework to learn binary classifiers with rich query types, consisting of item ranking and exemplar selection. We first introduce probabilistic human response models for these rich queries motivated by the relationship experimentally observed between the perceived implicit score of an item and its distance to the unknown classifier. Using these models, we then design active learning algorithms that leverage the rich queries to increase the information gained per interaction. We provide theoretical bounds on sample complexity and develop a tractable and computationally efficient variational approximation. Through experiments with simulated annotators derived from crowdsourced word-sentiment and image-aesthetic datasets, we demonstrate significant reductions on sample complexity. We further extend active learning strategies to select queries that maximize information rate, explicitly balancing informational value against annotation cost. This algorithm in the word sentiment classification task reduces learning time by more than 57\% compared to traditional label-only active learning.
