Search papers, labs, and topics across Lattice.
This paper investigates the potential of Multimodal Large Language Models (MLLMs) to replace human annotators for image tagging, finding that MLLMs can reduce annotation costs by orders of magnitude while achieving reasonable annotation quality. They identify a performance gap between MLLM-generated and human annotations and propose TagLLM, a framework comprising candidate generation via structured prompting and label disambiguation through interactive calibration. Experiments demonstrate that TagLLM significantly narrows the performance gap, closing 60-80% of the difference in downstream training performance compared to human annotations.
MLLMs can slash image annotation costs by 1000x while TagLLM closes the performance gap with human annotations by 60-80%.
Image tagging, a fundamental vision task, traditionally relies on human-annotated datasets to train multi-label classifiers, which incurs significant labor and costs. While Multimodal Large Language Models (MLLMs) offer promising potential to automate annotation, their capability to replace human annotators remains underexplored. This paper aims to analyze the gap between MLLM-generated and human annotations and to propose an effective solution that enables MLLM-based annotation to replace manual labeling. Our analysis of MLLM annotations reveals that, under a conservative estimate, MLLMs can reduce annotation cost to as low as one-thousandth of the human cost, mainly accounting for GPU usage, which is nearly negligible compared to manual efforts. Their annotation quality reaches about 50\% to 80\% of human performance, while achieving over 90\% performance on downstream training tasks.Motivated by these findings, we propose TagLLM, a novel framework for image tagging, which aims to narrow the gap between MLLM-generated and human annotations. TagLLM comprises two components: Candidates generation, which employs structured group-wise prompting to efficiently produce a compact candidate set that covers as many true labels as possible while reducing subsequent annotation workload; and label disambiguation, which interactively calibrates the semantic concept of categories in the prompts and effectively refines the candidate labels. Extensive experiments show that TagLLM substantially narrows the gap between MLLM-generated and human annotations, especially in downstream training performance, where it closes about 60\% to 80\% of the difference.
