Search papers, labs, and topics across Lattice.
This paper introduces Fine-grained Captioning Control Using Scene Rewards (FoCUS), a method that enhances the controllability of image captioning by allowing users to specify semantic emphases through natural-language prompts. By employing a prompt-conditioned control objective that aligns with scene-graph components, FoCUS differentially weights these components to steer the generated captions toward desired attributes, relations, or regions. The method was evaluated using the Semantic Control and Precision Evaluation (SCoPE) benchmark, demonstrating significant improvements in controllability and caption quality across two Vision-Language Model backbones without sacrificing overall performance.
Users can now dictate the focus of image captions with unprecedented precision, steering outputs toward specific attributes or relations through simple prompts.
Large Vision-Language Models produce fluent image descriptions but offer limited semantic control: users cannot reliably specify whether captions should emphasize attributes, relations, or particular image regions. We present Fine-grained Captioning Control Using Scene Rewards (FoCUS), a controllable image captioning method that lets users steer captions toward specific semantic emphases through natural-language control prompts. The core idea is a prompt-conditioned control objective based on scene-graph-aligned component scores. Generated captions are parsed and aligned to scene-graph components such as objects, attributes, and relations. These components are differentially weighted, including negative weights, according to the requested emphasis. We optimize this objective with GRPO and further improve its reliability through a stricter object validity threshold and reasoning-based verification for attribute and relation scoring. To evaluate controllability, we introduce Semantic Control and Precision Evaluation (SCoPE), a benchmark with contrastive Include/Avoid constraints for measuring both target content coverage and out-of-scope suppression. Experiments on two VLM backbones show that FoCUS consistently improves controllability and fine-grained caption quality without degrading general caption performance.
