Search papers, labs, and topics across Lattice.
The paper introduces UrbanAlign, a training-free post-hoc calibration pipeline to align VLMs with human preferences in subjective perception tasks. It leverages a concept-bottleneck approach consisting of concept mining, multi-agent structured scoring, and geometric calibration, optimized through an end-to-end dimension optimization loop. Applied to urban perception on Place Pulse 2.0, UrbanAlign achieves 72.2% accuracy, surpassing supervised baselines by +15.1 pp and uncalibrated VLM scoring by +16.3 pp.
Forget fine-tuning: this training-free pipeline aligns VLMs with human preferences by calibrating concept scores, outperforming supervised methods in urban perception tasks.
Aligning vision-language model (VLM) outputs with human preferences in domain-specific tasks typically requires fine-tuning or reinforcement learning, both of which demand labelled data and GPU compute. We show that for subjective perception tasks, this alignment can be achieved without any model training: VLMs are already strong concept extractors but poor decision calibrators, and the gap can be closed externally. We propose a training-free post-hoc concept-bottleneck pipeline consisting of three tightly coupled stages: concept mining, multi-agent structured scoring, and geometric calibration, unified by an end-to-end dimension optimization loop. Interpretable evaluation dimensions are mined from a handful of human annotations; an Observer-Debater-Judge chain extracts robust continuous concept scores from a frozen VLM; and locally-weighted ridge regression on a hybrid visual-semantic manifold calibrates these scores against human ratings. Applied to urban perception as UrbanAlign, the framework achieves 72.2% accuracy ($κ=0.45$) on Place Pulse 2.0 across six categories, outperforming the best supervised baseline by +15.1 pp and uncalibrated VLM scoring by +16.3 pp, with full dimension-level interpretability and zero model-weight modification.
