Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores a novel approach to solving visual CAPTCHAs using a combination of a fine-tuned YOLOv8 detector and a vision-language model (VLM) that learns from each encounter. The system achieves an impressive 85.4% overall accuracy by allowing the detector to adapt and improve its performance on categories it was not initially trained on, leveraging the VLM's outputs as training labels. In a simulated arms race against CAPTCHA perturbations, the adaptive solver consistently recovers its accuracy, highlighting the limitations of static defenses against evolving AI capabilities.
Adaptive solvers can learn from every CAPTCHA encounter, recovering from adversarial perturbations faster than traditional systems can adapt.
Vision-language models (VLMs) can solve visual CAPTCHAs without task-specific training, but the agents built on them approach every challenge from scratch. For such an agent, the hundredth instance of a familiar puzzle costs as much time and compute as the first. Specialized detectors invert the trade-off, answering in milliseconds but only for categories they were trained on. Neither improves with exposure. We study what changes when a solver improves with use. Our system pairs a fine-tuned YOLOv8 detector with an open-weight VLM behind a confidence-based router, and runs entirely from screenshots and operating-system input events, with no browser automation or DOM access. It reaches 85.4% overall and 84.2% macro accuracy across 16 classes, exceeding either component alone. Every answer VLM produces also serves as a training label, so the detector absorbs categories it was never trained for, typically after one or two encounters and without human annotation. The same loop also repairs it. A CAPTCHA operator can perturb images against the publicly released detector and drive its accuracy to 0%, but the perturbations leave VLM untouched, and its labels let the detector recover. Under a year-long simulated arms race in which the CAPTCHA operator re-crafts its perturbations each month, the solver recovers every round, and a cheap ~70%-accurate open-weight teacher hardens it as effectively as a perfect oracle. Visual CAPTCHA defenses that assume a failing bot stays failing therefore understate how quickly an adaptive solver returns.