Search papers, labs, and topics across Lattice.
The paper introduces Whisperer, a visual prompting framework that uses diffusion-based preprocessors to adapt inputs in pixel space for frozen OCR models like EasyOCR. Whisperer frames the process as behavioral cloning of stochastically discovered improvement policies, training a diffusion model to reproduce "lucky" improvements in Character Error Rate (CER) found through stochastic exploration. Experiments on a degraded synthetic text dataset show an 8% absolute reduction in CER compared to baselines, demonstrating effective adaptation of a frozen model without modifying its weights.
You can now boost the performance of your frozen OCR model by 8% CER just by "whispering" pixel-space enhancements to its inputs via a diffusion-based preprocessor.
In the landscape of modern machine learning, frozen pre-trained models provide stability and efficiency but often underperform on specific tasks due to mismatched data distributions. This paper introduces the Whisperer, a novel visual prompting framework that learns diffusion-based preprocessors to adapt inputs in pixel space, effectively"whispering"enhancements to frozen downstream models like EasyOCR. By framing the process as behavioral cloning of stochastically discovered improvement policies, our method achieves an 8% absolute (10.6% relative) reduction in Character Error Rate (CER) on a challenging dataset of 300k degraded synthetic text images, surpassing hand-engineered baselines such as CLAHE. The key innovation is a four-stage training curriculum that uses behavioral cloning to amplify"lucky"improvements discovered through the stochastic exploration of a partially trained diffusion model. This approach is highly sample-efficient and avoids the pitfalls of traditional reinforcement learning. Crucially, we frame this not as naive reinforcement learning, but as behavioral cloning of an exploration policy: we stochastically sample intermediate diffusion outputs, select those that improve CER by chance, and then train the model to reproduce them. This bootstrapping curriculum (4 stages over 60 GPU-hours) amplifies random successes into a systematic strategy. In summary, by whispering to the frozen OCR through its inputs, we improve an imperfect classifier without touching its weights.
