Massimiliano Mancini

Textual refusal directions can be harnessed to enhance multimodal safety without the need for unsafe multimodal data, revealing a powerful alignment strategy.

Data mixing, especially with instruction-heavy data, emerges as the crucial factor for optimizing VLM training, challenging traditional filtering approaches.

Unlocking fairer vision-language models may be as simple as intervening in the sparse latent space of a sparse autoencoder, enabling targeted bias removal without harming performance.

Forget specialized classifiers—LMMs, enhanced with in-context learning and a novel iterative refinement method (CIRCLE), can outperform even fine-tuned VLMs in both closed and open-world classification.