Search papers, labs, and topics across Lattice.
Aalto University, ELLIS Institute
2
0
3
Benign-looking visual fine-tuning can silently trigger system-wide alignment failures, causing VLMs to hallucinate facts, execute risky agentic actions, and drop jailbreak defenses across completely unrelated tasks.
Chain-of-thought reasoning in multimodal LLMs falls apart when faced with slightly different scenarios, but surprisingly, mixing text formats in the reasoning process helps them generalize better than using images.
