Sunayana Sitaram

There is an urgent need to develop a standardized evaluation infrastructure to assess whether ambient clinical scribes systems are safe, reliable, and well-suited to the Indian healthcare setting, and the development of a publicly shared, real-world, multilingual benchmark for ACS evaluation is called for.

Action policies in multilingual tool-using agents are more consistent than previously thought, but only for models above a certain size threshold.

VLMs are prone to critical failures that vary significantly across cultures, exposing the inadequacy of Western-centric safety benchmarks.