Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study benchmarked the consistency of exercise prescriptions generated by GPT-4.1, Claude Sonnet 4.6, and Gemini 2.5 Flash across repeated generations under identical temperature settings. The analysis revealed significant inter-model differences in semantic similarity and output reproducibility, with GPT-4.1 generating unique but semantically similar outputs, while Gemini 2.5 Flash exhibited high semantic similarity due to text duplication. These findings highlight the importance of evaluating LLM output behavior under repeated generation conditions for reliable deployment in clinical settings.
Identical decoding settings can yield fundamentally different consistency profiles across LLMs, meaning that single-output evaluations can't be trusted for clinical applications.
This study compared repeated generation consistency of exercise prescription outputs across three large language models (LLMs), specifically GPT-4.1, Claude Sonnet 4.6, and Gemini 2.5 Flash, under temperature=0 conditions. Each model generated prescriptions for six clinical scenarios 20 times, yielding 360 total outputs analyzed across four dimensions: semantic similarity, output reproducibility, FITT classification, and safety expression. Mean semantic similarity was highest for GPT-4.1 (0.955), followed by Gemini 2.5 Flash (0.950) and Claude Sonnet 4.6 (0.903), with significant inter-model differences confirmed (H = 458.41, p<.001). Critically, these scores reflected fundamentally different generative behaviors: GPT-4.1 produced entirely unique outputs (100%) with stable semantic content, while Gemini 2.5 Flash showed pronounced output repetition (27.5% unique outputs), indicating that its high similarity score derived from text duplication rather than consistent reasoning. Identical decoding settings thus yielded fundamentally different consistency profiles, a distinction that single-output evaluations cannot capture. Safety expression reached ceiling levels across all models, confirming its limited utility as a differentiating metric. These results indicate that model selection constitutes a clinical rather than merely technical decision, and that output behavior under repeated generation conditions should be treated as a core criterion for reliable deployment of LLM-based exercise prescription systems.