Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluated the effectiveness of agentic large language models (LLMs) in generating treatment recommendations for breast cancer across various clinical stages using 72 real cases and 1,147 case-specific rubrics generated via Asymmetric Information Rubric Generation (AIRG). Seven different configurations were tested, with the Claude Opus 4.8 using the D&C+SA pipeline achieving the highest performance score of 0.594 ± 0.025. Despite showing potential in generating relevant recommendations, the models exhibited significant limitations, including incorrect recommendations and overconfidence, indicating they are not yet reliable for unsupervised clinical application.
Agentic LLMs can generate clinically relevant breast cancer treatment recommendations, but their reliability remains questionable due to persistent errors.
Large language models (LLMs) are increasingly being explored for clinical decision support, but their reliability in complex oncology treatment planning remains unclear. We evaluated agentic LLM systems for breast cancer treatment recommendation generation using 72 real clinical cases across stages I to IV and 1,147 case-specific rubrics generated through Asymmetric Information Rubric Generation (AIRG), in which the rubric generator had access to real clinical decisions unavailable to the evaluated models. Seven pipelines were compared, including single-LLM baselines, tool-augmented systems, and multi-agent architectures with fact checking and autonomous subagent spawning. The best-performing configuration, Claude Opus 4.8 with the D&C+SA pipeline, achieved a global score of 0.594 $\pm$ 0.025. Tool use and increased agent autonomy had mixed effects, improving performance in some settings but degrading it in others. Performance varied by clinical domain and disease stage, and oncologist-led error analysis revealed persistent clinically relevant failures, including incorrect or missing recommendations, flawed justifications, citation errors, outdated claims, and overconfidence. These findings suggest that agentic LLM systems can generate clinically relevant breast cancer recommendations, but remain insufficient for unsupervised clinical use.