Search papers, labs, and topics across Lattice.
The paper introduces Selective Chain-of-Thought (Selective CoT), an inference-time strategy for medical question answering that predicts whether a question requires explicit reasoning before generating a rationale. This approach aims to improve efficiency by avoiding unnecessary reasoning steps for simpler questions while maintaining accuracy on complex ones. Experiments on biomedical QA benchmarks using Llama-3.1-8B and Qwen-2.5-7B showed Selective CoT reduced inference time by 13-45% and token usage by 8-47% with minimal accuracy loss, and in some cases, improved accuracy.
Why waste tokens on reasoning when you don't need to? Selective Chain-of-Thought cuts LLM inference costs by up to 47% in medical QA, with minimal accuracy loss.
Objective: To improve the efficiency of medical question answering (MedQA) with large language models (LLMs) by avoiding unnecessary reasoning while maintaining accuracy. Methods: We propose Selective Chain-of-Thought (Selective CoT), an inference-time strategy that first predicts whether a question requires reasoning and generates a rationale only when needed. Two open-source LLMs (Llama-3.1-8B and Qwen-2.5-7B) were evaluated on four biomedical QA benchmarks-HeadQA, MedQA-USMLE, MedMCQA, and PubMedQA. Metrics included accuracy, total generated tokens, and inference time. Results: Selective CoT reduced inference time by 13-45% and token usage by 8-47% with minimal accuracy loss ($\leq$4\%). In some model-task pairs, it achieved both higher accuracy and greater efficiency than standard CoT. Compared with fixed-length CoT, Selective CoT reached similar or superior accuracy at substantially lower computational cost. Discussion: Selective CoT dynamically balances reasoning depth and efficiency by invoking explicit reasoning only when beneficial, reducing redundancy on recall-type questions while preserving interpretability. Conclusion: Selective CoT provides a simple, model-agnostic, and cost-effective approach for medical QA, aligning reasoning effort with question complexity to enhance real-world deployability of LLM-based clinical systems.
