Search papers, labs, and topics across Lattice.
This study evaluates the impact of different types of natural language explanations (NLEs) on in-context learning (ICL) across six benchmarks and four instruction-tuned models. The findings reveal that externally generated LLM-NLEs often outperform human-written rationales and self-generated explanations, particularly in classification tasks, while the effectiveness of self-NLEs is highly dependent on the selection strategy used. Additionally, the research highlights that faithfulness-based filtering can yield variable performance outcomes, underscoring the complexity of selecting effective explanations for model prompting.
Externally generated explanations can rival human-written ones in boosting model accuracy, but the selection strategy for self-generated explanations can make or break their utility.
Natural language explanations (NLEs) are increasingly used as inputs, for example, as few-shot rationales that influence model behavior in in-context learning (ICL). However, it remains unclear how different types of NLEs compare in their effects on downstream model performance in explanation-augmented prompting. Therefore, we provide a comparative evaluation across six benchmarks and four instruction-tuned models, studying how NLE source (human-written when available, self-generated explanations, generated by an external LLM) and NLE selection (random vs faithfulness-based filtering) affect downstream utility of NLEs when used in ICL settings. Our extensive evaluation shows that, on classification-style benchmarks, adding NLEs to few-shot prompts often improves accuracy over few-shot prompting without explanations; among NLE sources, externally generated LLM-NLEs often provide strong downstream utility and remain competitive with human rationales where both are available, whereas self-NLEs are more sensitive to the selection strategy. On math reasoning, the effects are more model- and source-dependent. We further show that faithfulness-based selection of self-NLEs yields small average gains overall, but can improve or reduce performance depending on the metric, task, and model. Different faithfulness metrics can disagree substantially, affecting which self-NLE examples are selected and their downstream predictive utility. Robustness tests with randomly swapped and out-of-distribution rationales indicate partial robustness, suggesting that semantic alignment contributes to performance gains. Overall, our results provide insights for selecting and reporting explanations that influence model behavior in practical prompting pipelines.
