Search papers, labs, and topics across Lattice.
This paper introduces CreditCardQA, a novel benchmark for assessing numerical reasoning in financial literacy based on real credit card agreements, comprising 1,800 consumer-centric questions. The evaluation of various large language and reasoning models using Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting reveals that PoT significantly enhances performance, particularly for models with weaker baseline reasoning, while also reducing disparities between open- and closed-source systems. Error analysis indicates that the primary challenges stem from misapplied financial rules and misunderstandings of contractual terms rather than arithmetic errors, highlighting critical areas for improvement in model training and application.
PoT prompting not only boosts model performance on financial reasoning tasks but also bridges the gap between open- and closed-source systems.
We introduce CreditCardQA, the first financial literacy benchmark for numerical reasoning derived from real credit card agreements. The dataset contains 1,800 questions, including first-person variants that reflect how consumers naturally ask about fees, interest, and payments. We evaluate a range of large language and reasoning models under Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting. Overall, PoT yields consistent performance gains, particularly for models with weaker baseline reasoning, and narrows gaps between open- and closed-source systems. Through error analysis, we show that failures arise less from arithmetic and more from misapplied financial rules, missed conditions, and misunderstandings of contractual terms. We further analyze question difficulty and find that comparisons, conditional logic, and monetary constraints are especially challenging. We also find that errors often arise in edge cases such as late-payment penalties or small-balance scenarios that are more likely to affect lower-income or financially vulnerable individuals.
