Search papers, labs, and topics across Lattice.
This study evaluates the performance of open-weight language models on the updated Financial Touchstone benchmark, which includes nearly 3,000 question context-answer triplets from international annual reports. The findings reveal that Anthropic's Claude Opus 4.6 leads in accuracy at 88.4%, while the open-weight model Kimi K2.6 ranks third, challenging the notion that proprietary models are essential for financial text comprehension. Additionally, the research highlights that information retrieval is a significant failure point, and geopolitical content filters in Chinese models can restrict legitimate financial inquiries.
Open-weight models can achieve competitive performance in financial text comprehension, with Kimi K2.6 ranking impressively high against proprietary counterparts.
Open-weight language models from Chinese AI labs caught up on benchmarks relative to proprietary frontier models in recent months. Yet their reliability on real-world financial tasks remains largely untested. We updated the Financial Touchstone benchmark, which now has 2,967 question context-answer triplets across 495 international annual reports. We also apply a new set of models on the benchmark, expanding coverage from eleven to twenty models across ten providers, including recent open-weight models such as GLM 4.7, GLM 5, Kimi K2.6, and DeepSeek V3.2, as well as Alibaba's proprietary flagship Qwen3-Max. Anthropic's Claude Opus 4.6 achieves the highest accuracy (88.4%), while Google's Gemini 2.5 Pro maintains the lowest hallucination rate (0.08%). Notably, the open-weight Kimi K2.6 ranks third in accuracy, and the non-reasoning models GLM 5 and Mistral 3 rank fourth and fifth, challenging the assumption that reasoning architectures or proprietary weights are a prerequisite for strong financial comprehension. Information retrieval remains the primary bottleneck, accounting for 48.9% of all failures. We also document a new finding: geopolitical content filters in Chinese models refuse legitimate financial questions (0.08% of attempts), sometimes without clear reason, and the refusal behavior depends on the access route as much as on the model. The complete dataset and evaluation framework are publicly available.
