Search papers, labs, and topics across Lattice.
This study investigates the reliability of financial named entity recognition (NER) systems under domain shifts by evaluating various confidence estimation methods across different text sources, including SEC filings, financial news, and social media. The authors find that while whole-output probability is effective in detecting errors in-domain, it significantly deteriorates in out-of-domain contexts, with self-consistency proving to be a more robust and well-calibrated alternative. Their findings suggest that implementing a staged deployment strategy can significantly reduce error rates by leveraging confidence gating in response to distribution shifts.
Self-consistency outperforms traditional confidence measures, reducing financial NER errors from 34.3% to below 2% in high-confidence scenarios.
Financial AI systems often train information extractors on one textual register and deploy them across filings, news, and user-generated content, while standard F1 scores do not indicate which predictions remain safe to automate when the input distribution changes. We study confidence estimation and selective prediction for financial named entity recognition (NER) on a three-tier stress test spanning SEC filings, financial news, and general-topic social media as an extreme out-of-domain condition. We evaluate a BERT tagger and LoRA-tuned Qwen2.5-0.5B/1.5B models using five inference-time confidence signals, three training seeds, and bootstrap intervals. Confidence rankings themselves change under distribution shift: whole-output probability is the strongest in-domain error detector but deteriorates out of domain, whereas entity-span probability and self-consistency are more robust; self-consistency is also better calibrated without post-hoc fitting. Abstention reduces sentence error from 34.3% to below 2% on the highest-confidence 40% of in-domain inputs and remains useful on financial news, but recovers no usefully large clean subset under the extreme social-media shift. These results motivate a staged deployment strategy that detects severe distribution shift upstream before applying prediction-level confidence gating.
