Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of Large Language Models (LLMs) in automating the summarization of financial news, addressing the challenge faced by analysts and investors in processing vast amounts of information. By integrating news articles, company backgrounds, and stock data, the researchers developed a pipeline that utilizes Summarize Chains and Retrieval-Augmented Generation (RAG) methods across various models. The findings reveal that Falcon-7B with Summarize Chains outperformed other approaches, accurately capturing all relevant news events while highlighting the limitations of RAG, particularly in terms of hallucination and repetition.
Falcon-7B with Summarize Chains not only automates financial news summarization but does so with remarkable accuracy, outperforming traditional methods and exposing critical weaknesses in RAG techniques.
Stock market analysts and investors face a daily challenge: too much financial news, too little time. Manually reading and synthesizing hundreds of company-specific articles is impractical, yet missing key information can directly affect investment decisions. This project, conducted at George Washington University in Fall 2023, explores whether Large Language Models can automate this process reliably. We built a pipeline that pulls news articles from the News API, company background from Wikipedia, and stock price data from Yahoo Finance for ten major companies (AAPL, MSFT, GOOGL, AMZN, META, TSLA, JPM, NVDA, WMT, DIS). Because LLMs cannot directly process numerical tables, we developed a simple but effective template that converts stock data into natural language narratives. We then tested two summarization approaches (Summarize Chains and Retrieval-Augmented Generation with FAISS) across three open-source models (Falcon-7B-Instruct, DistilBART-CNN-12-6, BART-Large-XSum) for news, and GPT (text-davinci-003) for stock summaries. Falcon-7B with Summarize Chains gave the best results, covering all news events accurately and coherently. RAG, while promising in theory, caused severe repetition in Falcon and hallucinated facts in BART-Large when k was large. Both LLM-based approaches outperformed a simple Lead-3 baseline on ROUGE-1. We also built a Streamlit dashboard for interactive stock visualization. The work was done in Fall 2023, before RAG-based financial tools became widespread, and the failure modes we document, particularly hallucination under RAG in smaller models, remain relevant today.
