Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of cluster-based semantic chunking in Retrieval-Augmented Generation (RAG) systems when applied to long academic texts. The research finds that while cluster-based chunking was hypothesized to enhance retrieval and answer quality, it did not outperform simpler fixed-size and recursive chunking methods. Additionally, the reliability of the RAGAs framework for assessing faithfulness was found to be limited, indicating that document formatting and preprocessing significantly influence performance on different types of questions.
Cluster-based chunking fails to deliver on its promise, showing no performance advantage over simpler methods in RAG systems for academic texts.
Retrieval-Augmented Generation (RAG) systems use the question-answering capabilities of Large Language Models (LLMs) to access information outside their parameters. We evaluate if cluster-based semantic chunking improves retrieval and answer quality compared to fixed-size and recursive chunking evaluating on long, structured academic theses using the Retrieval Augmented Generation Assessment (RAGAs) framework. RAGAs based faithfulness shows limited reliability in this setup. Performance on fixed versus document specific questions varied substantially, likely related to the formatting of documents and preprocessing. Under the tested configuration, cluster-based chunking did not outperform simpler strategies.
