Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of fine-tuning versus hybrid retrieval methods for accurately citing the Ontario Residential Tenancies Act, using a four-arm comparison of the Qwen2.5-7B-Instruct model. The findings reveal that while fine-tuning alone leads to significant misrecalls, the hybrid approach combining supervised fine-tuning (SFT) and retrieval-augmented generation (RAG) achieves the highest exact-match score of 0.481 with no hallucinations. Notably, this hybrid method outperforms larger, specialized retrieval models without requiring additional training data, highlighting its efficiency in legal citation tasks.
Hybrid retrieval combined with fine-tuning achieves a citation accuracy of 48.1% without hallucinations, outperforming larger models and complex pipelines.
Self-represented tenants, landlords, and help-desk staff need to be pointed at the provision of law that actually governs a question, with a correct statutory citation. We study this task on the Ontario Residential Tenancies Act, 2006 (RTA) and its core regulation, asking the operator's question empirically: is fine-tuning enough, or is hybrid retrieval needed? We run a four-arm head-to-head on Qwen2.5-7B-Instruct (base zero-shot, LoRA SFT-only, RAG-only, and an SFT+RAG hybrid), scored on citation exact-match (section+subsection) over a small, human-verification-pending real eval set. The base model cannot cite the RTA and SFT-only mis-recalls sections; retrieval is essential and drives hallucination to zero by construction; and the SFT+RAG hybrid scores highest at 0.481 exact-match with zero hallucinated citations. Its edge comes from SFT making provision selection more robust to the higher-recall candidate sets that hurt zero-shot RAG. Notably, this cheap bge-small hybrid matches or beats a pipeline built on bigger, specialized retrieval models (a larger embedder and a cross-encoder reranker), and a larger/improved training set does not help either: strong statutory-citation performance here does not require specialized retrieval models or more data. The artifact zeroes hallucination and clears the lift-over-base bar but does not reach the aspirational 0.70 exact-match target. All results are on a small, human-verification-pending real eval set and are reported as preliminary.
