Search papers, labs, and topics across Lattice.
This study investigates the effects of token distribution versus data volume in the context of multi-domain meeting summarization by fine-tuning the Mistral-7B model with QLoRA on both balanced and natural token mixtures across five meeting corpora. The findings reveal that balancing token distribution significantly enhances performance in data-scarce minority domains while only slightly impacting data-rich domains, suggesting that a balanced approach is crucial when minority domains are important. Additionally, the research demonstrates that pruning low-value transcript lines can effectively reduce token count without sacrificing quality, providing a nuanced understanding of how to optimize training data for multi-domain tasks.
Balancing token distribution can dramatically improve performance in underrepresented domains, challenging the assumption that more data is always better.
Jointly fine-tuning an LLM on meeting-summarisation corpora of widely varying size raises a question that prior work leaves confounded: when a domain-balanced training mixture helps, is the gain due to the distribution of tokens across domains, or merely to the volume of data seen? We disentangle these factors by constructing balanced and natural (native-proportional) token mixtures at matched token budgets (2-32M) over five English meeting corpora, fine-tuning Mistral-7B with QLoRA, and evaluating per domain. Balancing redistributes quality, improving the data-scarce minority domains at a low cost to the data-rich ones. The trade favours balancing whenever the minority domains matter: their share under proportional allocation is fixed at 1-2% regardless of budget, so matching balanced quality on those domains requires far more total data. We further find that pruning low-value transcript lines removes ~15% of tokens from the conversational corpora at no measurable cost, and that balancing by tokens is not the same as balancing by examples. A two-annotator study of 741 judge-labelled facts validates our fact-level evaluation. Together these results give practitioners a basis for deciding when to balance an imbalanced multi-domain mixture, and on what unit.
