Search papers, labs, and topics across Lattice.
This paper introduces Tevatron 3.0, a reranker training framework that integrates a Megatron-Core backend to enhance training efficiency and quality for large-scale mixture-of-experts (MoE) models within academic constraints. By benchmarking against existing configurations, the authors demonstrate that the new backend not only matches the quality of FSDP but also improves training speed by up to 22% in single-node setups. The findings reveal that a 30B-parameter MoE reranker can achieve comparable performance to an 8B dense model while utilizing significantly fewer active parameters, thus optimizing inference throughput.
Achieving high-quality reranking with a 30B MoE model is now feasible on an academic budget, outperforming traditional dense models in efficiency.
Modern reranking recipes---billion-scale cross-encoders, mixture-of-experts (MoE) backbones, and distillation against strong teachers---have outpaced the training infrastructure available to most academic groups. Existing Tevatron reranker training relies on the Hugging Face Trainer with DeepSpeed or PyTorch FSDP1, but these backends lack efficient support for large-scale MoE training. We present Tevatron 3.0, which integrates a Megatron-Core training backend into Tevatron while preserving its data pipeline, evaluation workflow, and Hugging Face-compatible checkpoints. We benchmark existing distributed training configurations against the new backend, showing that Megatron matches FSDP reranker quality and training efficiency under comparable data-parallel settings, is up to 22% faster in the recommended single-node configuration, and supports both LoRA and full-parameter fine-tuning. Crucially, expert parallelism enables training a 30B-parameter Qwen3-30B-A3B MoE reranker, which is infeasible with PyTorch FSDP1. Using this framework, we conduct a controlled comparison of MoE versus dense models, LoRA versus full-parameter tuning, and distillation versus contrastive training on BEIR-15 with three first-stage retrievers, and report serving throughput for Hugging Face and vLLM. We find that the MoE reranker matches dense 8B quality while activating less than half as many parameters and achieving substantially higher inference throughput. We will release the framework and trained checkpoints.
CMU ML