Search papers, labs, and topics across Lattice.
This paper introduces MADA-RL, a novel post-training framework that enhances compact language models by employing a debate-aware learning signal to fine-tune a limited number of parameters through LoRA adapters. By redefining the critic's advantage using a counterfactual approach, MADA-RL optimizes the critic's performance to exceed generator consensus, resulting in more effective credit assignment. The method achieves a statistically significant accuracy improvement of 2.0 percentage points on mathematical reasoning tasks, while utilizing 16 times fewer trainable parameters compared to fully fine-tuned models, positioning it favorably on the accuracy-parameter trade-off curve.
MADA-RL boosts compact model reasoning accuracy by 2% with 16 times fewer trainable parameters, redefining how critics learn from generators.
Large language models achieve strong reasoning performance, but often at prohibitive training cost - a challenge that is especially acute for compact models ($\leq 4 \, \mathrm{B}$ parameters) trained under limited budgets. We introduce MADA-RL, a post-training framework that specializes compact models into generator and critic roles and trains them with a debate-aware learning signal, fine-tuning only a small subset of parameters via LoRA adapters. Our central contribution is a counterfactual critic advantage: a dynamic, role-conditioned baseline that redefines the critic's advantage as its reward minus the generator ensemble's per-instance accuracy. This explicitly optimizes critics to improve over generator consensus rather than to merely reproduce a correct answer, yielding more targeted credit assignment than static mean-reward normalization. At deployment, the specialized agents are composed in a lightweight multi-round protocol. Across five mathematical reasoning benchmarks, MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from $39.9 \, \%$ to $41.9 \, \%$ ($+2.0$ points, $p<0.001$) using $16$ times fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy-trainable-parameter Pareto front. It approaches, but does not surpass, the strongest baselines (DeepScaleR, STILL-3), which are trained on substantially larger datasets; we analyse this gap and the associated inference-time cost directly. A controlled study isolates the source of MADA-RL's gains: the counterfactual advantage produces the highest critic improvement rate of any model evaluated, indicating that trained critics learn to correct generator errors rather than to imitate them.
