Search papers, labs, and topics across Lattice.
FlowBalance integrates dense, trajectory-level self-guidance from privileged-context rollouts directly into GFlowNet trajectory balance, calibrating the guidance score via verifier-derived group advantages to dynamically reinforce, invert, or disable self-supervision. This addresses the fundamental trade-off in reasoning self-improvement between uninformative sparse terminal rewards and hallucination-prone dense internal guidance without requiring auxiliary token-level imitation losses. Applied to mathematical reasoning with Qwen3-4B and Qwen3-8B, the method outperforms FlowRL in accuracy and sample efficiency while preventing the response-length collapse seen in on-policy distillation and preserving diverse reasoning strategies on AIME24.
Inverting dense self-guidance whenever a verifier rejects a trajectory prevents reasoning models from reinforcing their own hallucinations, enabling stable, energy-based self-improvement without mode collapse.
A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level self-guidance score. FlowBalance calibrates this score with the verifier-derived group advantage: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when the rollout group provides no outcome preference. The resulting energy exponentially reweights a reference policy, and profiled trajectory balance fits the normalized target with one log-partition estimate per rollout group. This realizes outcome-calibrated self-guidance via trajectory balance, without a separate token-level imitation loss. Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses. On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3-4B and Qwen3-8B, while also improving training speed and stability, avoiding direct OPSD's response-length collapse, and exhibiting higher correct-strategy diversity in a controlled AIME24 diagnostic.
