Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for critique-guided refinement in non-verifiable generation, where an actor receives feedback from a dynamically adapting critic. The core innovation, TAIScore, evaluates the effectiveness of critiques by assessing their relevance to the actor's weaknesses and the actor's ability to implement the feedback. Experimental results demonstrate that an 8B critic trained with TAIScore significantly outperforms both a larger zero-shot critic and traditional reward signals, highlighting the benefits of a co-evolving critic-actor system.
TAIScore reveals that effective critique in AI generation hinges on the dynamic interplay between actor capabilities and targeted feedback, outperforming larger models in practical applications.
Natural-language critiques provide supervision beyond scalar rewards for non-verifiable generation, which lacks deterministic verifiers. In critique-guided refinement, a critic gives feedback on an initial response and an actor revises it. However, final revision quality does not reveal whether the critique was actually useful: a capable actor may improve without following the feedback, while valid feedback may fail if the actor cannot execute it. We frame critique as actor-conditioned revision guidance, where usefulness depends on whether the feedback helps the target actor address the intended weakness. We introduce TAIScore (Targeted Actionable Improvement Score), a reward that evaluates the instruction, initial response, critique, and revision together, assessing whether the critique targets a real weakness, whether the actor follows it, and whether the intended aspect improves. We use this reward to train an actor-tailored critic with GRPO, and use critique-guided refinements to construct DPO preference pairs for the actor, forming a co-evolving critic-actor loop where the critic adapts to the actor's changing capability. Experiments show that an 8B critic trained with TAIScore outperforms both a zero-shot 120B critic and critics trained with outcome-only or critique-only reward signals. Co-evolving the critic and actor further improves performance, suggesting that effective critique supervision should adapt as the actor changes.
