Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
This study critiques the conventional evaluation methods for continual knowledge updating by showing that performance rankings can shift dramatically based on the timing of evaluations and the capacity of replay mechanisms. Through comparative analysis over a 24-month Wikidata stream, the authors reveal that a hierarchical approach can initially outperform replay methods but can later fall short as replay capacity increases. The findings advocate for a more nuanced evaluation framework that accounts for temporal and capacity variations, suggesting that a periodic hierarchy may be more cost-effective but not necessarily superior in quality.
Performance rankings in continual knowledge updating can flip dramatically based on evaluation timing and replay capacity, challenging conventional wisdom.
Continual knowledge-updating methods are often declared superior from one final checkpoint and one conventional adapter rank. We show that this can be insufficient to identify the better operating point. Holding a periodic hierarchy fixed, we compare it with cumulative replay over a 24-month Wikidata stream while varying evaluation month, replay LoRA rank, and query formulation. The apparent winner changes across this region: on Qwen2.5-1.5B, the hierarchy's 5.0-point advantage over rank-8 replay becomes an 11.6-point deficit against rank-72 replay, and at high ranks a consolidation-aligned endpoint can suggest a tie while time-averaged replay leads by 9-13 points. The same rank-conditioned reversal appears on Llama-3.2-1B and held-out paraphrases. These results show that method ranking in continual updating can depend jointly on when performance is measured and how much replay-side adaptation capacity the baseline receives. We therefore propose reporting trajectories and capacity sweeps, and declaring a robust winner only when the ordering is stable across the evaluation region; otherwise, comparisons should report winner regions and retention-stability-cost frontiers. Under this protocol, the periodic hierarchy is a lower-update-cost operating point, not a quality winner.