Clara Meister

Data imbalance in MoE models is overcome by biasing expert selection, unlocking capacity for underrepresented categories without sacrificing cross-category sharing.

Two new tokenisation algorithms are introduced by introducing two new tokenisation algorithms that complete this 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokeniser, and TopDownComp, a top-down compression-based tokeniser.

Tokenizer design choices can drastically affect model performance, with intrinsic properties predicting abilities in language modeling and task accuracy more effectively than traditional metrics.