Search papers, labs, and topics across Lattice.
The paper introduces Bielik v3, a 7B and 11B parameter series of Polish language models, optimized through a transition from a universal Mistral-based tokenizer to a dedicated Polish-optimized vocabulary. This optimization leverages FOCUS-based embedding initialization and a multi-stage pretraining curriculum, followed by post-training alignment using SFT, DPO, and GRPO with verifiable rewards. Results demonstrate improved tokenization efficiency, reducing inference costs and expanding the effective context window for Polish language processing.
Polish language models get a major efficiency boost: Bielik v3 slashes tokenization overhead with a custom vocabulary, outperforming generic tokenizers.
The development of the Bielik v3 PL series, encompassing both the 7B and 11B parameter variants, represents a significant milestone in the field of language-specific large language model (LLM) optimization. While general-purpose models often demonstrate impressive multilingual capabilities, they frequently suffer from a fundamental architectural inefficiency: the use of universal tokenizers. These tokenizers, typically designed to cover a broad spectrum of languages, often fail to capture the morphological nuances of specific languages like Polish, leading to higher fertility ratios, increased inference costs, and restricted effective context windows. This report details the transition from the universal Mistral-based tokenization to a dedicated Polish-optimized vocabulary for the Bielik v3 models, exploring the FOCUS-based embedding initialization, the multi-stage pretraining curriculum, and the subsequent post-training alignment involving Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning through Group Relative Policy Optimization with verifiable rewards.
