Search papers, labs, and topics across Lattice.
Affiliation:
1
0
3
0
Tokenization disparities can inflate costs and reduce effective context length for underserved languages, with Bengali and Yoruba facing tokenization premiums of up to 4.5 times that of English.
