Search papers, labs, and topics across Lattice.
This paper evaluates the ability of LLMs to perform in-context translation by framing it as a formal language transduction problem using synchronous context-free grammars. The authors systematically vary grammar size, sentence length, and linguistic properties to assess how well LLMs can translate between these formal languages given the grammar in-context. Results show that translation accuracy drops significantly with grammar complexity and sentence length, and is further impacted by morphological and script differences between the languages.
LLMs struggle to translate even simple formal languages in-context when faced with increasing grammar size, sentence length, and cross-linguistic differences, revealing limitations in their ability to effectively utilize provided grammatical descriptions.
Low-resource languages pose a challenge for machine translation with large language models (LLMs), which require large amounts of training data. One potential way to circumvent this data dependence is to rely on LLMs' ability to use in-context descriptions of languages, like textbooks and dictionaries. To do so, LLMs must be able to infer the link between the languages' grammatical descriptions and the sentences in question. Here we isolate this skill using a formal analogue of the task: string transduction based on a formal grammar provided in-context. We construct synchronous context-free grammars which define pairs of formal languages designed to model particular aspects of natural language grammar, morphology, and written representation. Using these grammars, we measure how well LLMs can translate sentences from one formal language into another when given both the grammar and the source-language sentence. We vary the size of the grammar, the lengths of the sentences, the syntactic and morphological properties of the languages, and their written script. We note three key findings. First, LLMs' translation accuracy decreases markedly as a function of grammar size and sentence length. Second, differences in morphology and written representation between the source and target languages can strongly diminish model performance. Third, we examine the types of errors committed by models and find they are most prone to recall the wrong words from the target language vocabulary, hallucinate new words, or leave source-language words untranslated.
