Search papers, labs, and topics across Lattice.
This paper introduces UniLang, a unified generative framework that integrates machine-native symbols into pretrained large language models (LLMs), allowing these models to treat both symbolic and natural language tokens as first-class generative units. By expanding the LLM's vocabulary and embedding space, UniLang enables a single autoregressive objective to model and generate both types of tokens without the need for verbalization or task-specific architectures. The framework is evaluated on sequential recommendation and legal precedent prediction tasks, where it consistently outperforms strong baselines, highlighting its effectiveness in bridging the gap between language modeling and structured prediction.
UniLang enables pretrained LLMs to seamlessly integrate machine-native symbols, outperforming traditional models in diverse structured prediction tasks.
Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language. While these representations are compact and preserve task-relevant structure, they lie outside the linguistic token space of pretrained large language models (LLMs), creating a fundamental divide between language modeling and structured prediction. We introduce UniLang, a unified generative framework that bridges this divide by extending pretrained LLMs to treat machine-native symbols as first-class generative units alongside natural-language tokens. UniLang expands the LLM's vocabulary and embedding space with grounded machine-native representations, enabling textual and symbolic tokens to be jointly modeled and generated under a single autoregressive objective. This unified interface allows pretrained LLMs to directly operate on machine-native representations without requiring them to be verbalized as natural language or relying on task-specific architectures. We evaluate UniLang on two structurally distinct tasks, sequential recommendation and legal precedent prediction, spanning different domains and types of structured prediction. Across both tasks, UniLang consistently outperforms strong baselines, demonstrating a path toward extending pretrained LLMs beyond language and using them as a common generative modeling backbone for heterogeneous machine-native representations.
