Search papers, labs, and topics across Lattice.
This study explores activation steering as a novel approach for generating synthetic data in low-resource languages, contrasting it with traditional few-shot prompting methods that can limit diversity and increase costs. The authors implement two steering strategies鈥擫anguage Steering and Quality Steering鈥攁cross multiple LLMs and languages, demonstrating that steering on early layers enhances data diversity and downstream task performance. Key findings reveal that activation steering not only improves the quality of generated data but also significantly benefits the performance of classifiers trained on this data, particularly in low-resource contexts.
Steering activation in LLMs can enhance synthetic data diversity and boost classifier performance, especially for low-resource languages.
Large language models (LLMs) have become an effective tool for synthetic data generation, including for low-resource languages, where generated data can improve downstream task performance. Current best-performing approaches typically rely on few-shot prompting with target-language examples, which increases inference costs and may reduce diversity through lexical anchoring. In this work, we investigate activation steering as an alternative for low-resource synthetic data generation. We study two steering strategies: Language Steering, which targets the linguistic identity of a language, and Quality Steering, which captures well-formedness by contrasting human-written and backtranslated text representations. We evaluate these methods across four open-source LLMs, multiple layers, and 11 typologically diverse languages by generating sentiment and topic classification data and finetuning smaller classifiers. Steering is applied in both zero-shot and few-shot prompting settings and compared against non-steered counterparts. Our results show that steering on early layers consistently improves the diversity of generated data while often yielding stronger downstream model performance, particularly for low-resource languages.
