Search papers, labs, and topics across Lattice.
CodeAlchemy is a synthetic data generation framework that enhances the training of language models on code by transforming publicly sourced code into semantically-rich data through five innovative strategies. By processing three corpora across 15 programming languages, it generates over 500 billion tokens of synthetic data and 350 billion reasoning tokens, significantly surpassing previous efforts in scale and quality. The framework reveals that even advanced models like Claude Sonnet 4.5 struggle with semantic understanding, achieving only 5.6% exact match on the newly introduced TraceEval benchmark, while smaller models outperform larger ones in specific tasks.
Even the most advanced models fall short, with only 5.6% accuracy on new benchmarks, highlighting a critical gap in semantic understanding in code generation.
Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats. While synthetic data has proven transformative for language models, code remains largely unexplored beyond limited quality improvements. We present CodeAlchemy, a synthetic data generation framework that transforms publicly sourced code into semantically-rich training data through 5 strategies: CodeEnhance (quality-aware rewriting), CodeQA (template-based problems), CodeDev (developer tasks), CodeDialogue (multi-turn conversations), and CodeTrace (execution traces). We process 3 corpora across 15 languages to generate 500B+ tokens of synthetic data plus 350B reasoning tokens, orders of magnitude more than prior efforts. CodeTrace instruments and executes 1.3M+ files across 14 languages and 5K libraries, capturing control flow, state tracking, and library knowledge. We introduce DevEval (developer tasks) and TraceEval (execution prediction) benchmarks; frontier models like Claude Sonnet 4.5 achieve only 5.6% exact match on TraceEval, revealing critical gaps in semantic understanding. Our 3B models achieve 83.5% on HumanEval, 63.2% on MBPP, 8.09% win rate on DevEval, and 15.36 ROUGE-2 on TraceEval, outperforming frontier models 10x the size including 27B Gemma-3 and 32B Granite-4.0.
MIT CSAIL
NVIDIA