Data Curation & Synthetic Data - Weekly Roundup

Tailoring public dataset mixtures for DP pre-training via private linear regression unlocks substantial utility gains, boosting X-ray classification AUC by up to 0.037 and reducing language model perplexity by 16%.

A transformer pretrained on synthetic, never-repeating "languages" learns to infer and predict real-world text and perform complex reasoning tasks, suggesting a fundamental mechanism for language acquisition.

Humanoid robots can now learn complex loco-manipulation tasks from sparse demonstrations by iteratively evolving their own training data and policies.

LLM agents can now dynamically curate multimodal memory on-demand, achieving optimal performance-cost-latency trade-offs by intelligently choosing between pre-computed and runtime-generated evidence.

A single graph foundation model learns to wander across diverse graph structures and tasks, achieving SOTA on node classification, homogeneous link prediction, and KG link prediction.

Tailoring public dataset mixtures for DP pre-training via private linear regression unlocks substantial utility gains, boosting X-ray classification AUC by up to 0.037 and reducing language model perplexity by 16%.

A transformer pretrained on synthetic, never-repeating "languages" learns to infer and predict real-world text and perform complex reasoning tasks, suggesting a fundamental mechanism for language acquisition.

Humanoid robots can now learn complex loco-manipulation tasks from sparse demonstrations by iteratively evolving their own training data and policies.

LLM agents can now dynamically curate multimodal memory on-demand, achieving optimal performance-cost-latency trade-offs by intelligently choosing between pre-computed and runtime-generated evidence.

A single graph foundation model learns to wander across diverse graph structures and tasks, achieving SOTA on node classification, homogeneous link prediction, and KG link prediction.

A single pre-trained transformer now solves matrix completion across causal inference, recommendation, and tabular imputation tasks by treating matrices natively.

Confident conflicts in LLM distillation are mitigated by dynamically mixing student and teacher distributions based on student entropy, boosting reasoning and preserving general capabilities.

Clinical ML models are only as good as their filtered data, yet current practices obscure crucial scientific decisions that shape task complexity and performance.

The optimal LoRA rank is not intrinsic to the model but dynamically dictated by the data's geometric properties, enabling principled data selection for efficient fine-tuning.

Multi-omics cancer analysis becomes practical even with missing data, achieving state-of-the-art reconstruction and classification via a novel representation learning approach.

Most reported gains in crop disease classification from environmental sensors are illusory, driven by simple session memorization rather than genuine multimodal learning.

A novel bandit algorithm achieves near-optimal dynamic minimax regret for LLM post-training, improving worst-group robustness without re-evaluating historical data or altering standard training pipelines.

A dual VAE architecture effectively bridges the sim-to-real gap for robotic navigation, enabling high performance with minimal real-world data.

Reasoning in base LLMs can be unlocked by simply prepending specific tokens, matching RL-tuned performance on math and coding.

A novel meta-network framework for data selection achieves unprecedented transferability across diverse LLM training tasks by overcoming the limitations of existing MTS objectives.

A handful of verified poisoned examples, not just clean ones, dramatically boosts data poisoning defense against clean-label attacks.

Relational learning with tabular foundation models can now bypass feature explosion and interaction blindness by propagating information directly through the schema graph.

Expert pairwise rankings, not just absolute scores, can efficiently guide scientific discovery, even when experts have varying fidelity and cost.

Generating high-quality, diverse datasets from scratch is now possible, outperforming frontier models by 17% in quality and 19% in diversity.

LLM agents can now build lasting, reusable "competence" for specific knowledge sources, moving beyond simple retrieval to deep, adaptive understanding.

Off-policy data, long thought to be incompatible with effective SFT, can be transformed via MCMC sampling to unlock generalization capabilities that rival or exceed on-policy RL.

Synthetic pre-pretraining boosts LLM efficiency at scale, but not by teaching grammar; it's the retrieval skills that matter.

StreamMAE unlocks self-supervised learning on continuous video streams, matching i.i.d. MAE performance by tackling intra-batch similarity with novel regularization.

Forecast errors, not drift alarms, directly signal when to retrain time series models, achieving state-of-the-art performance with a plug-in module.

LLM agents can improve their decision-making by learning from their own diagnosed failures, with first-proposal corrections boosting success rates by over 30 percentage points.

Wild AI-generated text can actively harm LLM performance on human text, with its value flipping from beneficial to detrimental as token ratios increase.

LLM agents trained solely in rule-generated fictional worlds achieve state-of-the-art transfer to real-world search tasks, outperforming agents trained on actual data.

Language models can learn new knowledge without forgetting old by dynamically prioritizing replay based on observed forgetting.

Human password generation exhibits persistent linguistic regularities that can be exploited for a 16% boost in password recovery by adapting pre-trained language models.

State-of-the-art precipitation downscaling models struggle to generalize across continents, revealing a critical gap in current ML-based climate forecasting.

Finite-sample performance guarantees for label-noise transition matrix estimation are now achievable by bypassing class-posterior estimation through one-sided selective classification.

Zero-shot in-context inference trained strictly on synthetic causal graphs can outperform fully tuned AutoML pipelines across diverse real-world tabular benchmarks.

Non-generative "System One" models can beat closed-source baselines in Chinese decision tasks with a 20x speedup and 15 ms latency, proving encoder-only architectures remain radically underutilized for structured reasoning.

Mixing auxiliary data doesn't just offer more tokens—under the right spectral alignment and dataset growth ratio, it provably accelerates the downstream power-law scaling exponent itself.

Collapsing pre-, mid-, and post-training into a single fully synthetic run matches open-weight model performance while slashing required training tokens by up to 140×.

Training data attribution on 32B-parameter models no longer requires intractable gradient storage: extreme 1-bit eigenbasis projection achieves competitive influence estimation accuracy at over 100× lower per-example footprint than standard baselines.

Cross-lingual data pooling acts as a powerful low-resource jumpstart—boosting macro-F1 by up to 0.43 at 25 labels—but predictably degenerates into negative transfer once local annotations surpass 800 examples.

Standard refusal detectors miss up to 99.6% of noncompliant instruction-tuning data, but multi-pass consensus from a 1.6T-parameter judge successfully isolates millions of latent refusals and evasions at 86.7% precision.

Standard next-token adaptation fails on financial time series unless fine-tuning objectives explicitly penalize rollouts for distorting higher-order temporal structures like volatility clustering.

Synthesizing historical street views is now possible with 17% better realism and consistency by treating it as a reward-guided editing problem.

Collecting more errors doesn't guarantee better models: targeting only the high-impact failure modes signaled by a network's own internal representations outperforms standard rarity-based active learning while saving annotation budget.

Static problem pools cause LLM reasoning post-training to stall as models saturate the dataset, but dynamically synthesizing frontier tasks via regret-guided procedural generation keeps learning signals saturated for sustained gains.

Even when provided with only target column metadata, LLMs outperform specialized AutoML heuristics like AutoGluon at inferring machine learning task types (0.98 vs. 0.93 macro F1), unlocking fully autonomous pipeline configuration.

Contemporary frontier speech models still catastrophically fail on living Arabic dialects, with top systems like GPT-4o achieving an error rate no better than 35.0% WER across 17 regional vernaculars.

Jev’s apparent miscalibration on binary choices is an artifact of forced binarization: recovering a hidden, unexpressed "I don't know" probability mass catapults the model's calibration accuracy from 0.771 to 0.978.

Verifying LLM span extractions no longer requires full manual re-annotation: character-level policy reconciliation isolates genuine disagreements, slashing human review to just 5.4 focused conflicts per document.

Industrial anomaly detection usually breaks down where data is sparsest, but baking reaction kinetics directly into conditional diffusion enables high-fidelity trajectory synthesis of catastrophic failure states far outside the training distribution.

Model collapse in generative self-training can be weaponized: intentionally amplifying a generator's Jacobian-derived geometric distortions creates a far more potent negative guidance signal than standard synthetic outputs.

High-fidelity 3D texture generation is now achievable without any 3D assets, by cleverly transforming 2D images into effective training data.