Data Curation & Synthetic Data - Weekly Roundup

Generative models can now produce inorganic crystals with oxidation-state distributions mirroring real-world synthesis, significantly boosting discovery potential.

A simulation flywheel recursively bootstraps embodied data, generating richer scenes and tasks for robot pretraining and evaluation.

Inserted objects can now be made "alive" and interact realistically within videos, solving a long-standing challenge in video editing.

Deterministic dermoscopic preprocessing actually degrades performance in leakage-controlled evaluations, suggesting current automated analysis might be over-engineering for minimal gains.

Transcriptomics-specific pretraining, not model scale, unlocks state-of-the-art clinical outcome prediction from bulk RNA-seq, even for survival tasks, with a 4.2M parameter model.

Generative models can now produce inorganic crystals with oxidation-state distributions mirroring real-world synthesis, significantly boosting discovery potential.

A simulation flywheel recursively bootstraps embodied data, generating richer scenes and tasks for robot pretraining and evaluation.

Inserted objects can now be made "alive" and interact realistically within videos, solving a long-standing challenge in video editing.

Deterministic dermoscopic preprocessing actually degrades performance in leakage-controlled evaluations, suggesting current automated analysis might be over-engineering for minimal gains.

Transcriptomics-specific pretraining, not model scale, unlocks state-of-the-art clinical outcome prediction from bulk RNA-seq, even for survival tasks, with a 4.2M parameter model.

Satellite images are not $n^2$ independent samples; their effective sample size plummets to $\Theta(n^2/r^2)$ due to spatial correlation, fundamentally altering generalization analysis.

Time-series foundation models, despite their promise for "what-if" analysis, exhibit attenuated dynamics and memorylessness, requiring synthetic fine-tuning to recover accurate causal inference.

Distributed MMD with Random Fourier Features achieves near-perfect correlation with expected drift magnitude on massive e-commerce data, while per-dimension KS tests falter.

LLMs can now generate novel research ideas by explicitly learning to synthesize literature through structured "anchor" specifications, outperforming prior prompting and feedback methods.

Discovering molecules with rare properties can now be an order of magnitude faster by testing complex mixtures and deconvoluting activity, slashing experimental costs from $\mathcal{O}(d)$ to $\mathcal{O}(\log d)$ or $\mathcal{O}(1)$.

Tailoring public dataset mixtures for DP pre-training via private linear regression unlocks substantial utility gains, boosting X-ray classification AUC by up to 0.037 and reducing language model perplexity by 16%.

A transformer pretrained on synthetic, never-repeating "languages" learns to infer and predict real-world text and perform complex reasoning tasks, suggesting a fundamental mechanism for language acquisition.

Humanoid robots can now learn complex loco-manipulation tasks from sparse demonstrations by iteratively evolving their own training data and policies.

LLM agents can now dynamically curate multimodal memory on-demand, achieving optimal performance-cost-latency trade-offs by intelligently choosing between pre-computed and runtime-generated evidence.

A single graph foundation model learns to wander across diverse graph structures and tasks, achieving SOTA on node classification, homogeneous link prediction, and KG link prediction.

A single pre-trained transformer now solves matrix completion across causal inference, recommendation, and tabular imputation tasks by treating matrices natively.

Confident conflicts in LLM distillation are mitigated by dynamically mixing student and teacher distributions based on student entropy, boosting reasoning and preserving general capabilities.

Clinical ML models are only as good as their filtered data, yet current practices obscure crucial scientific decisions that shape task complexity and performance.

The optimal LoRA rank is not intrinsic to the model but dynamically dictated by the data's geometric properties, enabling principled data selection for efficient fine-tuning.

Multi-omics cancer analysis becomes practical even with missing data, achieving state-of-the-art reconstruction and classification via a novel representation learning approach.

Most reported gains in crop disease classification from environmental sensors are illusory, driven by simple session memorization rather than genuine multimodal learning.

A novel bandit algorithm achieves near-optimal dynamic minimax regret for LLM post-training, improving worst-group robustness without re-evaluating historical data or altering standard training pipelines.

A dual VAE architecture effectively bridges the sim-to-real gap for robotic navigation, enabling high performance with minimal real-world data.

Reasoning in base LLMs can be unlocked by simply prepending specific tokens, matching RL-tuned performance on math and coding.

A novel meta-network framework for data selection achieves unprecedented transferability across diverse LLM training tasks by overcoming the limitations of existing MTS objectives.

A handful of verified poisoned examples, not just clean ones, dramatically boosts data poisoning defense against clean-label attacks.

Relational learning with tabular foundation models can now bypass feature explosion and interaction blindness by propagating information directly through the schema graph.

Expert pairwise rankings, not just absolute scores, can efficiently guide scientific discovery, even when experts have varying fidelity and cost.

Generating high-quality, diverse datasets from scratch is now possible, outperforming frontier models by 17% in quality and 19% in diversity.

LLM agents can now build lasting, reusable "competence" for specific knowledge sources, moving beyond simple retrieval to deep, adaptive understanding.

Off-policy data, long thought to be incompatible with effective SFT, can be transformed via MCMC sampling to unlock generalization capabilities that rival or exceed on-policy RL.

Synthetic pre-pretraining boosts LLM efficiency at scale, but not by teaching grammar; it's the retrieval skills that matter.

StreamMAE unlocks self-supervised learning on continuous video streams, matching i.i.d. MAE performance by tackling intra-batch similarity with novel regularization.

Forecast errors, not drift alarms, directly signal when to retrain time series models, achieving state-of-the-art performance with a plug-in module.

LLM agents can improve their decision-making by learning from their own diagnosed failures, with first-proposal corrections boosting success rates by over 30 percentage points.

Wild AI-generated text can actively harm LLM performance on human text, with its value flipping from beneficial to detrimental as token ratios increase.

LLM agents trained solely in rule-generated fictional worlds achieve state-of-the-art transfer to real-world search tasks, outperforming agents trained on actual data.

Language models can learn new knowledge without forgetting old by dynamically prioritizing replay based on observed forgetting.

Human password generation exhibits persistent linguistic regularities that can be exploited for a 16% boost in password recovery by adapting pre-trained language models.

State-of-the-art precipitation downscaling models struggle to generalize across continents, revealing a critical gap in current ML-based climate forecasting.

Finite-sample performance guarantees for label-noise transition matrix estimation are now achievable by bypassing class-posterior estimation through one-sided selective classification.

Zero-shot in-context inference trained strictly on synthetic causal graphs can outperform fully tuned AutoML pipelines across diverse real-world tabular benchmarks.

Non-generative "System One" models can beat closed-source baselines in Chinese decision tasks with a 20x speedup and 15 ms latency, proving encoder-only architectures remain radically underutilized for structured reasoning.

Mixing auxiliary data doesn't just offer more tokens—under the right spectral alignment and dataset growth ratio, it provably accelerates the downstream power-law scaling exponent itself.

Collapsing pre-, mid-, and post-training into a single fully synthetic run matches open-weight model performance while slashing required training tokens by up to 140×.

Training data attribution on 32B-parameter models no longer requires intractable gradient storage: extreme 1-bit eigenbasis projection achieves competitive influence estimation accuracy at over 100× lower per-example footprint than standard baselines.

Cross-lingual data pooling acts as a powerful low-resource jumpstart—boosting macro-F1 by up to 0.43 at 25 labels—but predictably degenerates into negative transfer once local annotations surpass 800 examples.

Standard refusal detectors miss up to 99.6% of noncompliant instruction-tuning data, but multi-pass consensus from a 1.6T-parameter judge successfully isolates millions of latent refusals and evasions at 86.7% precision.

Standard next-token adaptation fails on financial time series unless fine-tuning objectives explicitly penalize rollouts for distorting higher-order temporal structures like volatility clustering.

Synthesizing historical street views is now possible with 17% better realism and consistency by treating it as a reward-guided editing problem.