Data Curation & Synthetic Data - Weekly Roundup

Slash LLM evaluation costs by over 270x without sacrificing accuracy, revealing that even top open-weight models struggle to outperform simple baselines on AML transaction fraud detection.

Small models can now learn new skills without forgetting old ones, thanks to a novel distillation technique that leverages a larger model to guide the learning process.

One million parallel environments are insufficient without intelligent sampling: Success Guided Sampling unlocks mega-scale RL for robots by focusing training on the edge of learned capabilities.

Rare concepts in LLM fine-tuning are significantly hindered by "prior barriers" that require targeted supervision, not just more data.

A novel hierarchical decomposition into soft communities allows for scalable attributed graph generation that balances structural fidelity and downstream utility without memorizing the reference graph.

Slash LLM evaluation costs by over 270x without sacrificing accuracy, revealing that even top open-weight models struggle to outperform simple baselines on AML transaction fraud detection.

Small models can now learn new skills without forgetting old ones, thanks to a novel distillation technique that leverages a larger model to guide the learning process.

One million parallel environments are insufficient without intelligent sampling: Success Guided Sampling unlocks mega-scale RL for robots by focusing training on the edge of learned capabilities.

Rare concepts in LLM fine-tuning are significantly hindered by "prior barriers" that require targeted supervision, not just more data.

A novel hierarchical decomposition into soft communities allows for scalable attributed graph generation that balances structural fidelity and downstream utility without memorizing the reference graph.

Human motion alone, when translated into a shared latent space, unlocks zero-shot generalist humanoid control for locomotion and manipulation tasks.

Out-of-distribution data can dramatically boost discrete diffusion model performance under extreme data scarcity by strategically injecting it at low noise levels.

Large observational studies can finally boost the precision of small RCTs for treatment effect estimation without sacrificing unbiasedness or making strong confounding assumptions.

Satellite imagery unlocks fine-grained socioeconomic predictions with calibrated uncertainty, even without direct ground truth.

Frontier LLMs fail to reliably identify high-value training data or accurately predict performance gains, highlighting a critical gap in their self-improvement potential.

Synthetic data alone can match real-world GAD performance, but unlocking further gains requires models that explicitly fuse topological and semantic evidence.

Human tactile gloves costing under $11 can now generate robot-ready tactile data, slashing acquisition costs by over 95% and boosting efficiency 3.5x for dexterous manipulation.

Language models can actively select replay targets to improve their own broad capability preservation, not just retain explicitly trained tasks.

Synthesized software environments unlock a 50% boost in CUA performance on unseen benchmarks, proving scalable supervision is key.

Audio-language models excel at identifying sounds but fail to grasp the crucial context of how speakers react to them.

Forget-only unlearning demands models memorize far more than standard training, challenging the premise of efficient deletion.

A 300K-sample dataset and diagnostic benchmark unlock the challenging task of generating product-preserving, narrative-driven advertisement videos.

Pretraining neural CFD surrogates across diverse domains unlocks substantial gains in zero- and few-shot generalization, drastically reducing data requirements for new applications.

Recurrent non-stationarity unlocks data reuse, but its "safe" application hinges on a bias-variance tradeoff akin to classical settings, with a novel algorithm achieving near-optimal regret.

A unified, open-source framework and a massive 3.3 TB dataset now enable rigorous ML-driven prediction of complex multiphase flows in porous media, accelerating research in critical energy and engineering applications.

Instruction-tuned models can achieve monolithic training performance on new domains via parallel, independent LoRA adapter training and parameter-free routing, eliminating costly full-model retraining.

Differentially private synthetic time-series forecasting utility hinges on forecaster-specific window assembly, not just generator quality.

Unbiased dataset pruning can be framed as a linear program, enabling efficient, label-free subset selection that outperforms geometric methods at small budgets.

Uncoded crash factors like phone use and medical episodes are vastly undercounted in police records, with narratives revealing 2x more injury crashes than coded fields.

Transformers learn translation, rotation, and scale invariance through distinct, identifiable mechanisms, paving the way for structured equivariant function learning.

Smoothness in treatment effects unlocks near-optimal policy learning in high-dimensional, low-signal digital experimentation.

Learned relational structure from dynamics fails to transfer to inverse problems, even when edge accuracy is high, due to uncaptured approximation errors.

Optimizing BO hyperparameters with BO itself reveals configurations that significantly reduce cumulative regret on GP-inferred sample paths.

AI is poised to revolutionize ARPES, moving beyond data analysis to enable fully autonomous, self-driving experimental labs.

Provably private tabular classification is now achievable with a single forward pass, outperforming baselines and drastically cutting fitting time.

A single 500-hour visual-tactile dataset boosts zero-shot contact prediction from 13% to 38% IoU and quadruples robot manipulation success rates, proving scale and consistency are paramount for embodied learning.

Generative models can now produce inorganic crystals with oxidation-state distributions mirroring real-world synthesis, significantly boosting discovery potential.

A simulation flywheel recursively bootstraps embodied data, generating richer scenes and tasks for robot pretraining and evaluation.

Inserted objects can now be made "alive" and interact realistically within videos, solving a long-standing challenge in video editing.

Deterministic dermoscopic preprocessing actually degrades performance in leakage-controlled evaluations, suggesting current automated analysis might be over-engineering for minimal gains.

Transcriptomics-specific pretraining, not model scale, unlocks state-of-the-art clinical outcome prediction from bulk RNA-seq, even for survival tasks, with a 4.2M parameter model.

Satellite images are not $n^2$ independent samples; their effective sample size plummets to $\Theta(n^2/r^2)$ due to spatial correlation, fundamentally altering generalization analysis.

Time-series foundation models, despite their promise for "what-if" analysis, exhibit attenuated dynamics and memorylessness, requiring synthetic fine-tuning to recover accurate causal inference.

Distributed MMD with Random Fourier Features achieves near-perfect correlation with expected drift magnitude on massive e-commerce data, while per-dimension KS tests falter.

LLMs can now generate novel research ideas by explicitly learning to synthesize literature through structured "anchor" specifications, outperforming prior prompting and feedback methods.

Discovering molecules with rare properties can now be an order of magnitude faster by testing complex mixtures and deconvoluting activity, slashing experimental costs from $\mathcal{O}(d)$ to $\mathcal{O}(\log d)$ or $\mathcal{O}(1)$.

Tailoring public dataset mixtures for DP pre-training via private linear regression unlocks substantial utility gains, boosting X-ray classification AUC by up to 0.037 and reducing language model perplexity by 16%.

A transformer pretrained on synthetic, never-repeating "languages" learns to infer and predict real-world text and perform complex reasoning tasks, suggesting a fundamental mechanism for language acquisition.

Humanoid robots can now learn complex loco-manipulation tasks from sparse demonstrations by iteratively evolving their own training data and policies.

LLM agents can now dynamically curate multimodal memory on-demand, achieving optimal performance-cost-latency trade-offs by intelligently choosing between pre-computed and runtime-generated evidence.

A single graph foundation model learns to wander across diverse graph structures and tasks, achieving SOTA on node classification, homogeneous link prediction, and KG link prediction.

A single pre-trained transformer now solves matrix completion across causal inference, recommendation, and tabular imputation tasks by treating matrices natively.

Confident conflicts in LLM distillation are mitigated by dynamically mixing student and teacher distributions based on student entropy, boosting reasoning and preserving general capabilities.

Clinical ML models are only as good as their filtered data, yet current practices obscure crucial scientific decisions that shape task complexity and performance.

The optimal LoRA rank is not intrinsic to the model but dynamically dictated by the data's geometric properties, enabling principled data selection for efficient fine-tuning.

Multi-omics cancer analysis becomes practical even with missing data, achieving state-of-the-art reconstruction and classification via a novel representation learning approach.

Most reported gains in crop disease classification from environmental sensors are illusory, driven by simple session memorization rather than genuine multimodal learning.

A novel bandit algorithm achieves near-optimal dynamic minimax regret for LLM post-training, improving worst-group robustness without re-evaluating historical data or altering standard training pipelines.

A dual VAE architecture effectively bridges the sim-to-real gap for robotic navigation, enabling high performance with minimal real-world data.

Reasoning in base LLMs can be unlocked by simply prepending specific tokens, matching RL-tuned performance on math and coding.