Data Curation & Synthetic Data - Weekly Roundup

Audio-language models excel at identifying sounds but fail to grasp the crucial context of how speakers react to them.

Forget-only unlearning demands models memorize far more than standard training, challenging the premise of efficient deletion.

A 300K-sample dataset and diagnostic benchmark unlock the challenging task of generating product-preserving, narrative-driven advertisement videos.

Pretraining neural CFD surrogates across diverse domains unlocks substantial gains in zero- and few-shot generalization, drastically reducing data requirements for new applications.

Recurrent non-stationarity unlocks data reuse, but its "safe" application hinges on a bias-variance tradeoff akin to classical settings, with a novel algorithm achieving near-optimal regret.

Audio-language models excel at identifying sounds but fail to grasp the crucial context of how speakers react to them.

Forget-only unlearning demands models memorize far more than standard training, challenging the premise of efficient deletion.

A 300K-sample dataset and diagnostic benchmark unlock the challenging task of generating product-preserving, narrative-driven advertisement videos.

Pretraining neural CFD surrogates across diverse domains unlocks substantial gains in zero- and few-shot generalization, drastically reducing data requirements for new applications.

Recurrent non-stationarity unlocks data reuse, but its "safe" application hinges on a bias-variance tradeoff akin to classical settings, with a novel algorithm achieving near-optimal regret.

A unified, open-source framework and a massive 3.3 TB dataset now enable rigorous ML-driven prediction of complex multiphase flows in porous media, accelerating research in critical energy and engineering applications.

Instruction-tuned models can achieve monolithic training performance on new domains via parallel, independent LoRA adapter training and parameter-free routing, eliminating costly full-model retraining.

Differentially private synthetic time-series forecasting utility hinges on forecaster-specific window assembly, not just generator quality.

Unbiased dataset pruning can be framed as a linear program, enabling efficient, label-free subset selection that outperforms geometric methods at small budgets.

Uncoded crash factors like phone use and medical episodes are vastly undercounted in police records, with narratives revealing 2x more injury crashes than coded fields.

Transformers learn translation, rotation, and scale invariance through distinct, identifiable mechanisms, paving the way for structured equivariant function learning.

Smoothness in treatment effects unlocks near-optimal policy learning in high-dimensional, low-signal digital experimentation.

Learned relational structure from dynamics fails to transfer to inverse problems, even when edge accuracy is high, due to uncaptured approximation errors.

Optimizing BO hyperparameters with BO itself reveals configurations that significantly reduce cumulative regret on GP-inferred sample paths.

AI is poised to revolutionize ARPES, moving beyond data analysis to enable fully autonomous, self-driving experimental labs.

Provably private tabular classification is now achievable with a single forward pass, outperforming baselines and drastically cutting fitting time.

A single 500-hour visual-tactile dataset boosts zero-shot contact prediction from 13% to 38% IoU and quadruples robot manipulation success rates, proving scale and consistency are paramount for embodied learning.

Generative models can now produce inorganic crystals with oxidation-state distributions mirroring real-world synthesis, significantly boosting discovery potential.

A simulation flywheel recursively bootstraps embodied data, generating richer scenes and tasks for robot pretraining and evaluation.

Inserted objects can now be made "alive" and interact realistically within videos, solving a long-standing challenge in video editing.

Deterministic dermoscopic preprocessing actually degrades performance in leakage-controlled evaluations, suggesting current automated analysis might be over-engineering for minimal gains.

Transcriptomics-specific pretraining, not model scale, unlocks state-of-the-art clinical outcome prediction from bulk RNA-seq, even for survival tasks, with a 4.2M parameter model.

Satellite images are not $n^2$ independent samples; their effective sample size plummets to $\Theta(n^2/r^2)$ due to spatial correlation, fundamentally altering generalization analysis.

Time-series foundation models, despite their promise for "what-if" analysis, exhibit attenuated dynamics and memorylessness, requiring synthetic fine-tuning to recover accurate causal inference.

Distributed MMD with Random Fourier Features achieves near-perfect correlation with expected drift magnitude on massive e-commerce data, while per-dimension KS tests falter.

LLMs can now generate novel research ideas by explicitly learning to synthesize literature through structured "anchor" specifications, outperforming prior prompting and feedback methods.

Discovering molecules with rare properties can now be an order of magnitude faster by testing complex mixtures and deconvoluting activity, slashing experimental costs from $\mathcal{O}(d)$ to $\mathcal{O}(\log d)$ or $\mathcal{O}(1)$.

Tailoring public dataset mixtures for DP pre-training via private linear regression unlocks substantial utility gains, boosting X-ray classification AUC by up to 0.037 and reducing language model perplexity by 16%.

A transformer pretrained on synthetic, never-repeating "languages" learns to infer and predict real-world text and perform complex reasoning tasks, suggesting a fundamental mechanism for language acquisition.

Humanoid robots can now learn complex loco-manipulation tasks from sparse demonstrations by iteratively evolving their own training data and policies.

LLM agents can now dynamically curate multimodal memory on-demand, achieving optimal performance-cost-latency trade-offs by intelligently choosing between pre-computed and runtime-generated evidence.

A single graph foundation model learns to wander across diverse graph structures and tasks, achieving SOTA on node classification, homogeneous link prediction, and KG link prediction.

A single pre-trained transformer now solves matrix completion across causal inference, recommendation, and tabular imputation tasks by treating matrices natively.

Confident conflicts in LLM distillation are mitigated by dynamically mixing student and teacher distributions based on student entropy, boosting reasoning and preserving general capabilities.

Clinical ML models are only as good as their filtered data, yet current practices obscure crucial scientific decisions that shape task complexity and performance.

The optimal LoRA rank is not intrinsic to the model but dynamically dictated by the data's geometric properties, enabling principled data selection for efficient fine-tuning.

Multi-omics cancer analysis becomes practical even with missing data, achieving state-of-the-art reconstruction and classification via a novel representation learning approach.

Most reported gains in crop disease classification from environmental sensors are illusory, driven by simple session memorization rather than genuine multimodal learning.

A novel bandit algorithm achieves near-optimal dynamic minimax regret for LLM post-training, improving worst-group robustness without re-evaluating historical data or altering standard training pipelines.

A dual VAE architecture effectively bridges the sim-to-real gap for robotic navigation, enabling high performance with minimal real-world data.

Reasoning in base LLMs can be unlocked by simply prepending specific tokens, matching RL-tuned performance on math and coding.

A novel meta-network framework for data selection achieves unprecedented transferability across diverse LLM training tasks by overcoming the limitations of existing MTS objectives.

A handful of verified poisoned examples, not just clean ones, dramatically boosts data poisoning defense against clean-label attacks.

Relational learning with tabular foundation models can now bypass feature explosion and interaction blindness by propagating information directly through the schema graph.

Expert pairwise rankings, not just absolute scores, can efficiently guide scientific discovery, even when experts have varying fidelity and cost.

Generating high-quality, diverse datasets from scratch is now possible, outperforming frontier models by 17% in quality and 19% in diversity.

LLM agents can now build lasting, reusable "competence" for specific knowledge sources, moving beyond simple retrieval to deep, adaptive understanding.

Off-policy data, long thought to be incompatible with effective SFT, can be transformed via MCMC sampling to unlock generalization capabilities that rival or exceed on-policy RL.

Synthetic pre-pretraining boosts LLM efficiency at scale, but not by teaching grammar; it's the retrieval skills that matter.

StreamMAE unlocks self-supervised learning on continuous video streams, matching i.i.d. MAE performance by tackling intra-batch similarity with novel regularization.

Forecast errors, not drift alarms, directly signal when to retrain time series models, achieving state-of-the-art performance with a plug-in module.

LLM agents can improve their decision-making by learning from their own diagnosed failures, with first-proposal corrections boosting success rates by over 30 percentage points.

Wild AI-generated text can actively harm LLM performance on human text, with its value flipping from beneficial to detrimental as token ratios increase.

LLM agents trained solely in rule-generated fictional worlds achieve state-of-the-art transfer to real-world search tasks, outperforming agents trained on actual data.

Language models can learn new knowledge without forgetting old by dynamically prioritizing replay based on observed forgetting.

Human password generation exhibits persistent linguistic regularities that can be exploited for a 16% boost in password recovery by adapting pre-trained language models.

State-of-the-art precipitation downscaling models struggle to generalize across continents, revealing a critical gap in current ML-based climate forecasting.

Finite-sample performance guarantees for label-noise transition matrix estimation are now achievable by bypassing class-posterior estimation through one-sided selective classification.