Search papers, labs, and topics across Lattice.
This study evaluated the impact of skill-augmented AI agents on the quality of transcriptomic research analysis in a non-small cell lung cancer biomarker task, comparing outputs from native AI models to those enhanced with a medical research skill package. The evaluation involved six model backbones and included assessments from both non-expert and expert reviewers, revealing that skill-augmented outputs had a directionally higher overall quality rating than native outputs, although the differences were not statistically significant. The findings suggest that while skill augmentation may improve output quality, the variability in expert ratings indicates the need for further investigation with more robust methodologies.
Skill-augmented AI agents may enhance the quality of biomedical research outputs, but the improvement is subtle and requires further validation.
Background. Large language models and AI agents are increasingly used to support biomedical research, but native model outputs may omit key analytical steps, misuse methods, or overstate conclusions. We evaluated whether autonomous access to a medical research skill package was associated with higher-quality AI-generated transcriptomic research-analysis outputs compared with native AI without skills. Methods. We conducted an exploratory multi-model human evaluation using a non-small cell lung cancer immunotherapy biomarker task. Six model backbones were tested. The evaluation included 21 anonymized outputs: 9 native-AI outputs and 12 skill-augmented outputs generated through an AI agent implementation represented by OpenClaw. Four non-expert biomedical reviewers and two blinded experts evaluated each output, with two ratings from each reviewer type. The primary outcome was expert-rated overall quality. Results. Skill-augmented outputs showed directionally higher expert overall quality than native-AI outputs (mean 5.50 vs 5.11; difference=0.39; bootstrap 95\% CI, -0.04 to 0.90; Welch p=0.156). Non-expert reviewer quality showed the same direction (mean 4.72 vs 4.47; difference=0.26; bootstrap 95\% CI, -0.25 to 0.80; Welch p=0.373). Expert agreement was limited (single-rating ICC=-0.15), and model-specific effects were descriptive and heterogeneous. Conclusions. Autonomous skill access showed a directional quality signal in this exploratory sample, but the signal was smaller than expert-rating noise and should not be interpreted as confirmatory evidence. The findings primarily motivate larger evaluations of skill-augmented AI agents with stronger reliability controls, platform replication, and biological-validity assessment.
Tsinghua AI