Search papers, labs, and topics across Lattice.
This paper introduces a self-composable Big-Data-as-a-Service (BDaaS) framework that utilizes LLM-orchestrated multi-agent collaboration to automate the entire data engineering lifecycle, from ingestion to drift detection. By decomposing the BDaaS process into specialized agents and employing a central orchestration layer, the framework enhances workflow reliability and governance while incorporating human oversight and adaptive feedback mechanisms. The prototype evaluation demonstrates that this approach not only matches the predictive performance of existing methods but also significantly improves lifecycle-level metrics such as artifact traceability and drift recovery.
LLM-orchestrated multi-agent systems can revolutionize BDaaS by providing a trustworthy, adaptive framework that outperforms traditional methods in lifecycle reliability and governance.
Big-Data-as-a-Service (BDaaS) platforms require re liable automation across data ingestion, cleaning, feature engi neering, model development, deployment, and post-deployment monitoring. However, existing LLM-based data science agents and AutoML systems mainly focus on isolated workflow stages, leaving limited support for lifecycle-level orchestration, artifact governance, human oversight, and drift-aware adaptation. This paper proposes a trustworthy self-composable BDaaS frame work based on LLM-orchestrated multi-agent collaboration. The proposed architecture decomposes the BDaaS lifecycle into specialized agents for data ingestion, data cleaning, feature engineering, AutoML training, model evaluation, MLOps de ployment, monitoring, and drift detection. A central LLM or chestration layer coordinates agent execution, validates interme diate outputs, manages workflow context, and enables dynamic workflow composition. The framework also incorporates shared artifact governance, reproducibility support, human-in-the-loop checkpoints, and drift-aware feedback loops. A prototype-based evaluation is conducted using controlled tabular benchmark datasets with missing values, categorical variables, outliers, class imbalance, and simulated covariate drift. Compared with manual ML, AutoML-only, and single-agent LLM baselines, the pro posed multi-agent BDaaS pipeline achieves competitive predictive performance while improving lifecycle-level reliability, including workflow completion, artifact traceability, deployment readiness, reproducibility, and drift recovery. The results suggest that LLM-orchestrated multi-agent systems can extend conventional AutoML toward trustworthy, adaptive, and production-oriented BDaaS lifecycle automation.
