Search papers, labs, and topics across Lattice.
This study tackles the challenge of managing code deployments at Amazon Prime Video by introducing a framework that utilizes diff-aware features derived from code modifications to predict deployment risks. By employing large language models (LLMs) as multi-language feature extractors, the framework identifies essential quantitative and qualitative indicators for risk prediction, achieving an impressive average recall of 0.83 and F1 score of 0.81 across diverse datasets. The findings highlight that while traditional metrics like change volume are less effective, structural code complexity serves as a robust predictor, enabling more precise risk assessments without compromising developer privacy.
Thoughtful feature curation reveals that structural code complexity is a far stronger predictor of deployment risk than traditional change volume metrics.
At Amazon Prime Video, we face the critical operational challenge of managing code deployments during live events and rapid feature releases without causing service outages. Current change control approaches use blanket deployment freezes that block all changes regardless of risk, creating significant developer toil. While prior research has explored risky change predictors, these rely on developer-specific metadata or extensive historical data, raising privacy concerns and limiting applicability to new projects. We introduce a framework centered on diff-aware features, characteristics derived directly from code modifications. Our key contribution is the systematic identification of which quantitative metrics (code-level and change-level metrics) and qualitative indicators (coding style violations, change type classification) are necessary for risk prediction. We employ LLMs as multi-language feature extractors, demonstrating their effectiveness for code analysis beyond generation tasks and eliminating the need for language-specific tooling. We evaluated our framework on two datasets: Prime Video's production environment and the public ApacheJIT dataset. Our best-performing model achieves an average recall of 0.83 and F1 score of 0.81 across both datasets for detecting risky code changes. Notably, ablation analysis reveals that change-level volume metrics (e.g., lines added/deleted) are noisy predictors, while structural code complexity provides a substantially stronger risk signal. These results demonstrate that thoughtful feature curation enables effective change risk assessment across different programming languages and organizational contexts while avoiding privacy concerns.
Amazon Science