Search papers, labs, and topics across Lattice.
The paper introduces NvRec, an N-version API recommendation system designed to improve the reliability of API recommendations, particularly for infrequently used APIs found in the long tail of code datasets. NvRec profiles the performance of different ML models on individual API methods, considering their tail properties, and uses this profile to filter unreliable recommendations during inference. Experiments using five API recommendation models (CodeBERT, CodeT5, MulaRec, UniXcoder, and CodeT5+) on a Java benchmark dataset demonstrate that NvRec can achieve a high true accept rate while effectively rejecting unreliable API suggestions.
Mitigating long-tail distributions in code datasets boosts API recommendation reliability by up to 10% using an ensemble of models that strategically reject low-confidence predictions.
Machine learning (ML)-based API recommendation helps developers efficiently identify suitable APIs to complement the application code. However, code datasets used to train ML models often exhibit a long-tail distribution, leading to unreliable API recommendations, especially for infrequently used API methods at the tail of the distribution. To address this issue, we propose N-version API Recommendation (NvRec), which leverages N different versions of ML models to enhance the reliability of API sequence recommendations by suppressing unreliable outputs entailing tail APIs. NvRec leverages a set of available ML models and profiles their performance on individual API methods with their tail properties. The generated model profile is used at inference time to filter out unreliable API recommendations and determine the final output. We implement NvRec using five API recommendation models, including CodeBERT, CodeT5, MulaRec, UniXcoder, and CodeT5+, and evaluate it on a public benchmark dataset constructed from compilable Java projects. For the three-version NvRec, we find that the combination of CodeT5, MulaRec, and UniXcoder achieves the highest true accept rate of 83.8%, with a rejection rate of 80.7%, when majority voting is restricted to highly reliable candidates. In contrast, the five-version configuration achieves its highest true accept rate of 83.1% with simple majority voting, while reducing the rejection rate to 69.0%. Overall, the five-version configuration offers a better balance between true accept rate and rejection rate.
