Search papers, labs, and topics across Lattice.
This paper addresses the challenges in Method Name Prediction (MNP) by contrasting traditional token similarity metrics with human semantic evaluations, revealing that LLM-based evaluators, particularly those using DeepSeek, align more closely with human judgments. The authors introduce a novel approach, SMNP, which integrates summarization and chain-of-thought refinement to enhance the semantic quality of generated method names. Experimental results across multiple LLMs and datasets confirm that SMNP significantly outperforms existing methods in generating meaningful and contextually appropriate method names.
LLM-based evaluators can outperform traditional metrics in method name prediction, but the real breakthrough comes from a novel approach that enhances name quality through summarization and refinement.
Method names are critical to software quality, affecting code comprehensibility, maintainability, and developer collaboration. However, manually designing meaningful method names is challenging. Method Name Prediction (MNP), which automatically generates method names from code snippets, has recently attracted attention. Although large language models (LLMs) show promising performance for MNP, two challenges remain. First, existing evaluations mainly rely on token similarity metrics, which often fail to reflect human judgments of semantic quality. Second, current LLM-based MNP methods usually generate names through direct code-to-name mapping, which differs from the human process of understanding functionality before naming. To address these challenges, we conduct empirical studies on LLM-based evaluation and MNP strategies. We compare 6 metric-based evaluators, 5 LLM-based evaluators, and 6 human evaluators. Results show that LLM-based evaluators, especially DeepSeek-based evaluators, are more consistent with human judgments than traditional metrics. We further compare direct generation and summarization-and-refinement strategies. Results indicate that summarization and refinement generally improve the semantic quality of generated names. Case studies reveal three limitations: inaccurate summaries, semantic misalignment, and close semantic scores. Based on these findings, we propose SMNP, an MNP approach combining MNP-oriented summarization and chain-of-thought enhanced refinement. Experiments on 5 LLMs and 2 datasets demonstrate the effectiveness and robustness of SMNP.
