Search papers, labs, and topics across Lattice.
This paper conducts a systematic empirical study on the effectiveness of fine-tuning and the validity of metrics for neural decompilation of Dart AOT binaries, evaluating six fine-tuned model variants across three architectures using a new benchmark. The findings reveal that no fine-tuning configuration leads to a statistically significant improvement in pass@k, with one configuration resulting in a significant regression, and highlight the impact of cross-lingual interference and metric divergence on evaluation outcomes. Additionally, the study identifies assembly sequence length as a critical predictor of task difficulty, underscoring the need for pass@k as the primary evaluation metric in this domain.
Fine-tuning for neural decompilation of Dart binaries fails to yield significant improvements, revealing critical pitfalls in current evaluation metrics.
Neural decompilation is increasingly studied as a code-generation problem, yet its evaluation methodology remains underdeveloped for modern languages. We present a systematic empirical study of fine-tuning effectiveness and metric validity for Dart Ahead-of-Time (AOT) neural decompilation. We evaluate six fine-tuned model variants across three base architectures (4B-8B parameters) using three metrics: CodeBLEU, compile@k, and pass@k on a new 154-task HumanEval-Dart benchmark. Our study yields three principal findings grounded in paired task-level statistical tests. First, no fine-tuning configuration produces a statistically significant pass@k improvement. The sole positive case yields +0.71 pp (McNemar p=0.21), while fine-tuning the strongest base (Qwen3-8B) causes a highly significant regression of -5.65 pp (p<0.001). This capacity-dependent trend is consistent across architectures but needs broader scale sweeps. Second, cross-lingual interference from Swift training is highly significant at 4B (-2.66 pp, p<0.001) but statistically indistinguishable from zero at 8B, consistent with the scaling hypothesis. Third, we demonstrate metric divergence: CodeBLEU and compile@k can improve significantly while pass@k moves in the opposite direction. This has implications for any LLM code generation task where fine-tuning targets superficial similarity. Error analysis reveals assembly sequence length is the strongest predictor of task difficulty (p=0.001), with a capability cliff at 200 instructions. We contribute the HumanEval-Dart benchmark, a Dart-adapted CodeBLEU, and empirical evidence that pass@k must be the primary evaluation metric for neural decompilation.
