Search papers, labs, and topics across Lattice.
This paper investigates the semantic consistency of call graphs generated by four Java static analysis frameworks (Soot, SootUp, WALA, and Doop). By systematically comparing call graphs produced with varying levels of precision within and across frameworks, the authors identify frequent violations of expected precision orders and semantic inconsistencies. The study reveals that algorithmic precision orders break down due to language features, configuration choices interact synergistically with algorithms, and frameworks operate with incompatible notions of call-graph ground truth.
Widely used Java static analysis frameworks produce call graphs with irreconcilable semantic gaps, challenging the assumption that they yield comparable results.
Java static analysis frameworks are commonly compared under the assumption that analysis algorithms and configurations compose monotonically and yield semantically comparable results across tools. In this work, we show that this assumption is fundamentally flawed. We present a large-scale empirical study of semantic consistency within and across four widely used Java static analysis frameworks: Soot, SootUp, WALA, and Doop. Using precision partial orders over analysis algorithms and configurations, we systematically identify violations where increased precision introduces new call-graph edges or amplifies inconsistencies. Our results reveal three key findings. First, algorithmic precision orders frequently break within frameworks due to modern language features such as lambdas, reflection, and native modeling. Second, configuration choices strongly interact with analysis algorithms, producing synergistic failures that exceed the effects of algorithm or configuration changes alone. Third, cross-framework comparisons expose irreconcilable semantic gaps, demonstrating that different frameworks operate over incompatible notions of call-graph ground truth. These findings challenge prevailing evaluation practices in static analysis and highlight the need to reason jointly about algorithms, configurations, and framework semantics when assessing precision and soundness.
