Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates LLM-based automated test generation, comparing performance on the open-source LevelDB against the proprietary SAP HANA database system to assess generalization. They use mutation score and compiler-feedback repair loops to evaluate test accuracy and reasoning strategies. The results show that LLMs perform well on the familiar LevelDB but struggle with the unseen SAP HANA codebase, prioritizing compilability over semantic correctness, suggesting a lack of robust reasoning.
LLMs can ace test generation on open-source code, but fall apart when faced with proprietary systems, revealing a reliance on memorization over genuine reasoning.
Large Language Models (LLMs) have achieved impressive results on public benchmarks, often leading to claims of advanced reasoning and understanding. However, recent research in cognitive science reveals that these models sometimes rely on shallow heuristics and memorization, taking shortcuts rather than demonstrating genuine cognitive abilities. This paper investigates LLM behavior in automated test generation for software, contrasting performance on an open-source system (LevelDB) with SAP HANA, one of the most widely deployed commercial database systems worldwide, whose proprietary codebase is guaranteed to be absent from training data. We combine cognitive evaluation principles, drawing on Mitchell's mechanism-focused assessment methodology, with empirical software testing, employing mutation score and iterative compiler-feedback repair loops to assess both accuracy and underlying reasoning strategies. Results show that LLMs excel on familiar, open-source benchmarks but struggle with unseen, complex domains, often prioritizing compilability over semantic effectiveness. These findings provide independent software engineering evidence for the broader claim that current LLMs lack robust reasoning, and highlight the need for evaluation frameworks that penalize trivial shortcuts and reward true generalization.