Search papers, labs, and topics across Lattice.
This paper investigates whether Language Models (LMs) can perform conceptual analysis through iterative counterexample generation and definition repair, mimicking philosophical methodology. They evaluate LMs across 20 concepts, using one LM to generate counterexamples and another to refine definitions, iterating thousands of times. The study reveals that while LMs generate many invalid counterexamples, an LM judge accepts significantly more than human experts, and extended iteration leads to verbose definitions without accuracy gains, suggesting limitations in LMs' ability to sustain high-level iterated philosophical reasoning.
Language models can play the counterexample game, but their philosophical reasoning hits diminishing returns fast, and they're far more lenient judges than humans.
Conceptual analysis -- proposing definitions and refining them through counterexamples -- is central to philosophical methodology. We study whether language models can perform this task through iterated analysis and repair chains: one model instance generates counterexamples to a proposed definition, another repairs the definition, and the process repeats. Across 20 concepts and thousands of counterexample-repair cycles, we find that, although many LM-generated counterexamples are judged invalid by both expert humans and an LM judge, the LM judge accepts roughly twice as many as humans do. Nonetheless, per-item validity judgments are moderately consistent across humans and between humans and the LM. We further find that extended iteration produces increasingly verbose definitions without improving accuracy. We also see that some concepts resist stable definitions in general. These findings suggest that while LMs can engage in philosophical reasoning, the counterexample-repair loop hits diminishing returns quickly and could be a fruitful test case for evaluating whether LMs can sustain high-level iterated philosophical reasoning.
