Search papers, labs, and topics across Lattice.
This paper investigates whether the parallelogram model, a geometric representation of word analogies, is a poor model of analogical relations or if humans simply struggle to generate relation-preserving analogies. By comparing human and LLM-generated analogies on a shared dataset, the authors found that LLMs consistently produce better analogies, as judged by humans, and these analogies exhibit stronger alignment with the parallelogram structure in distributional embedding space. The improvement stems from LLMs' greater parallelogram alignment and reduced reliance on easily accessible words, suggesting that LLMs are better at capturing relational constraints in analogies than humans.
LLMs aren't just regurgitating facts; they're actually better at generating high-quality, relation-preserving word analogies than humans.
Four-term word analogies (A:B::C:D) are classically modeled geometrically as''parallelograms,''yet recent work suggests this model poorly captures how humans produce analogies, with simple local-similarity heuristics often providing a better account (Peterson et al., 2020). But does the parallelogram model fail because it is a bad model of analogical relations, or because people are not very good at generating relation-preserving analogies? We compared human and large language model (LLM) analogy completions on the same set of analogy problems from (Peterson et al., 2020). We find that LLM-generated analogies are reliably judged as better than human-generated ones, and are also more closely aligned with the parallelogram structure in a distributional embedding space (GloVe). Crucially, we show that the improvement over human analogies was driven by greater parallelogram alignment and reduced reliance on accessible words rather than enhanced sensitivity to local similarity. Moreover, the LLM advantage is driven not by uniformly superior responses by LLMs, but by humans producing a long tail of weak completions: when only modal (most frequent) responses by both systems are compared, the LLM advantage disappears. However, greater parallelogram alignment and lower word frequency continue to predict which LLM completions are rated higher than those of humans. Overall, these results suggest that the parallelogram model is not a poor account of word analogy. Rather, humans may often fail to produce completions that satisfy this relational constraint, whereas LLMs do so more consistently.
