Search papers, labs, and topics across Lattice.
This paper analyzes Transformer mechanisms in LLMs using measure theory and frequency analysis, arguing that hallucination arises from the embedding space's failure to be isomorphic to the semantic truth set. To address this, they introduce WavePhaseNet, a method that uses Discrete Fourier Transform (DFT) to construct a Semantic Conceptual Hierarchy Structure (SCHS) by separating global semantics (low frequencies) from local syntax (high frequencies). They demonstrate that reducing GPT-4's embedding space from 24,576 to 3,000 dimensions, combined with cohomological regularization, preserves meaning while enabling more rigorous reasoning and hallucination suppression.
Hallucinations in LLMs? This paper says it's a structural problem, and their DFT-based WavePhaseNet can fix it by shrinking GPT-4's embedding space by 8x.
This paper reformulates Transformer/Attention mechanisms in Large Language Models (LLMs) through measure theory and frequency analysis, theoretically demonstrating that hallucination is an inevitable structural limitation. The embedding space functions as a conditional expectation over a σ-algebra, and its failure to be isomorphic to the semantic truth set fundamentally causes logical consistency breakdown. WavePhaseNet Method The authors propose WavePhaseNet, which explicitly constructs a Semantic Conceptual Hierarchy Structure (SCHS) using Discrete Fourier Transform (DFT). By applying DFT along the sequence dimension, semantic information is decomposed into frequency bands: low-frequency components capture global meaning and intent, while high-frequency components represent local syntax and expression. This staged separation enables precise semantic manipulation in diagonalized space. Dimensionality Reduction GPT-4's 24,576-dimensional embedding space exhibits a 1/f spectral structure based on language self-similarity and Zipf's law. Through cumulative energy analysis, the authors derive that approximately 3,000 dimensions constitute the lower bound for "complete representation." This demonstrates that reduction from 24,576 to 3,000 dimensions preserves meaning and intent while enabling rigorous reasoning and suppressing hallucination. Cohomological Consistency Control The reduced embedding space, constructed via cohomological regularization over overlapping local windows, allows defining a graph structure and cochain complex. This quantifies inconsistencies among local inferences as coboundary-based losses. Applying harmonic projection based on Hodge theory positions cohomology as a computable regularization principle for controlling semantic consistency, extracting maximally consistent global representations.
