Search papers, labs, and topics across Lattice.
This paper systematically investigates the impact of different tree traversal methods鈥擠epth-First Traversal (DFT), Breadth-First Traversal (BFT), and a novel hybrid approach called Production-Rule Traversal (PRT)鈥攐n the performance of Transformer Grammars (TGs) in language modeling. By integrating these traversal strategies with various tree configurations and masking techniques, the authors empirically assess their effects on language modeling, syntactic generalization, and summarization tasks. The findings highlight critical trade-offs between nested composition and global lookahead, offering valuable insights for optimizing TG design based on specific task requirements.
The choice of tree traversal method can significantly alter the performance of Transformer Grammars, revealing trade-offs that could redefine how we approach syntactic modeling in NLP.
Transformer Grammars (TGs) enhance language modeling by incorporating syntactic tree structures. Despite the potentially significant impact on model performance of how syntactic trees are linearized in TGs, existing studies rely solely on Depth-First Traversal (DFT) for linearization. In this paper, we expand the traversal design space by exploring Breadth-First Traversal (BFT) and a novel hybrid traversal strategy, Production-Rule Traversal (PRT), which combines the structural lookahead of BFT with the early lexical generation of DFT. We integrate these traversal methods with varying tree configurations and masking strategies, and empirically evaluate their performance on language modeling, syntactic generalization and summarization. We reveal the inherent trade-offs between nested composition and global lookahead, providing actionable recommendations for designing task-aware Transformer Grammars.
