Search papers, labs, and topics across Lattice.
This paper introduces FRAGMENT, a novel generative framework that models documents as typed relational graphs, effectively capturing the dependencies between spatial layout, textual content, and logical structure. By employing a two-stage approach鈥攆irst generating the graph topology with a causally masked Transformer and then enriching it with a GATv2-based graph attention network鈥擣RAGMENT achieves a tractable document-level likelihood for applications such as forgery detection and controlled editing. Experimental results demonstrate that FRAGMENT outperforms traditional autoregressive and layout-only models across multiple datasets, highlighting its effectiveness in structured document generation and editing tasks.
FRAGMENT's factorized graph representation enables precise document generation and editing, outperforming conventional models in capturing complex dependencies.
Structured documents such as invoices, forms, reports, and scientific articles derive meaning from the interplay between spatial layout, textual content, and logical structure. Generative models operating at the pixel or token level often struggle to capture these dependencies effectively. We explore FRAGMENT, a generative framework that represents a document as a typed relational graph and factorizes its distribution as p(structure, content) = p(structure) * p(content | structure). The framework consists of two stages. The first stage, the Architect, is a causally masked Transformer conditioned on document category that autoregressively generates the graph topology and typed spatial relations. The second stage, the Builder, is a GATv2-based graph attention network that enriches the graph with normalized bounding boxes, text, and visual style attributes. Both stages define explicit likelihood models, yielding a tractable document-level likelihood that serves as an anomaly score for forgery detection. For controlled editing, a prompt-conditioned extension injects instruction embeddings into the Builder through cross-attention, enabling semantic and entity-aware modifications. We describe training on DocLayNet and fine-tuning on FUNSD and SROIE. Experiments on DocLayNet, FUNSD, and SROIE evaluate FRAGMENT alongside representative autoregressive, layout-only, and graph-based baselines, providing an empirical analysis of the characteristics and trade-offs of the proposed factorized graph generation framework.
