Search papers, labs, and topics across Lattice.
This paper introduces ReconSpan, a novel method for adaptive latent tokenization that reconstructs text chunks from a single contextual prefix code, significantly improving the efficiency of representation. By applying a reconstruction criterion during chunk formation, the method achieves average chunk lengths between 6.5 and 12.2 while preserving more information than traditional random boundaries. The results show that while the latent sequences maintain reliable topic information, they present challenges in retrieving precise details, highlighting a trade-off in the balance between compression and detail retention.
ReconSpan achieves superior text preservation with adaptive tokenization, outperforming random chunking by retaining more contextual information.
Adaptive latent tokenization maps a fine-grained input to a shorter sequence of continuous representations associated with input-dependent spans. We introduce ReconSpan, which divides text into chunks that a backward decoder can reconstruct from a single contextual prefix code and retains one such code as the latent token for each chunk. The reconstruction criterion is applied when chunks are formed, allowing one trained autoencoder to produce average chunk lengths from 6.5 to 12.2. At matched average length, reconstruction-guided boundaries preserve more text than random boundaries. Readers of the resulting latent sequence recover topic information reliably but struggle to extract exact details.
