Search papers, labs, and topics across Lattice.
This study evaluates four tokenization strategies鈥擜ffine, AIM, JetFormer, and VQ-VAE鈥攚ithin a unified transformer framework to understand their impact on learned representations in astronomical imaging. By analyzing 640,000 galaxy images from the DESI Legacy Survey, the authors assess each method's performance in terms of reconstruction fidelity and the prediction of physical properties. The findings reveal that while JetFormer excels in reconstruction quality, VQ-VAE demonstrates superior performance in predicting physical properties, highlighting the trade-offs in tokenization approaches and the decoupling of reconstruction and representation quality.
JetFormer outperforms other tokenization methods in reconstruction quality, but VQ-VAE shines in predicting galaxy physical properties, revealing critical trade-offs in scientific data representation.
Tokenization is central to adapting scientific data for transformer-based foundation models, yet its impact on learned representations remains poorly understood. We compare four tokenization strategies, Affine, AIM, JetFormer, and VQ-VAE, within a unified transformer framework for astronomical imaging. Using 640,000 galaxy images from the DESI Legacy Survey and a shared AstroPT backbone, we evaluate each method on reconstruction fidelity and prediction of physical properties. Our results reveal trade-offs across approaches. The flow-based JetFormer achieves higher reconstruction quality, while VQ-VAE yields strong probe performance for galaxy physical properties. Affine and AIM better preserve localized morphological information. We find that reconstruction and representation quality are decoupled, and no single method consistently performs best across the tasks considered here. By grounding our evaluation in independently measured physical quantities, we hope this study serves to highlight the potential of scientific data as a basis for constructing interpretable benchmarks for foundation models.
