Search papers, labs, and topics across Lattice.
This paper investigates neural scaling laws for masked-reconstruction transformers applied to single-cell RNA sequencing (scRNA-seq) data from the CELLxGENE Census. By training models of varying sizes on both data-rich (200k cells, 512 genes) and data-limited (10k cells, 1024 genes) datasets, the study demonstrates the emergence of power-law scaling in the data-rich regime, characterized by a specific irreducible loss floor. The findings highlight the importance of sufficient data for effective scaling in single-cell transcriptomics and provide an initial estimate of entropy per masked gene position.
Single-cell transcriptomic transformers follow NLP-like scaling laws, but only if you have enough data.
Neural scaling laws -- power-law relationships between loss, model size, and data -- have been extensively documented for language and vision transformers, yet their existence in single-cell genomics remains largely unexplored. We present the first systematic study of scaling behaviour for masked-reconstruction transformers trained on single-cell RNA sequencing (scRNA-seq) data. Using expression profiles from the CELLxGENE Census, we construct two experimental regimes: a data-rich regime (512 highly variable genes, 200,000 cells) and a data-limited regime (1,024 genes, 10,000 cells). Across seven model sizes spanning three orders of magnitude in parameter count (533 to 3.4 x 10^8 parameters), we fit the parametric scaling law to validation mean squared error (MSE). The data-rich regime exhibits clear power-law scaling with an irreducible loss floor of c ~ 1.44, while the data-limited regime shows negligible scaling, indicating that model capacity is not the binding constraint when data are scarce. These results establish that scaling laws analogous to those observed in natural language processing do emerge in single-cell transcriptomics when sufficient data are available, and they identify the data-to-parameter ratio as a critical determinant of scaling behaviour. A preliminary conversion of the data-rich asymptotic floor to information-theoretic units yields an estimate of approximately 2.30 bits of entropy per masked gene position. We discuss implications for the design of single-cell foundation models and outline the additional measurements needed to refine this entropy estimate.
