Search papers, labs, and topics across Lattice.
This study addresses the cold start problem in item calibration by leveraging modern text embeddings to predict item parameters through a novel evaluation framework that combines regularized regression and performance upper bounds. The findings reveal that while item difficulty can be predicted with a moderate degree of accuracy (R虏 = 0.53), the predictability of discrimination and pseudo guessing is significantly lower, constrained by the reliability ceilings of these parameters. Importantly, the research highlights the necessity of repeated cross validation to avoid inflated accuracy estimates and emphasizes the limitations of current benchmarks in capturing true performance variance.
Text embeddings can predict item difficulty with surprising accuracy, but the predictability of other parameters is limited by their inherent reliability ceilings.
Newly developed items must ordinarily be field tested before their psychometric properties are known, creating a cold start problem for item calibration. Predicting item parameters from features is a long standing measurement problem dating back to the Linear Logistic Test Model; modern text embeddings now automate the design matrices traditionally specified by hand. We propose an evaluation framework combining regularized regression on item text embeddings, repeated cross validated R squared reported with its resampling standard deviation, and two performance upper bounds: a reliability ceiling derived from parameter standard errors, and a design ceiling derived from simulation based power calibration. Applying this framework to a mathematics item bank (EEDI) and a medical licensure benchmark (BEA 2024), we find that item difficulty is highly predictable from text (repeated cross validated R squared = 0.53, or about 57% of its reliability ceiling), whereas discrimination and pseudo guessing appear less predictable. However, evaluating these results against our ceilings reveals that this apparent hierarchy stems from target reliability rather than text signal strength: text uniformly recovers 57 to 63% of the reliable variance across difficulty targets, whereas the 3PL pseudo guessing parameter has a reliability ceiling near zero, making it an unviable target at current precision. On BEA, embedding based regression matches leaderboard RMSE despite explaining almost no variance, highlighting the critical need for scale free metrics and explicit ceilings in benchmarking. Finally, we show that a single train and test split can inflate apparent accuracy by 0.1 to 0.15 in R squared, underscoring the necessity of repeated cross validation for calibration support applications and future benchmark construction.
