Search papers, labs, and topics across Lattice.
This paper explores using GPT-4.1-nano to predict the difficulty of data visualization literacy test items based on text, visual, and multimodal features. The study found that a multimodal approach, combining visual and textual information, achieved the lowest mean absolute error (MAE) of 0.224 in predicting item difficulty. External evaluation on a held-out test set further validated the multimodal model's effectiveness, suggesting LLMs can assist in psychometric analysis and automated item generation.
Forget expert surveys: GPT-4.1-nano can predict the difficulty of data visualization test questions with surprisingly high accuracy, especially when combining visual and textual cues.
This project investigates the capabilities of large language models (LLMs) to determine the difficulty of data visualization literacy test items. We explore whether features derived from item text (question and answer options), the visualization image, or a combination of both can predict item difficulty (proportion of correct responses) for U.S. adults. We use GPT-4.1-nano to analyze items and generate predictions based on these distinct feature sets. The multimodal approach, using both visual and text features, yields the lowest mean absolute error (MAE) (0.224), outperforming the unimodal vision-only (0.282) and text-only (0.338) approaches. The best-performing multimodal model was applied to a held-out test set for external evaluation and achieved a mean squared error of 0.10805, demonstrating the potential of LLMs for psychometric analysis and automated item development.
