Search papers, labs, and topics across Lattice.
This paper introduces LLMography, a framework designed to evaluate the interaction process between humans and Large Language Models (LLMs) by transforming conversation histories into measurable indicators of provenance, human contribution, and AI dependency. By analyzing 19 anonymized audit reports from engineering students, the authors found that most interactions were classified as Human-AI co-produced, achieving high scores in key performance indicators such as Human Direction and Prompt Quality. The findings advocate for a shift in AI transparency from merely detecting AI-generated outputs to documenting the entire history of Human-AI interactions, enhancing accountability and reproducibility in AI-assisted tasks.
Most Human-AI interactions are co-produced, with an average Human Direction Score of 86.8, highlighting the importance of documenting conversation histories for transparency.
The growing use of Large Language Models (LLMs) in education, software engineering, academic writing, and technical documentation raises a key question: how can we evaluate not only AI-assisted outputs, but also the interaction process that produced them? Current debates often focus on detecting whether a final artifact was generated by AI, while overlooking the conversation history that reveals human direction, AI contribution, corrections, validation, and traceability. This paper introduces LLMography, a framework for transforming Human-AI conversations into measurable indicators of provenance, human contribution, AI dependency, reproducibility, and auditability. By analogy with bibliography and webography, LLMography documents the dynamic trajectory of interaction between a human and a Large Language Model as a structured trace of Human-AI co-production. We present a prototype that analyzes Human-AI conversation traces and generates KPI reports including Prompt Quality Score, Human Direction Score, AI Dependency Level, Auditability Score, Final Output Traceability, Privacy Risk Level, and a recommended LLMography label. A preliminary exploratory evaluation was conducted on 19 anonymized audit reports from engineering students. Most interactions were classified as Human-AI co-produced, with average scores of 86.8/100 for Human Direction, 81.9/100 for Prompt Quality, 72.8/100 for Auditability, and 77.1/100 for Final Output Traceability. The paper also applies LLMography to its own writing process, classified as human-originated, human-directed, AI-assisted co-production. The findings suggest that AI transparency should move beyond output detection toward documenting the history of interaction.
