Search papers, labs, and topics across Lattice.
This study introduces Principal Trait Analysis (PTA), an algorithm inspired by Principal Component Analysis, to identify and quantify traits that influence task success in human-AI collaboration. By analyzing conversation data from two distinct settings鈥攕tudents with an AI tutor and developers with an AI coding agent鈥攖he authors reveal that specific traits significantly correlate with effective collaboration and task outcomes. However, the research raises questions about the generalizability of these traits as skills, highlighting the need for further investigation into their stability over time.
Identifying key traits in human-AI collaboration could redefine how we understand and teach the skills necessary for successful teamwork with AI agents.
Large Language Model-powered agents are increasingly used in the workplace via human-artificial intelligence (AI) collaboration. In this new era of work, it is important to understand the kinds of prompting traits that contribute to task success. Moreover, we need to uncover key skills required for modern professionals and inform educators on how to foster these skills among students. Existing guidelines for human-AI collaboration are built from either top-down theory or context-specific observations of human-AI interactions. However, since LLM capabilities are rapidly improving, theory may not be able to explain emerging interaction patterns, and empirical guidelines may become obsolete quickly. In this work, we explore an automated, data-driven approach to uncover patterns, which we term traits, of effective human-AI interaction that are aligned with task outcomes. We propose Principal Trait Analysis, a Principal Component Analysis-inspired algorithm for deriving common traits from patterns in LLM conversations. Our algorithm uses LLM-based processing stages to analyze corpora of human-AI collaborative session traces, deriving common traits across the dataset and scoring each human collaborator's usage style by each trait. The approach also allows domain expertise to be injected during trait discovery and selects the most distinguishing traits to be those that exhibit the highest variance across collaborators. We evaluate PTA on two human-AI collaborative coding datasets, an educational setting (students working with an AI tutor) and a professional setting (developers working with an AI coding agent). We find that PTA-derived traits are significant in explaining collaborator behavior across both settings and can help predict task outcomes. However, whether traits qualify as skills remains to be seen, due to inconclusive results on generalizability and how user traits change over time.
