Search papers, labs, and topics across Lattice.
This study investigates how user domain knowledge and AI literacy affect the effectiveness of human-AI interaction within Large Language Model (LLM)-integrated Building Energy Management Systems (BEMS). Researchers conducted a role-playing experiment with 85 participants interacting with GPT-4o to identify behavioral changes for reducing home energy use, analyzing prompt-response data and participant conclusions using a hierarchical analytical framework. The key finding is that AI literacy, rather than domain knowledge, significantly impacted appliance identification rate, suggesting LLMs can equalize expertise levels in BEMS applications.
LLMs in building energy management systems can level the playing field: AI literacy matters more than domain expertise for effective use.
This study aimed to comprehend how user domain knowledge and artificial intelligence (AI) literacy impact the effective use of human-AI interactive building energy management system (BEMS). While prior studies have investigated the potential of integrating large language models (LLMs) into BEMS or building energy modeling, very few studies have examined how user interact with such systems. We conducted a systematic role-playing experiment, where 85 human subjects interacted with an advanced generative pre-trained transformer (OpenAI GPT-4o). Participants were tasked with identifying the top five behavioral changes that could reduce home energy use with the GPT model that functioned as an LLM-integrated BEMS. Then, the collected prompt-response data and participant conclusions were analyzed using an analytical framework that hierarchically assessed and scored human-AI interactions and their home energy analysis approaches. Also, participants were classified into four groups based on their self-evaluated domain knowledge of building energy use and AI literacy, and Kruskal-Wallis H tests with post-hoc pairwise comparisons were conducted across 20 quantifiable metrics. Key takeaways include: most participants employed concise prompts (median: 16.2 words) and relied heavily on GPT's analytical capabilities; and notably, only 1 of 20 metrics, appliance identification rate, showed statistically significant group differences (p=0.037), driven by AI literacy rather than domain knowledge, suggesting an equalizing effect of LLMs across expertise levels. This study provides foundational insights into human-AI collaboration dynamics and promising development directions in the context of LLM-integrated BEMS and contributes to realizing human-centric LLM-integrated energy systems.
