Search papers, labs, and topics across Lattice.
This study investigates the usability of a conversational product advisor designed to enhance transparency in laptop searches by providing ranking explanations and comparison features. Despite high satisfaction and ease of use reported by participants, the study revealed that the transparency intended to aid understanding often led to confusion, highlighting a disconnect between users' appreciation for explanations and their actual comprehension. Additionally, while participants valued the time saved by the advisor, many expressed a desire for more direct control over their interactions, indicating areas for improvement in design.
Transparency in conversational product advisors can backfire, leading to confusion rather than clarity for users.
Large language models can make conversational product advisors fluent but opaque. If they hide the logic behind a ranking and the evidence for a recommendation inside natural-language replies, they challenge users'ability to understand, trust, and steer the results. One response is to build transparency into the advisor. We report a formative, moderated think-aloud usability study of one such system: a chatbot for laptop search with constrained natural-language generation, an on-demand ranking explanation, and a comparison feature. Seven participants completed three laptop-search tasks and reported post-task usability measures. We coded their sessions into severity-rated usability problems. Ease and satisfaction during the tasks were high, but two findings stand out. First, transparency by design did not guarantee understanding: several participants valued the ranking explanation in principle, yet it caused the most severe problem. Second, participants valued the effort the advisor saved, but some wanted additional direct-manipulation controls. We contribute a severity-prioritized set of usability problems and design implications for human-centered conversational product advisors.
