Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors introduce TalkSketchD, a novel dataset of freehand toaster sketches paired with temporally aligned spontaneous speech, capturing designers' intent during early-stage ideation. They then evaluate the impact of incorporating speech transcripts alongside sketches as input to multimodal LLMs for sketch-to-image generation. Results show that adding speech significantly improves the alignment of generated images with designers' self-reported intent, as judged by a reasoning MLLM.
Spontaneous speech paired with sketches dramatically improves multimodal LLMs' ability to generate design images that align with user intent, outperforming sketch-only inputs.
Early-stage design ideation often relies on rough sketches created under time pressure, leaving much of the designer's intent implicit. In practice, designers frequently speak while sketching, verbally articulating functional goals and ideas that are difficult to express visually. We introduce TalkSketchD, a sketch-while-speaking dataset that captures spontaneous speech temporally aligned with freehand sketches during early-stage toaster ideation. To examine the dataset's value, we conduct a sketch-to-image generation study comparing sketch-only inputs with sketches augmented by concurrent speech transcripts using multimodal large language models (MLLMs). Generated images are evaluated against designers' self-reported intent using a reasoning MLLM as a judge. Quantitative results show that incorporating spontaneous speech significantly improves judged intent alignment of generated design images across form, function, experience, and overall intent. These findings demonstrate that temporally aligned sketch-and-speech data can enhance MLLMs' ability to interpret user intent in early-stage design ideation.