Search papers, labs, and topics across Lattice.
The paper introduces NLI4VolVis, a system enabling natural language interaction with volume visualizations by combining multi-view semantic segmentation, vision-language models, and a multi-agent LLM architecture. This system interprets user intents via function calls to external tools and declarative VolVis commands, controlling a 3D editable Gaussian Splatting engine. The system facilitates open-vocabulary object querying, real-time scene editing, best-view selection, and 2D stylization, demonstrated through case studies and user evaluation.
Forget rigid transfer functions: NLI4VolVis lets you explore and edit volumetric data with natural language, powered by LLMs and editable 3D Gaussians.
Traditional volume visualization (VolVis) methods, like direct volume rendering, suffer from rigid transfer function designs and high computational costs. Although novel view synthesis approaches enhance rendering efficiency, they require additional learning effort for non-experts and lack support for semantic-level interaction. To bridge this gap, we propose NLI4VolVis, an interactive system that enables users to explore, query, and edit volumetric scenes using natural language. NLI4VolVis integrates multi-view semantic segmentation and vision-language models to extract and understand semantic components in a scene. We introduce a multi-agent large language model architecture equipped with extensive function-calling tools to interpret user intents and execute visualization tasks. The agents leverage external tools and declarative VolVis commands to interact with the VolVis engine powered by 3D editable Gaussians, enabling open-vocabulary object querying, real-time scene editing, best-view selection, and 2D stylization. We validate our system through case studies and a user study, highlighting its improved accessibility and usability in volumetric data exploration. We strongly recommend readers check out our case studies, demo video, and source code at https://nli4volvis.github.io/.
