Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper presents APOLLO, a hybrid framework for personalized object rearrangement that integrates a lightweight personalized embedding model (PEM) with selective large language model (LLM) assistance to address the challenges of cluttered and partially erroneous environments. By training PEM on a small number of user-specific demonstrations, the framework produces uncertainty estimates that guide when to invoke LLM reasoning, thereby enhancing efficiency and privacy. Experimental results on the newly introduced APOR dataset and existing benchmarks show that APOLLO outperforms prior LLM-based approaches while significantly reducing LLM usage, highlighting its practical applicability in real-world scenarios.
APOLLO not only enhances object rearrangement efficiency but also intelligently decides when to leverage LLM assistance, striking a balance between performance and resource usage.
Robotic assistance in household environments requires not only predicting where objects should be placed, but also reasoning about when objects should not be placed at all. Existing approaches to personalized object rearrangement primarily focus on placement decisions under the assumption of clean observations and complete actionability, limiting their applicability in realistic, cluttered, and partially erroneous settings. In this paper, we introduce APOLLO, a hybrid framework for abstention-aware personalized object rearrangement that combines a lightweight, personalized embedding model (PEM) with selective large language model (LLM) assistance. PEM is trained for each user-environment pair using a small number of demonstrations, operates entirely on CPU, and produces uncertainty estimates, which are used to selectively invoke LLM-based reasoning only for ambiguous decisions, balancing efficiency, privacy, and reasoning capability. To evaluate this formulation beyond existing benchmarks, we introduce APOR, a synthetic, LLM-generated dataset that captures room-level, multi-furniture environments, diverse organizational profiles, explicit abstention behavior, and noisy partial scene context. Extensive experiments on both PARSEC and APOR provide initial evidence that APOLLO improves over prior LLM-based baselines in controlled benchmark settings while substantially reducing LLM usage. Code is available at https://github.com/PaInt-Lab/APOLLO.