Search papers, labs, and topics across Lattice.
This paper introduces $\text{DT}^2$, a novel training paradigm for digital twins (DTs) that focuses on optimizing decision-making by improving policy ranking rather than merely minimizing transition errors. The authors demonstrate that traditional DT training methods can lead to suboptimal policy evaluations, particularly when model capacity is constrained, and empirically validate that $\text{DT}^2$ significantly enhances policy ranking and reduces decision regret across various settings. By leveraging fitted Q-evaluation and an architecture-agnostic loss function, $\text{DT}^2$ maintains high simulation fidelity while outperforming conventional approaches in both seen and unseen policy scenarios.
Training digital twins for decision-making can drastically improve policy ranking and reduce regret, even with limited model capacity.
A digital twin (DT) is a virtual model of a real-world system that can assist decision-making by simulating scenarios induced by different policies. However, typical machine learning-based DTs do not optimise for this use case. We prove that, when model capacity is limited, training DTs to minimise one-step transition errors can produce suboptimal models for ranking sets of policies according to a reward function. We further show that this holds empirically, even with expressive model classes. To address this, we introduce $\text{DT}^2$, a decision-targeted DT training paradigm. Firstly, $\text{DT}^2$ uses fitted Q-evaluation to estimate values of candidate policies from offline data. A DT is then trained to generate rollouts that preserve pairwise policy rankings derived from these proxy ground-truth values with an architecture-agnostic loss function. We empirically demonstrate the efficacy of our method across a range of settings and architectures. $\text{DT}^2$ consistently improves policy ranking and reduces decision regret during policy selection relative to conventional DT training, both for policies used during training and for unseen policies, while maintaining a good level of raw simulation fidelity.
