Search papers, labs, and topics across Lattice.
The paper introduces SeqLight, a hierarchical deep learning framework that maps music to multi-light Hue-Saturation-Value (HSV) space for automated stage lighting control. It uses a customized SkipBART model to predict the full light color distribution for each frame, then employs hybrid Imitation Learning (IL) to decompose the global color distribution among individual lights. The IL component, trained as a Goal-Conditioned Markov Decision Process (GCMDP) with Hindsight Experience Replay (HER), allows adaptation to diverse venue-specific lighting configurations without professional demonstrations.
Automating stage lighting control across diverse venues is now possible without expert demonstrations, thanks to a novel imitation learning approach that decomposes global color distributions into individual light controls.
Music-inspired Automatic Stage Lighting Control (ASLC) has gained increasing attention in recent years due to the substantial time and financial costs associated with hiring and training professional lighting engineers. However, existing methods suffer from several notable limitations: the low interpretability of rule-based approaches, the restriction to single-primary-light control in music-to-color-space methods, and the limited transferability of music-to-controlling-parameter frameworks. To address these gaps, we propose SeqLight, a hierarchical deep learning framework that maps music to multi-light Hue-Saturation-Value (HSV) space. Our approach first customizes SkipBART, an end-to-end single primary light generation model, to predict the full light color distribution for each frame, followed by hybrid Imitation Learning (IL) techniques to derive an effective decomposition strategy that distributes the global color distribution among individual lights. Notably, the light decomposition module can be trained under varying venue-specific lighting configurations using only mixed light data and no professional demonstrations, thereby flexibly adapting across diverse venues. In this stage, we formulate the light decomposition task as a Goal-Conditioned Markov Decision Process (GCMDP), construct an expert demonstration set inspired by Hindsight Experience Replay (HER), and introduce a three-phase IL training pipeline, achieving strong generalization capability. To validate our IL solution for the proposed GCMDP, we conduct a series of quantitative analysis and human study. The code and trained models are provided at https://github.com/RS2002/SeqLight .
