Search papers, labs, and topics across Lattice.
This paper introduces a Dual-Transformer architecture with Cross-Attention for multi-camera view recommendation, significantly enhancing the task of selecting appropriate camera angles in video editing. By decoupling the temporal encoding of past frames from the evaluation of candidate camera views, the model achieved a Precision@0.5 score of 56.60%, surpassing the previous state-of-the-art of 37.16%. Additionally, the use of a SwinV2 backbone allowed for further improvements, and the model demonstrated effective adaptation to individual editing styles with minimal fine-tuning data.
A novel Dual-Transformer architecture boosts multi-camera view recommendation performance, achieving a 56.60% Precision@0.5 and enabling personalized editing style adaptation with just 20% of the video for training.
Multi-camera systems are foundational to modern media production, and multi-camera editing is a critical task. This involves the proper selection of the appropriate camera view at each moment. In this paper, we propose a novel Dual-Transformer architecture with Cross-Attention that heavily outperformed the current SOTA models over the TVMCE dataset (TV Shows Multicamera Editing dataset). Our model decouples these tasks: (1) a dedicated temporal encoder first processes the sequence of past frames to build a rich memory of the recent history, and (2) the candidate camera views then act as queries to this memory via a cross-attention module, allowing each candidate to independently interrogate the historical context and find the most relevant information for its own evaluation. Our approach achieved 56.60% Precision@0.5, representing a substantial improvement over the prior best result of 37.16%. We further conducted an ablation study exploring the use of lightweight backbone architectures, where the SwinV2 backbone yielded the best performance, achieving 69.65% Precision@0.5. Using this best-performing configuration, we then investigated the feasibility of adapting the model to replicate the editing style of a specific human editor. To this end, we fine-tuned the model using varying proportions of the initial segment of a target video. Our results demonstrate that even with only 20% of the video used for fine-tuning, the model exhibited measurable improvements in Precision@0.5, indicating strong potential for data-efficient personalization of editing style adapted to each individual TV show or producer.
