Search papers, labs, and topics across Lattice.
This paper presents FilmGPT, an autoregressive transformer model that transforms unwatchable raw footage into coherent cinematic sequences by learning the "grammar" of film from a large corpus of movies. The model employs a novel footage-constrained decoding algorithm to select optimal shots based on learned statistical patterns, achieving superior performance on shot sequence ordering benchmarks and user studies for film editing tasks. Key results indicate that FilmGPT not only outperforms existing generative models but also demonstrates versatility across various video montage applications, including automatic segment trimming and collaborative editing.
FilmGPT redefines video editing by turning chaotic footage into polished sequences without generating new frames, leveraging learned cinematic grammar instead.
This work introduces FilmGPT, an autoregressive transformer designed to address the challenge of video montage – turning a collection of raw, “unwatchable” footage into coherent cinematic sequences. Inspired by language learning in modern LLMs, we train a long-context autoregressive transformer on a large corpus of movies. The aim is to implicitly capture the “grammar” of film directly from data rather than from hand-coded rules. Unlike other generative models, FilmGPT does not generate any new video frames. Instead, at inference time, we introduce a footage-constrained decoding algorithm to select the best next shot from the input raw footage according to the statistical patterns learned from films. We first evaluate these learned statistics directly by using the FilmGPT autoregressive model for next shot prediction on a standard benchmark of shot sequence ordering, outperforming the previous state of the art. We then evaluate our footage-constrained decoding algorithm on the full film editing task via a user study, and find that our FilmGPT-based editing significantly outperforms previous approaches. Finally, we demonstrate the applicability of FilmGPT to a wide range of applications in video montage, from automatic video segment trimming to human-in-the-loop film editing. Please see our supplementary material for qualitative results.
BAIR