Search papers, labs, and topics across Lattice.
This paper details how Netflix enhanced its personalized promotional asset selection through the use of multimodal embeddings, addressing the limitations of traditional ID-based models that fail to leverage asset content. By employing a two-tower model augmented with CLIP image embeddings, Netflix achieved significant improvements in cold-start performance while consolidating five separate models into one for artwork personalization. Additionally, the introduction of MediaFM, a tri-modal foundation model that integrates visual, audio, and timed-text signals, demonstrated superior performance in video preview personalization compared to traditional visual-only approaches, validated through both offline and online testing.
Multimodal embeddings not only streamline asset personalization at Netflix but also deliver a dramatic boost in performance for cold-start scenarios and video previews.
Personalized promotional assets, namely artwork images and video preview clips, are critical to content discovery on Netflix. Traditional models for asset selection rely on ID-based interaction history, leaving them blind to asset content and unable to serve newly launched titles and assets. We describe how multimodal embeddings reshaped production systems at Netflix and report transferable lessons for practitioners adopting foundation-model embeddings into recommender systems. First, pretrained image embeddings unlock cross-title, cross-canvas knowledge transfer. Augmenting a two-tower model with CLIP image embeddings lets a single model serve all five Netflix artwork canvas types, replacing five separately trained per-canvas models and substantially improving cold-start performance. A lightweight extension reuses CLIP's joint text-image space to make artwork personalization query-aware in search. Second, multimodality decisively beats any single modality for video preview personalization. We describe MediaFM, our in-house tri-modal foundation model trained on a large-scale corpus of shots from the Netflix show catalog, fusing visual (SeqCLIP), audio (wav2vec 2.0), and timed-text signals; adopted for video preview personalization, it outperforms strong visual-only baselines both offline and in online A/B tests. Third, a simple offline proxy task whose performance correlates with online outcomes can accelerate the experimentation and productization cycle. Predicting the popularity-based winner from embeddings alone ranks embedding models and versions, pruning the choice space before any end-to-end integration or A/B test; it now gates every new MediaFM checkpoint. We also share the production engineering decisions (shared embedding infrastructure, low-latency serving, cheap screening) that made these deployments viable, along with the design tradeoffs and failure modes we encountered.
