Search papers, labs, and topics across Lattice.
This paper introduces Web-Scale Multimodal Summarization, a framework that generates summaries by retrieving and combining text and image data from web sources based on a user-defined topic. The system ranks retrieved images using a fine-tuned CLIP model to measure semantic alignment with the topic and retrieved text, and optionally uses BLIP for image captioning to enhance multimodal coherence. Evaluation on image-caption pairs demonstrates strong multimodal alignment, achieving a ROC-AUC of 0.9270, an F1-score of 0.6504, and an accuracy of 96.99%.
Forget monolithic models, this work shows how to build a surprisingly effective multimodal summarizer by smartly composing CLIP, BLIP, and web search.
We introduce Web-Scale Multimodal Summarization, a lightweight framework for generating summaries by combining retrieved text and image data from web sources. Given a user-defined topic, the system performs parallel web, news, and image searches. Retrieved images are ranked using a fine-tuned CLIP model to measure semantic alignment with topic and text. Optional BLIP captioning enables image-only summaries for stronger multimodal coherence.The pipeline supports features such as adjustable fetch limits, semantic filtering, summary styling, and downloading structured outputs. We expose the system via a Gradio-based API with controllable parameters and preconfigured presets.Evaluation on 500 image-caption pairs with 20:1 contrastive negatives yields a ROC-AUC of 0.9270, an F1-score of 0.6504, and an accuracy of 96.99%, demonstrating strong multimodal alignment. This work provides a configurable, deployable tool for web-scale summarization that integrates language, retrieval, and vision models in a user-extensible pipeline.
