Search papers, labs, and topics across Lattice.
This paper introduces DIFFCZSL, a novel framework that enhances Compositional Zero-Shot Learning (CZSL) by integrating generative priors from diffusion models into CLIP-based pipelines. By utilizing intermediate diffusion representations as auxiliary supervision for both image and text modalities, the method promotes a richer embedding geometry that captures structured relationships between primitive concepts and their compositions. Extensive experiments reveal that DIFFCZSL consistently outperforms strong CLIP-based baselines across multiple benchmarks, demonstrating the effectiveness of combining generative and discriminative approaches for compositional generalization.
Injecting diffusion representations into CLIP pipelines boosts CZSL performance, revealing the untapped potential of generative models in zero-shot learning tasks.
Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions by leveraging knowledge of primitive concepts learned from seen compositions. Although recent works achieve impressive performance in CZSL by leveraging large vision-language models, they primarily rely on discriminative representations that may not explicitly preserve the structured relationships between primitive concepts and their compositions. Motivated by the recent success of diffusion-based classifiers and their competitive performance relative to discriminative models, we investigate whether intermediate diffusion representations can provide complementary cues for CZSL. To this end, we propose DIFFCZSL, a diffusion-augmented framework that injects generative priors from pre-trained diffusion models into CLIP-based CZSL pipelines. We extract intermediate diffusion representations and project them into the CLIP embedding space to provide auxiliary supervision on both image and text modalities. Through contrastive alignment between CLIP embeddings and diffusion features during training, our method encourages the embedding geometry toward richer composition-aware semantics, while introducing no additional cost at inference time. Extensive experiments on three public CZSL benchmarks demonstrate consistent improvements over strong CLIP-based baselines under both closed-world and open-world settings. Our results highlight the complementary strengths of generative diffusion representations and discriminative vision-language models for compositional generalization.
