Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces Custom ZeroCLIP, a retrieval-augmented vision-language framework specifically designed for zero-shot captioning of traditional Indonesian garments using a dataset of 3,800 expert-annotated images. The model employs a province-level inductive zero-shot protocol, achieving impressive performance metrics such as a CLIPScore of 0.8536 and a 19.3% METEOR gain through retrieval, demonstrating its ability to generate culturally accurate captions without prior exposure to unseen provinces. The findings highlight the potential of this approach for enhancing cultural vocabulary recovery in low-resource heritage contexts, outperforming existing methods in both automated and human evaluations.
Retrieval-augmented zero-shot captioning can significantly enhance cultural accuracy in image analysis, achieving a 19.3% improvement in vocabulary recovery.
This paper presents Custom ZeroCLIP, a retrieval-augmented vision-language framework for zero-shot captioning of Indonesian traditional garments. The dataset contains 3,800 expert-annotated images from all 38 Indonesian provinces. Using a province-level inductive zero-shot protocol, the model is trained on 24 seen provinces, validated on 6 seen provinces, and evaluated on 8 unseen provinces. The framework combines a frozen CLIP ViT-B/32 image encoder, a CLIP text encoder, a BERT text encoder, and an LSTM caption decoder. During inference, unseen-province labels and captions are unavailable, and retrieval uses only captions from training provinces. No unseen-province image, label, or caption is used during training, validation, or retrieval-bank construction. Custom ZeroCLIP achieves a CLIPScore of 0.8536, BLEU-4 of 0.3342, and METEOR of 0.4859, outperforming existing baselines. Ablation results show that retrieval improves cultural vocabulary recovery with a 19.3\% METEOR gain, while human evaluation confirms stronger cultural accuracy and fluency. The results demonstrate the effectiveness of retrieval-augmented domain adaptation for culturally grounded caption generation in low-resource heritage settings. The dataset is publicly available at https://github.com/AnugrahAidinYotolembah/Traditional-Indonesian-Clothing-Captioning-Dataset.