Search papers, labs, and topics across Lattice.
This paper introduces MM-slotgate, a novel multimodal slot encoder that factorizes Fashion-CLIP embeddings into four distinct attribute slots, enabling more precise control over fashion retrieval based on specific attributes like color, category, and demographic. By learning independent text-image gates for each slot, the model effectively utilizes visual evidence for attributes such as color while relying on textual information for others, resulting in significant performance improvements. On the H&M dataset, MM-slotgate achieves a macro ConstraintSatisfied@10 score of 0.7566, surpassing traditional methods and demonstrating a remarkable 0.568 absolute gain in color retrieval accuracy.
Attribute-conditioned slots in MM-slotgate allow for unprecedented control in fashion retrieval, achieving a 15.3x improvement for color retrieval alone.
Fashion retrieval often requires satisfying multiple attributes at once, such as category, color, pattern, and demographic. Monolithic embeddings mix these signals into a single vector, making attribute-specific control difficult at retrieval time. Many existing semantic-ID methods provide discrete item codes, but these codes are typically optimized as item-level or residual addresses and do not expose named, independently controllable attribute slots. We introduce MM-slotgate, a multimodal slot encoder that factorizes Fashion-CLIP text and image embeddings into four named attribute slots. Each slot learns its own text-image gate, so visually grounded attributes such as color and pattern can rely more on image evidence, while taxonomy-oriented attributes such as category and demographic can remain more text-driven. On H&M, using a combined slot-similarity and slot-logit retrieval score, MM-slotgate achieves 0.7566 macro ConstraintSatisfied@10, outperforming equal-weight multimodal fusion (0.7142) and fCLIP text-only retrieval (0.4755). The largest gain is on color, which improves from 0.321 to 0.889 (+0.568 absolute), as the learned color gate assigns 57.4% weight to image evidence. The learned gates are interpretable without modality supervision: color is image-leaning, category is text-leaning, and pattern and demographic lie near the middle. The resulting slots also remain controllable: linear probes show no measured excess leakage beyond the label-correlation baseline, and quantized slot codes support targeted intervention, including a 15.3x lift for color. These results suggest that controllable fashion retrieval benefits from typed, attribute-conditioned multimodal slots rather than either a single global embedding or opaque item-level semantic IDs.
