Search papers, labs, and topics across Lattice.
This paper introduces the Modality-Invariant and Fair Representation (MIFR) framework for skin disease classification, addressing the dual challenges of modality reliance and performance disparities across skin tones. By employing ViT-based encoders and a multi-objective loss function that incorporates fairness and modality alignment, the model achieves competitive predictive performance and fairness metrics on diverse datasets. Notably, t-SNE visualizations demonstrate the effective geometric alignment of clinical and dermoscopic embeddings, confirming the framework's efficacy in learning modality-invariant representations.
Achieving competitive skin disease classification while ensuring fairness across skin tones, MIFR aligns clinical and dermoscopic data in a shared embedding space.
Skin diseases represent a major global public health burden, yet machine learning tools developed to assist in their diagnosis suffer from two critical limitations: reliance on only one modality for diagnosis and systematic performance disparities across skin tones. While existing approaches address each challenge separately, this work proposes a modality-invariant framework with fair representation (MIFR) for skin disease classification. The architecture pairs clinical photographs with dermoscopic images using ViT-based encoders, projecting each input into a high-dimensional embedding space via modality-specific projection heads. The resulting model is trained with a five-component multi-objective loss including weighted cross-entropy for classification, confusion and skin-type classification losses for fairness, per-modality supervised contrastive loss for class alignment, and a modality-invariance loss for clinical and dermoscopic modality alignment. Experiments on the HIBA+Derm7pt paired dataset and the external PAD-UFES-20 and ISIC 2019 datasets showed that modality-invariant representation learning provides competitive predictive performance compare to relevant baseline models and competitive fairness on the internal dataset. t-SNE visualizations confirmed that clinical and dermoscopic embeddings of the same disease are geometrically aligned, validating the joint objectives.
