Search papers, labs, and topics across Lattice.
This study explored the effectiveness of manual data annotation tasks in teaching students about the subjective nature of labeling in machine learning courses across two universities. By having 43 participants annotate skin lesion images, the researchers found that students significantly improved their understanding of annotation ambiguity, bias, and data quality, with many recognizing the importance of personal interpretation in annotations. The activity was rated as more effective than traditional lectures, highlighting the potential of hands-on experiences to foster critical thinking about AI data and models.
Students learned that annotation disagreement is a valuable insight into the complexity of data interpretation, not merely a flaw in labeling.
Machine learning courses often use pre-labeled datasets, hiding the subjectivity of human annotation. This creates students with an overly trusting view of AI data and models, undervaluing interpretive diversity. We investigated whether manual data annotation tasks teach students about subjective labeling. Study Design: An annotation activity was implemented at two universities: Fontys (Netherlands) and IT University Copenhagen (Denmark). Students annotated skin lesion images for hair coverage on a 3-point scale. Surveys were collected from 43 participants measuring their understanding of annotation ambiguity, data quality, bias, fairness, implementation barriers, and pedagogical effectiveness. Key Findings: Self-reported familiarity with course content increased substantially across all concepts. Most students recognised that personal interpretation affects annotations. Students rated the activity as more effective than traditional lectures for understanding bias. Participants were motivated to learn more. Main Drawbacks: Emotional unease from viewing medical images was the primary issue. Many students still requested clearer guidelines to reduce disagreement, suggesting they hadn't internalised that disagreement from different perspectives is a learning feature, not a bug. Recommendations for Future Iterations: Ensure sufficient interpretive ambiguity in materials. Reduce repetitive annotation workload. Mitigate emotional unease from sensitive content. Explicitly frame disagreement as a learning opportunity rather than a problem to solve. Manual data annotations effectively teach students that human judgment shapes model behavior and that disagreement reflects domain complexity, not just noise.
