El Dehaibi, Macdonald
2020.5.1Proceedings of the Design Society: DESIGN Conference
tlooto Summary
This study investigates inter-rater reliability measures of qualitative annotations for supervised learning with previously annotated product reviews from Amazon where phrases related to sustainability are highlighted and proposes suggestions to designers on measuring reliability of qualitative annotation for machine learning datasets.
Abstract
Abstract An important step when designers use machine learning models is annotating user generated content. In this study we investigate inter-rater reliability measures of qualitative annotations for supervised learning. We work with previously annotated product reviews from Amazon where phrases related to sustainability are highlighted. We measure inter-rater reliability of the annotations using four variations of Krippendorff's U-alpha. Based on the results we propose suggestions to designers on measuring reliability of qualitative annotations for machine learning datasets.
Citation format
DEHAIBI, El; MACDONALD. INVESTIGATING INTER-RATER RELIABILITY OF QUALITATIVE TEXT ANNOTATIONS IN MACHINE LEARNING DATASETS. Proceedings of the Design Society: DESIGN Conference, 2020, 1: 21–30.