Learning from multiple annotators took a valuable step towards modeling data that does not fit the usual single annotator setting, since multiple annotators sometimes offer varying degrees of expertise. When disagreements occur, the establishment of the correct label through trivial solutions such as majority voting may not be adequate, since without considering heterogeneity in the annotators, we risk generating a flawed model. In this paper, we generalize GP classification in order to account for multiple annotators with different levels expertise. By explicitly handling uncertainty, Gaussian processes (GPs) provide a natural framework for building proper multiple-annotator models. We empirically show that our model significantly outperforms other commonly used approaches, such as majority voting, without a significant increase in the computational cost of approximate Bayesian inference. Furthermore, an active learning methodology is proposed, which is able to reduce annotation cost even further.
Keywords
Machine Learning, crowdsourcing, gaussian processes, active learning
Subject
Efficient learning from crowdsourcing data
Related Project
Crowds - Understanding urban land use from digital footprints of crowds
Conference
International Conference on Machine Learning (ICML 2014), June 2014
Cited by
Year 2016 : 3 citations
C Long, G Hua, A Kapoor, A joint gaussian process model for active visual recognition with expertise estimation in crowdsourcing, International Journal of Computer Vision, 2016
D Padmanabhan, D Garg, S Shevadeâ?¦, A Robust UCB Scheme for Active Learning in Regression from Strategic Crowds, arXiv preprint arXiv: …, 2016
E Weigl, W Heidl, E Lughofer, T Radauerâ?¦, On improving performance of surface inspection systems by online active learning and flexible classifier updates, Machine Vision and …, 2016
Year 2015 : 3 citations
C Long, G Hua, Multi-class Multi-annotator Active Learning with Robust Gaussian Process for Visual Recognition, Proceedings of the IEEE International Conference …, 2015
N Rohani, P Ruiz, E Besler, R Molinaâ?¦, Variational Gaussian process for sensor fusion, … 2015 23rd European, 2015
M Venanzi, J Guiver, P Kohli, N Jennings, Time-Sensitive Bayesian Information Aggregation for Crowdsourcing Systems, arXiv preprint arXiv: …, 2015
