Understanding crowd worker behaviors
Interaction Design Lab
Level 9, Doug McDonell Building (Building 168)
Paid micro-task crowdsourcing has gained popularity also thanks to the rise of AI because of the convenient way it provides to generate large-scale manually annotated corpora. However, when using crowdsourcing platforms for data gathering purposes, human factors need to be taken into account as humans now become part of (and are not just users of) the system.
In this talk I will discuss our recent research in the area of micro-task crowdsourcing with a focus on understanding crowd worker behaviors and their implication on the quality of the collected data. I will first discuss open challenges in the crowdsourcing ecosystem including issues caused by adversarial approaches that may disrupt the crowdsourcing model as we know it. I will then present our work making use of behavioral data including an analysis of task abandonment and of the strategies employed by expert workers. Finally, I will discuss how human bias is reflected in the data which is being collected by means of crowdsourcing.
Gianluca Demartini, Associate Professor
University of Queensland
Dr Gianluca Demartini is an Associate Professor in Data Science at the University of Queensland, School of Information Technology and Electrical Engineering. His main research interests are Information Retrieval, Semantic Web, and Human Computation. His research has been supported by the Australian Research Council (ARC), the UK Engineering and Physical Sciences Research Council (EPSRC), and by the EU H2020 framework program. He received Best Paper Awards at the AAAI Conference on Human Computation and Crowdsourcing (HCOMP) in 2018 and at the European Conference on Information Retrieval (ECIR) in 2016 and the Best Demo Award at the International Semantic Web Conference (ISWC) in 2011. He has published more than 100 peerreviewed scientific publications at venues such as WWW, ACM SIGIR, VLDBJ, ISWC, and ACM CHI.