Understanding crowd worker behaviors


Understanding crowd worker behaviors

Interaction Design Lab
Level 9, Doug McDonell Building (Building 168)


Paid micro-task crowdsourcing has gained popularity also thanks to the rise of AI because of the convenient way it provides to generate large-scale manually annotated corpora. However, when using crowdsourcing platforms for data gathering purposes, human factors need to be taken into account as humans now become part of (and are not just users of) the system.

In this talk I will discuss our recent research in the area of micro-task crowdsourcing with a focus on understanding crowd worker behaviors and their implication on the quality of the collected data. I will first discuss open challenges in the crowdsourcing ecosystem including issues caused by adversarial approaches that may disrupt the crowdsourcing model as we know it. I will then present our work making use of behavioral data including an analysis of task abandonment and of the strategies employed by expert workers. Finally, I will discuss how human bias is reflected in the data which is being collected by means of crowdsourcing.


  •  Gianluca Demartini
    Gianluca Demartini, Associate Professor