Crowdsourcing marketplaces have emerged as an effective tool for high-speed, low-cost labeling of massive data sets. Since the labeling accuracy can greatly vary from worker to worker, we are faced with the problem of assigning labeling tasks to workers so as to maximize the accuracy associated with their answers. In this work, we study the problem of assigning workers to tasks under the assumption that workers’ reliability could change depending on their workload, as a result of, e.g., fatigue and learning. We offer empirical evidence of the existence of a workload-dependent accuracy variation among workers, and propose solution procedures for our Crowdsourced Labeling Task Assignment Problem, which we validate on both synthetic and real data sets.

(2017). A workload-dependent task assignment policy for crowdsourcing [journal article - articolo]. In WORLD WIDE WEB. Retrieved from http://hdl.handle.net/10446/229302

A workload-dependent task assignment policy for crowdsourcing

Coniglio, Stefano;
2017

Abstract

Crowdsourcing marketplaces have emerged as an effective tool for high-speed, low-cost labeling of massive data sets. Since the labeling accuracy can greatly vary from worker to worker, we are faced with the problem of assigning labeling tasks to workers so as to maximize the accuracy associated with their answers. In this work, we study the problem of assigning workers to tasks under the assumption that workers’ reliability could change depending on their workload, as a result of, e.g., fatigue and learning. We offer empirical evidence of the existence of a workload-dependent accuracy variation among workers, and propose solution procedures for our Crowdsourced Labeling Task Assignment Problem, which we validate on both synthetic and real data sets.
articolo
Catallo, Ilio; Coniglio, Stefano; Fraternali, Piero; Martinenghi, Davide
(2017). A workload-dependent task assignment policy for crowdsourcing [journal article - articolo]. In WORLD WIDE WEB. Retrieved from http://hdl.handle.net/10446/229302
File allegato/i alla scheda:
File Dimensione del file Formato  
WWWJ2017-CatalloConiglioFraternaliMartinenghi.pdf

Solo gestori di archivio

Versione: publisher's version - versione editoriale
Licenza: Licenza default Aisberg
Dimensione del file 2.02 MB
Formato Adobe PDF
2.02 MB Adobe PDF   Visualizza/Apri
Pubblicazioni consigliate

Caricamento pubblicazioni consigliate

Aisberg ©2008 Servizi bibliotecari, Università degli studi di Bergamo | Terms of use/Condizioni di utilizzo

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10446/229302
Citazioni
  • Scopus 6
  • ???jsp.display-item.citation.isi??? 4
social impact