University of Oulu

Jonas Oppenlaender, Elina Kuosmanen, Andrés Lucero, and Simo Hosio. 2021. Hardhats and Bungaloos: Comparing Crowdsourced Design Feedback with Peer Design Feedback in the Classroom. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 570, 1–14. DOI:

Hardhats and bungaloos : comparing crowdsourced design feedback with peer design feedback in the classroom

Saved in:
Author: Oppenlaender, Jonas1; Kuosmanen, Elina1; Lucero, Andrés2;
Organizations: 1University of Oulu, Oulu, Finland
2Aalto University, Espoo, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 1.2 MB)
Persistent link:
Language: English
Published: Association for Computing Machinery, 2021
Publish Date: 2021-05-17


Feedback is an important aspect of design education, and crowdsourcing has emerged as a convenient way to obtain feedback at scale. In this paper, we investigate how crowdsourced design feedback compares to peer design feedback within a design-oriented HCI class and across two metrics: perceived quality and perceived fairness. We also examine the perceived monetary value of crowdsourced feedback, which provides an interesting contrast to the typical requester-centric view of the value of labor on crowdsourcing platforms. Our results reveal that the students (N = 106) perceived the crowdsourced design feedback as inferior to peer design feedback in multiple ways. However, they also identified various positive aspects of the online crowds that peers cannot provide. We discuss the meaning of the findings and provide suggestions for teachers in HCI and other researchers interested in crowd feedback systems on using crowds as a potential complement to peers.

see all

ISBN Print: 978-1-4503-8096-6
Pages: 1 - 14
Article number: 570
DOI: 10.1145/3411764.3445380
Host publication: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Conference: ACM SIGCHI Annual Conference on Human Factors in Computing Systems
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
Funding: We thank the students of the HCI course and the workers from Amazon Mechanical Turk who contributed their feedback in this study. This research is partially enabled by the GenZ strategic profiling project at the University of Oulu, supported by the Academy of Finland (project number 318930).
Academy of Finland Grant Number: 318930
Detailed Information: 318930 (Academy of Finland Funding decision)
Copyright information: © 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems,