University of Oulu

Niels van Berkel, Jorge Goncalves, Simo Hosio, Zhanna Sarsenbayeva, Eduardo Velloso, Vassilis Kostakos, Overcoming compliance bias in self-report studies: A cross-study analysis, International Journal of Human-Computer Studies, Volume 134, 2020, Pages 1-12, ISSN 1071-5819,

Overcoming compliance bias in self-report studies : a cross-study analysis

Saved in:
Author: van Berkel, Niels1; Goncalves, Jorge2; Hosio, Simo3;
Organizations: 1University College London, United Kingdom
2The University of Melbourne, Australia
3University of Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 1.7 MB)
Persistent link:
Language: English
Published: Elsevier, 2020
Publish Date: 2021-10-10


A popular methodology used for in situ observations is the Experience Sampling Method (ESM), in which participants intermittently answer short questionnaires. We analyse a set of recent ESM studies and find substantial differences in the number of collected responses across participants. These differences amount to ’compliance bias’, as the experiences of responsive participants skew the results. Our work develops ways for researchers to ensure the collection of an adequate number of responses across participants. Through a cross-study analysis of ESM studies, we construct a model that describes the effect of contextual, routine, and study-specific factors on participants’ response rate. In addition to previous work, which aims to maximise the number of total responses, this work also aims to achieve a more equal distribution of responses between participants. In order to achieve this goal, we analyse which contextual cues can be personalised to achieve a higher response rate. Our results highlight a number of factors that have a strong effect on participants’ response rate and can guide the design of future experiments.

see all

Series: International journal of human-computer studies
ISSN: 1071-5819
ISSN-E: 1095-9300
ISSN-L: 1071-5819
Volume: 134
Pages: 1 - 12
DOI: 10.1016/j.ijhcs.2019.10.003
Type of Publication: A1 Journal article – refereed
Field of Science: 213 Electronic, automation and communications engineering, electronics
Copyright information: © 2019 Elsevier Ltd. All rights reserved. This manuscript version is made available under the CC-BY-NC-ND 4.0 license