University of Oulu

Niels van Berkel, Jorge Goncalves, Lauri Lovén, Denzil Ferreira, Simo Hosio, Vassilis Kostakos, Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports, International Journal of Human-Computer Studies, Volume 125, 2019, Pages 118-128, ISSN 1071-5819,

Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports

Saved in:
Author: van Berkel, Niels1; Goncalves, Jorge1; Lovén, Lauri2;
Organizations: 1The University of Melbourne
2University of Oulu
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.8 MB)
Persistent link:
Language: English
Published: Elsevier, 2018
Publish Date: 2020-12-03


The Experience Sampling Method is widely used to collect human labelled data in the wild. Using this methodology, study participants repeatedly answer a set of questions, constructing a rich overview of the studied phenomena. One of the methodological decisions faced by researchers is deciding on the question scheduling. The literature defines three distinct schedule types: randomised, interval-based, or event-based (in our case, smartphone unlock). However, little evidence exists regarding the side-effects of these schedules on response rate and recall accuracy, and how they may bias study findings. We evaluate the effect of these three contingency configurations in a 3-week within-subjects study (N=20). Participants answered various objective questions regarding their phone usage, while we simultaneously establish a ground-truth through smartphone instrumentation. We find that scheduling questions on phone unlock yields a higher response rate and accuracy. Our study provides empirical evidence for the effects of notification scheduling on participant responses, and informs researchers who conduct experience sampling studies on smartphones.

see all

Series: International journal of human-computer studies
ISSN: 1071-5819
ISSN-E: 1095-9300
ISSN-L: 1071-5819
Volume: 125
Pages: 118 - 128
DOI: 10.1016/j.ijhcs.2018.12.002
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This work is partially funded by the Academy of Finland (Grants 276786-AWARE, 286386-CPDSS, 285459-iSCIENCE, 304925-CARE), the European Commission (Grant 6AIKA-A71143-AKAI), and Marie Skłodowska-Curie Actions (645706-GRAGE).
EU Grant Number: (645706) GRAGE - Grey and green in Europe: elderly living in urban areas
Academy of Finland Grant Number: 276786
Detailed Information: 276786 (Academy of Finland Funding decision)
286386 (Academy of Finland Funding decision)
285459 (Academy of Finland Funding decision)
304925 (Academy of Finland Funding decision)
Copyright information: © 2018. This manuscript version is made available under the CC-BY-NC-ND 4.0 license