University of Oulu

V. Farrahi, M. Niemela, P. Tjurin, M. Kangas, R. Korpelainen and T. Jamsa, "Evaluating and Enhancing the Generalization Performance of Machine Learning Models for Physical Activity Intensity Prediction from Raw Acceleration Data," in IEEE Journal of Biomedical and Health Informatics. doi: 10.1109/JBHI.2019.2917565

Evaluating and enhancing the generalization performance of machine learning models for physical activity intensity prediction from raw acceleration data

Saved in:
Author: Farrahi, Vahid1; Niemelä, Maisa1; Tjurin, Petra1;
Organizations: 1Research Unit of Medical Imaging, Physics and Technology, University of Oulu, Oulu Finland
2Medical Research Center, University of Oulu and Oulu University Hospital, Oulu Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 1.5 MB)
Persistent link: http://urn.fi/urn:nbn:fi-fe2019091828613
Language: English
Published: Institute of Electrical and Electronics Engineers, 2019
Publish Date: 2019-09-18
Description:

Abstract

Purpose: To evaluate and enhance the generalization performance of machine learning physical activity intensity prediction models developed with raw acceleration data on populations monitored by different activity monitors.

Method: Five datasets from four studies, each containing only hip- or wrist-based raw acceleration data (two hip- and three wrist-based) were extracted. The five datasets were then used to develop and validate artificial neural networks (ANN) in three setups to classify activity intensity categories (sedentary behavior, light, and moderate-to-vigorous). To examine generalizability, the ANN models were developed using within-dataset (leave-one-subject-out) cross-validation, and then cross-tested to other datasets with different accelerometers. To enhance the models’ generalizability, a combination of four of the five datasets was used for training and the fifth dataset for validation. Finally, all the five datasets were merged to develop a single model that is generalizable across the datasets (50% of the subjects from each dataset for training, the remaining for validation).

Results: The datasets showed high performance in within-dataset cross-validation (accuracy 71.9–95.4%, Kappa K=0.63–0.94). The performance of the within-dataset validated models decreased when applied to datasets with different accelerometers (41.2–59.9%, K=0.21–0.48). The trained models on merged datasets consisting hip and wrist data predicted the left-out dataset with acceptable performance (65.9–83.7%, K=0.61–0.79). The model trained with all five datasets performed with acceptable performance across the datasets (80.4–90.7%, K=0.68–0.89).

Conclusions: Integrating heterogeneous datasets in training sets seems a viable approach for enhancing the generalization performance of the models. Instead, within-dataset validation is not sufficient to understand the models’ performance on other populations with different accelerometers.

see all

Series: IEEE journal of biomedical and health informatics
ISSN: 2168-2194
ISSN-E: 2168-2208
ISSN-L: 2168-2194
Volume: early access
Issue: early access
Pages: 1 - 12
DOI: 10.1109/JBHI.2019.2917565
OADOI: https://oadoi.org/10.1109/JBHI.2019.2917565
Type of Publication: A1 Journal article – refereed
Field of Science: 114 Physical sciences
Subjects:
Funding: This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 713645, the Ministry of Education and Culture in Finland [grant numbers OKM/86/626/2014, OKM/43/626/2015, OKM/17/626/2016, and OKM/54/626/2019], Infotech Oulu, Finland, and Northern Ostrobothnia Hospital District.
EU Grant Number: (713645) BioMEP - Biomedical Engineering and Medical Physics
Copyright information: © The Authors 2019. This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.
  https://creativecommons.org/licenses/by/3.0/