University of Oulu

Alasalmi T., Koskimäki H., Suutala J. and Röning J. (2018). Getting More Out of Small Data Sets - Improving the Calibration Performance of Isotonic Regression by Generating More Data. In Proceedings of the 10th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-275-2, pages 379-386. DOI: 10.5220/0006576003790386

Getting more out of small data sets : improving the calibration performance of isotonic regression by generating more data

Saved in:
Author: Alasalmi, Tuomo1; Koskimäki, Heli1; Suutala, Jaakko1;
Organizations: 1Biomimetics and Intelligent Systems Group, University of Oulu, Oulu, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 0.2 MB)
Persistent link:
Language: English
Published: Science and Technology Publications, 2018
Publish Date: 2018-06-05


Often it is necessary to have an accurate estimate of the probability that a classifier prediction is indeed correct. Many classifiers output a prediction score that can be used as an estimate of that probability but for many classifiers these prediction scores are not well calibrated. If enough training data is available, it is possible to post process these scores by learning a mapping from the prediction scores to probabilities. One of the most used calibration algorithms is isotonic regression. This kind of calibration, however, requires a decent amount of training data to not overfit. But many real world data sets do not have excess amount of data that can be set aside for calibration. In this work, we have developed a data generation algorithm to produce more data from a limited sized training data set. We used two variations of this algorithm to generate the calibration data set for isotonic regression calibration and compared the results to the traditional approach of setting aside part of the training data for calibration. Our experimental results suggest that this can be a viable option for smaller data sets if good calibration is essential.

see all

ISBN: 978-989-758-275-2
DOI: 10.5220/0006576003790386
Host publication: Proceedings of the 10th International Conference on Agents and Artificial Intelligence - (Volume 2), January 16-18, 2018, in Funchal, Madeira, Portugal
Conference: International Conference on Agents and Artificial Intelligence
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
213 Electronic, automation and communications engineering, electronics
Funding: The authors would like to thank Infotech Oulu, Jenny and Antti Wihuri Foundation, and Tauno Tönning Foundation for financial support of this work.
Copyright information: Copyright of the contribution is owned by the publisher, Science and Technology Publications (SCITEPRESS), Published in this repository with the kind permission of the publisher.