University of Oulu

M. Otani, R. Togashi, Y. Nakashima, E. Rahtu, J. Heikkilä and S. Satoh, "Optimal Correction Cost for Object Detection Evaluation," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 21075-21083, doi: 10.1109/CVPR52688.2022.02043.

Optimal correction cost for object detection evaluation

Saved in:
Author: Otani, Mayu1; Togashi, Riku1; Nakashima, Yuta2;
Organizations: 1CyberAgent, Inc.
2Osaka University
3Tampere University
4University of Oulu
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 13.4 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2022
Publish Date: 2023-01-24


Mean Average Precision (mAP) is the primary evaluation measure for object detection. Although object detection has a broad range of applications, mAP evaluates detectors in terms of the performance of ranked instance retrieval. Such the assumption for the evaluation task does not suit some downstream tasks. To alleviate the gap between downstream tasks and the evaluation scenario, we propose Optimal Correction Cost (OC-cost), which assesses detection accuracy at image level. OC-cost computes the cost of correcting detections to ground truths as a measure of accuracy. The cost is obtained by solving an optimal transportation problem between the detections and the ground truths. Unlike mAp, OC-cost is designed to penalize false positive and false negative detections properly, and every image in a dataset is treated equally. Our experimental result validates that OCscost has better agreement with human preference than a ranking-based measure, i.e., mAP for a single image. We also show that detectors’ rankings by OC-cost are more consistent on different data splits than mAP. Our goal is not to replace mAP with OC-cost but provide an additional tool to evaluate detectors from another aspect. To help future researchers and developers choose a target measure, we provide a series of experiments to clarify how mAP and OC-cost differ.

see all

Series: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
ISSN: 1063-6919
ISSN-E: 2575-7075
ISSN-L: 1063-6919
ISBN: 978-1-6654-6946-3
ISBN Print: 978-1-6654-6947-0
Pages: 21075 - 21083
DOI: 10.1109/cvpr52688.2022.02043
Host publication: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022)
Conference: IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
Copyright information: © 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.