Deep Learning Automation Risk: Identifying Object Detection Failure Modes Using Slice-Based Evaluation

Machine learning model evaluations in academic literature tend to fixate on summary performance metrics. However, the performance of a model architecture depends heavily on the context it is evaluated in. Superficial summary performance metrics fail to encapsulate this context and provide readers with virtually no understanding or ability to predict performance in new contexts. Slicing is a type of fine-grained machine learning evaluation, where data is separated into subsets and the performance on each subset is evaluated. Here, we demonstrate slice-based evaluation on a computer vision task, excavator detection in 2D color images. Critical slices are identified using metadata augmentation and feature-space clustering. Slices are created based on features including lighting, excavator color, weather, distance from camera, occlusion, view perspective, number of excavators, presence of other equipment, and environment. Meaningful performance trends identified using slice-based evaluation provide readers with insight about the task’s inherent hardness and training dataset imbalance. Slice-based evaluation should become standard practice in reporting machine learning method results in the academic literature.