Mahalanobis distance based accuracy prediction models for Sentinel-2 Image Scene Classification

Abstract

Over the years, due to the enrichment of paired-label datasets, supervised machine learning has become a prime component of any problem-solving. Examples include building classifiers for applications such as image/speech recognition, traffic prediction, product recommendation, virtual personal assistant (VPA), online fraud detection and many more. The performance of these developed classifiers is highly dependent upon the training dataset, and subsequently, without human intervention or true labels, the evaluation over unseen observations remains unknown. Using a statistical distance researchers did try to assess the model’s goodness-of-fit and compared multiple independent models. Nonetheless, given a train-test split and different classifiers built over the training set, the question ‘is it possible to find a prediction error using the relation between training and test set?’ remains unsolved. In this article, we propose a generalized statistical distance-based method measuring the prediction uncertainty at a new query point. To be specific, we propose a Mahalanobis distance-based Evidence Function Model to measure the misclassification caused by K-Nearest Neighbours (KNN), Extra Trees (ET), and Convolutional Neural Network (CNN) models when classifying Sentinel-2 image into six scene classes (Water, Shadow, Cirrus, Cloud, Snow, Other). The performance of the proposed method was assessed over two different datasets: (i) the test set, with an overall mean prediction uncertainty detection of 62.99%, 29.80% and 31.51%, leading to a mean micro-F1 performance of 67.89%, 39.30%, and 38.29% for KNN, ET, and CNN, respectively; (ii) a water-body set, with prediction uncertainty detection of 22.27%, 42.08%, and 27.67%, leading to a micro-F1 performance of 34.70%, 58.96%, and 43.32%, respectively.