is evaluation based on accuracy of classification algorithms misleading? an approach to model validation using bayes error rate

Researchers have long regarded model accuracy as the primary metric for evaluating the performance of classification algorithms. the current evaluation approach, which relies solely on model accuracy, often leads to inappropriate evaluation of classifiers, regardless of the dataset's separability and complexity. this limitation underscores the need for a new and more comprehensive method. we argue that accuracy-based evaluation can be misleading, even when considering measures of data separability and complexity. we compare the error rates of well-known classifiers on gaussian-generated datasets and show that, paradoxically, many algorithms' observed errors are lower than that of the theoretical optimal classifier, leading to an overestimation of their performance. we consider a model invalid if its error rate is lower than the optimal classifier error, known as the bayes error rate. to identify such invalid models, we introduce a procedure and propose an algorithm for model validation based on the bayes error rate.