What is a confusion matrix? The confusion matrix (or “error matrix“) is a table that is used to describe the performance of a classification model by comparing its predictions to a data set of which the true values are known. In a binary classification task, the confusion matrix is a…
What is data leakage? Within the field of machine learning, data leakage is a term used to describe how data from outside the training data set is used to create the model. This is a problem because, within machine learning, our goal is to develop a model that is good…
What is Data Shift? Data shift— or dataset shift, model drift, data drift– is the phenomenon that describes the change in input data in your model (over time), relative to the data it was trained on. It is one of the most common reasons for degrading model accuracy. That’s why…
Performance Metrics Performance metrics tell you something about the performance of a machine learning model. Each metric has a specific focus. Because of the confusion matrix’ nature, a lot of metrics have a close sibling. Equally confusing is that many performance metrics have multiple synonyms, depending on the context. Given…
What is the Accuracy? The Accuracy is a performance metric that tells you the fraction of the predictions that were correct, without distinguishing between positive and negative predictions. The Accuracy can be a very misleading metric when the data set is unbalanced (when the prevalence is either very high or very…
What is Balanced Accuracy? Balanced Accuracy is a performance metric to evaluate a binary classifier. Why not use regular accuracy? Balanced accuracy is a better instrument for assessing models that are trained on data with very imbalanced target variables. I.e. very high, or very low prevalence. This will result in…
What is the Classification Success Index? The Classification Success Index (CSI) is a (fairly uncommon) measure for evaluating classifiers. The CSI focuses exclusively on the positive class. It is calculated as follows: The terms (1-PPV) and (1-TPR) correspond to the proportions of type I and type II errors. The measure…
What is the Diagnostic Odds Ratio? The Diagnostic Odds Ratio (DOR) is a performance metric to assess the effectiveness of a diagnostic test or — within the context of machine learning — a binary classification model. It is defined as the ratio of the odds of the prediction being positive is…
What is the F1-Score? The F1-Score has many names: F-Score F-Measure Sørensen’s Similarity Coefficient Sørensen–Dice Coefficient Dice Similarity Coefficient (DSC) Dice’s Coincidence Index Hellden’s Mean Accuracy Index The F1-Score is a metric to evaluate the performance of a binary classifier. It is calculated as the harmonic mean of the precision…
What is the False Discovery Rate? Given a positive prediction, the False Discovery Rate (FDR) is the performance metric that tells you the probability that the true value is negative. It is closely related to the False Omission Rate, which is completely analogous. The complement of the False Discovery Rate…