Model Performance Metrics

Model performance metrics are quantitative measures used to assess how well a machine learning model is performing in making predictions. Depending on the type of task (classification, regression, etc.), different metrics can be applied. Below are some commonly used performance metrics along with examples for clarity.

Classification Metrics

1. Accuracy

Definition: The ratio of correctly predicted instances to the total instances.

Formula:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Example: In a binary classification problem where 70 out of 100 predictions are correct, the accuracy is 70%.

2. Precision

Definition: The ratio of true positive predictions to the total predicted positives.

Formula:

Precision = TP / (TP + FP)

Example: If a model identifies 50 positive cases (TP) and 10 false positives (FP), the precision is 83.3%.

3. Recall (Sensitivity)

Definition: The ratio of true positive predictions to the actual positives.

Formula:

Recall = TP / (TP + FN)

Example: If there are 50 actual positives and the model correctly identifies 40 of them, recall is 80%.

4. F1 Score

Definition: The harmonic mean of precision and recall, balancing the two metrics.

Formula:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

Example: If precision is 0.833 and recall is 0.8, then the F1 score is approximately 0.815.

5. AUC-ROC (Area Under the Curve - Receiver Operating Characteristic)

Definition: Measures the ability of a model to distinguish between classes. The ROC curve is a graphical representation of the true positive rate vs. false positive rate.

Example: An AUC of 0.9 indicates excellent model performance, while an AUC of 0.5 indicates no discrimination capability (like random guessing).

Regression Metrics

1. Mean Absolute Error (MAE)

Definition: The average of the absolute errors between predicted and actual values.

Formula:

MAE = (1/n) * Σ |yi - ŷi|

Example: If predictions are [3, 4, 2.5] and actuals are [2.5, 4, 3], the MAE is 0.5.

2. Mean Squared Error (MSE)

Definition: The average of the squared differences between predicted and actual values.

Formula:

MSE = (1/n) * Σ (yi - ŷi)²

Example: For the same predictions and actuals, MSE would be approximately 0.1667.

3. R-squared (Coefficient of Determination)

Definition: Indicates the proportion of the variance in the dependent variable that is predictable from the independent variables.

Formula:

R² = 1 - (SSres / SStot)

Example: If your model explains 80% of the variance in the outcome variable, it would have an R-squared of 0.8.

Summary

• Classification metrics focus on how well a model can classify instances correctly.
• Regression metrics assess the accuracy of predictions made by a regression model.