👉 Note: Reading: Hands-On ML - Chap 3: Classification (section “Confusion Matrices”)

Confusion matrix

• True Positive (TP): what we predict Positive is really Positive.
• True Negative (FN): what we predict Negative is really Negative.
• False Negative (FN): what we predict Negative is actually Positive.
• False Positive (FP): what we predict Positive is actually Negative.

This guy is pregnant?

How to remember?

• True/False indicates what we predicted is right/wrong.
• Positive/Negative is what we predicted (yes or no).

Type I / Type II errors

• FP = Type I error = rejection of true null hypothesis = negative results are predicted wrongly = what we predict positive is actually negative.
• FN = Type II error = non-rejection of a false null hypothesis = positive results are predicted wrongly = what we predict negative are actually positive.

Why CM is important?

Give a general view about our model, "is it really good?" thanks to precision and recall!

Precision & Recall

• Precision: How many of our positive predictions are really true? (Check the accuracy of our positive predictions).

  $$ \mathrm {precision}= \dfrac{\mathrm{true\, positive}}{\mathrm{positively\, predicted\, results}}= \dfrac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}. $$

• Recall: How many of positive results belong to our predictions? (Do we miss some negative predictions?)

  $$ \mathrm {recall}= \dfrac{\mathrm{true\, positive}}{\mathrm{positively\, actual\, results}}= \dfrac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}. $$

Recognizing number 5. Figure taken from this book.

When to use?

• Precision is importantly used when the "wrongly predicted yes" (FP) influences much (e.g. This email is spam? — results yes but actually no and we lost important emails!).
• Recall (Sensitivity) is importantly used when the "wrongly predicted no" (FN) influences much (e.g. In the banking industry, this transaction is fraudulent? — results no but actually yes and we lost money!).