Supervised machine learning is a general term for machine learning algorithms in which the training data includes both feature values and known label values.
Supervised machine learning is used to train models by determining a relationship between the features and labels in past observations, so that unknown labels can be predicted for features in future cases.
Regression is a form of supervised machine learning in which the label predicted by the model is a numeric value.
Linear Regression.
Poisson Regression.
Regression models are trained to predict numeric label values based on training data that includes both features and known labels.
The process for training a regression model (or indeed, any supervised machine learning model) involves multiple iterations in which you use an appropriate algorithm (usually with some parameterized settings) to train a model, evaluate the model's predictive performance, and refine the model by repeating the training process with different algorithms and parameters until you achieve an acceptable level of predictive accuracy.
Fig. Four key elements of the training process for supervised machine learning model.
Four key elements of the training process for supervised machine learning models:
Split the training data (randomly) to create a dataset with which to train the model while holding back a subset of the data that you'll use to validate the trained model.
Use an algorithm to fit the training data to a model. In the case of a regression model, use a regression algorithm such as linear regression.
Use the validation data you held back to test the model by predicting labels for the features.
Compare the known actual labels in the validation dataset to the labels that the model predicted. Then, aggregate the differences between the predicted and actual label values to calculate a metric that indicates how accurately the model predicted for the validation data.
After each train, validate, and evaluate iteration, you can repeat the process with different algorithms and parameters until an acceptable evaluation metric is achieved.
All of the metrics so far compare the discrepancy between the predicted and actual values in order to evaluate the model.
The calculation for R2 is more complex than for the previous metrics. It compares the sum of squared differences between predicted and actual labels with the sum of squared differences between the actual label values and the mean of actual label values, like this:
R2 = 1- ∑(y-ŷ)2 ÷ ∑(y-ȳ)2
The metrics described above are commonly used to evaluate a regression model. In most real-world scenarios, a data scientist will use an iterative process to repeatedly train and evaluate a model, varying:
Feature selection and preparation (choosing which features to include in the model, and calculations applied to them to help ensure a better fit).
Algorithm selection (We explored linear regression in the previous example, but there are many other regression algorithms)
Algorithm parameters (numeric settings to control algorithm behavior, more accurately called hyperparameters to differentiate them from the x and y parameters).
After multiple iterations, the model that results in the best evaluation metric that's acceptable for the specific scenario is selected.
Classification, like regression, is a supervised machine learning technique; and, therefore, follows the same iterative process of training, validating, and evaluating models. Instead of calculating numeric values like a regression model, the algorithms used to train classification models calculate probability values for class assignment, and the evaluation metrics used to assess model performance compare the predicted classes to the actual classes.
Classification Rate.
Decision Tree.
Logistic Regression.
Naive Bayes Classifiers.
K-Nearest Neighbour.
SVM.
Gaussian Mixture Models.
Binary classification algorithms are used to train a model that predicts one of two possible labels for a single class.
Essentially, predicting true or false. In most real scenarios, the data observations used to train and validate the model consist of multiple feature (x) values and a y value that is either 1 or 0.
To train the model, we'll use an algorithm to fit the training data to a function that calculates the probability of the class label being true (in other words, that the patient has diabetes).
Probability is measured as a value between 0.0 and 1.0, such that the total probability for all possible classes is 1.0.
Ex: if the probability of a patient having diabetes is 0.7, then there's a corresponding probability of 0.3 that the patient isn't diabetic.
Many algorithms can be used for binary classification, such as logistic regression, which derives a sigmoid (S-shaped) function with values between 0.0 and 1.0:
The function produced by the algorithm describes the probability of y being true (y=1) for a given value of x.
Mathematically, can express the function like this: f(x) = P(y=1|x)
The diagram above also includes a horizontal line to indicate the threshold at which a model based on this function will predict true (1) or false (0).
As with regression, when training a binary classification model you hold back a random subset of data with which to validate the trained model.
Based on whether the probability calculated by the function is above or below the threshold, the model generates a predicted label of 1 or 0 for each observation.
We can then compare the predicted class labels (ŷ) to the actual class labels (y)
Confusion matrix
Shows the prediction totals where:
ŷ=0 and y=0: True negatives (TN)
ŷ=1 and y=0: False positives (FP)
ŷ=0 and y=1: False negatives (FN)
ŷ=1 and y=1: True positives (TP)
The arrangement of the confusion matrix is such that correct (true) predictions are shown in a diagonal line from top-left to bottom-right.
Often, color-intensity is used to indicate the number of predictions in each cell, so a quick glance at a model that predicts well should reveal a deeply shaded diagonal trend.
Accuracy
The proportion of predictions that the model got right.
(TN+TP) ÷ (TN+FN+FP+TP)
Recall
Measures the proportion of positive cases that the model identified correctly.
TP ÷ (TP+FN)
Precision
Measures the proportion of predicted positive cases where the true label is actually positive.
TP ÷ (TP+FP)
F1-Score
F1-score is an overall metric that combines recall and precision.
(2 x Precision x Recall) ÷ (Precision + Recall)
Area Under the Curve (AUC)
Another name for the recall is the true positive rate (TPR), and there's an equivalent metric called the false positive rate (FPR) that is calculated as FP÷(FP+TN).
If we change the threshold above which the model predicts true (1), it would affect the number of positive and negative predictions; and therefore change the TPR and FPR metrics. These metrics are often used to evaluate a model by plotting a received operator characteristic (ROC) curve that compares the TPR and FPR for every possible threshold value between 0.0 and 1.0.
The ROC curve for a perfect model would go straight up the TPR axis on the left and then across the FPR axis at the top.
A diagonal line from the bottom-left to the top-right represents the results that would be achieved by randomly guessing a binary label; producing an area under the curve of 0.5.
Multiclass classification is used to predict to which of multiple possible classes an observation belongs.
As a supervised machine learning technique, it follows the same iterative train, validate, and evaluate the process as regression and binary classification in which a subset of the training data is held back to validate the trained model.
We need to use an algorithm to fit the training data to a function that calculates a probability value for each possible class. There are two kinds of algorithms you can use to do this:
One-vs-Rest (OvR) Algorithms
One-vs-Rest algorithms train a binary classification function for each class, each calculating the probability that the observation is an example of the target class.
Each function calculates the probability of the observation being a specific class compared to any other class. For our penguin species classification model, the algorithm would essentially create three binary classification functions:
f0(x) = P(y=0 | x)
f1(x) = P(y=1 | x)
f2(x) = P(y=2 | x)
Each algorithm produces a sigmoid function that calculates a probability value between 0.0 and 1.0. A model trained using this kind of algorithm predicts the class for the function that produces the highest probability output.
Multinomial Algorithms
As an alternative approach is to use a multinomial algorithm, which creates a single function that returns a multi-valued output. The output is a vector (an array of values) that contains the probability distribution for all possible classes - with a probability score for each class which when totaled add up to 1.0:
f(x) =[P(y=0|x), P(y=1|x), P(y=2|x)]
An example of this kind of function is a softmax function, which could produce an output like the following example:
[0.2, 0.3, 0.5]
The elements in the vector represent the probabilities for classes 0, 1, and 2 respectively; so in this case, the class with the highest probability is 2.
Regardless of which type of algorithm is used, the model uses the resulting function to determine the most probable class for a given set of features (x) and predicts the corresponding class label (y).
You can evaluate a multiclass classifier by calculating binary classification metrics for each individual class. Alternatively, you can calculate aggregate metrics that take all classes into account.
The confusion matrix for a multiclass classifier is similar to that of a binary classifier, except that it shows the number of predictions for each combination of predicted (ŷ) and actual class labels (y):
Y Yang, IG Morillo, TM Hospedales, "Deep Neural Decision Trees", arXiv preprint arXiv:1806.06988, 2018
Cameron, A. C.; Trivedi, P. K., Regression analysis of count data, Cambridge University Press. ISBN 978-0-521-63201-0, 1998.
