COMP4702 Lecture 2

General Concept

• Want to create a model that can map some input space into some output space
• Assume that we have some data that provides examples of input-output mappings known as the training set and denoted as

• We often assume that there are multiple inputs (features) as a vector for each data-point. The vector $\bf x$ has $p$ data-points

• For now, consider the case where there is a single scalar output
  • If the output $y$ is categorical, we have a classification problem
  • If the output $y$ is continuous, we have a regression problem
• The number of classes in a classification problem is denoted by $M$.
  • Often discuss $M=2$ which is the binary classification problem in which the classes are typically denoted as $\lbrace 0, 1 \rbrace$ or $\lbrace -1, 1 \rbrace$ depending on the problem

Example - Classifying Songs

Figure 1 - Length vs Perceived Energy of Songs

• In this example, we want to classify the energy of a song based on it's length
• Note how the x axis of the graph is $\ln s$ - this is a common trick applied in machine learning for inputs that vary in size over several orders of magnitude.

• It is evident from the graph that in general:
  • Kiss songs are more energetic
  • Bob Dylan songs are longer
  • Beatles songs are shorter
• However, the boundaries between these different classes are not clear.
  • Therefore, oftentimes achieving 100% accuracy in machine learning is not possible.
  • In this example, the length and energy features are not enough to precisely predict the song's energy
  • We could add other features in the hope that there is some correlation between (some of) those features and the song's energy

Example - Car Stopping Distances

Figure 2 - Car Stopping Distances Example

• We now consider the car stopping distances dataset which contains 62 samples
• The dataset has two variables:
  • Speed: The speed of the car when the brake signal is given
  • Distance: The distance travelled after the signal is given, until the car has stopped
• We can clearly see that there is a trend in the data - the fastest the car, the longer the stopping distance
• We could create a model / function for this regression problem - given some speed, what is the expected stopping distance?
  • Important to note that we don't care too much about the performance of the model on the training set
  • What matters is the ability of the model to generalise to data that was not used for training

$k$-Nearest Neighbours

• One of the simplest models
• The output or prediction is determined by looking at the labels or true values of nearby points in the training set.
  • This involves some sort of distance metric.

$k$-Nearest Neighbours for Classification

• Our data is as follows:

Figure 3  - Table of Values for KNN Classifier Example

• We can plot the data to visualise it as shown in Figure 4.
• To predict the output for some data-point ${\bf{x}_\star}=\begin{bmatrix}1&2\end{bmatrix}^T$, we compute the distance from that data-point to every other data-point
• We choose the $k$ closest data-points, and take a majority vote (in the case of classification) or average (in the case of regression) to determine the output value
• For $k=1$ in the classification problem, we simply just choose the class of the closest neighbour - in this case $i=6$ has class red, so ${\bf{x}_\star}=\begin{bmatrix}1&2\end{bmatrix}^T$ is red.

Figure 4 - KNN Classifier for k=1 and k=3

$k$-Nearest Neighbours Pseudocode

Data: Training data, denoted $\{\bf{x}_ i, y_ i\}_ {i=1}^{n}$ with test input denoted $\bf{x_ \star}$

Result: Predicted test output, $\hat{y}(\bf{x_ \star})$

1. Compute the distances $||\bf{x_ i}-\bf{x_ \star}||_2$ for all training points $i=1, \cdots, n$
2. Let $\mathcal{N}_ \star=\lbrace i:\bf{x}_ i \text{ is one of the }k \text{ data points closest to }\bf{x_ \star}\rbrace$
3. Compute the prediction $\hat{y}(\bf{x_ \star})$ using the following formula:

• $k$ Is a parameter that must be chosen by the user
  • When $k=1$, the model is highly sensitive to local variations in the data
  • When $k$ has a large value, it is less sensitive to local variations but can lose the ability to adapt to details in the data - boundaries are a bit smoother
• We want to choose a value of $k$ that is a compromise between the two - doesn't over-fit and doesn't under-fit.

Figure 5 - KNN Decision Boundary for varying k-values

• We can also see the effect of changing $k$ on the music dataset below:

Figure 6 - KNN Decision Boundary for varying k-values

$K$-Nearest Neighbours for Regression

• We can apply the $k$-Nearest Neighbours technique to regression problem as well.
• For each x-value, we compute the average of all data-points that exist within a range on the input axis.

Figure 7 - KNN Regression Problem

Decision Trees

• Conceptually, applying a series of if-then rules to partition the input space.
• To make a prediction, start at the root of the try and apply the decision rule at each point to decide what branch to traverse down'

Figure 7 - KNN Regression Problem

Learning a Regression Tree

• The prediction $\hat{y}({\bf{x_\star }})$ for a regression tree is a piecewise constant function of the input $\bf{x_\star}$ denoted

• Essentially, to learn a decision tree, we need to learn the threshold values at each level of the tree
• If our $x$ values (input values) are continuous in nature, there are infinitely many sets of threshold values for our regression tree
• However, the only time a prediction (on the training set) will change is when a threshold crosses a data point
  • Even still, the search space is intractable

• The standard algorithm used to compute the regression tree is a greedy algorithm
• To get the predicted value for regression, we take the average of the training data-points that fall into a given region

Recursive Binary Splitting Algorithm

• At each split, we can only choose one of the $p$ input variables $x_1, \cdots, x_p$
• We split a given region $R$ into two sub-regions denoted as:

• For each training data-point $({\bf{x}}_i, y_i)$, we can compute a prediction error by first determining which region the data-point falls in, and then computing the difference between $y_i$ and the constant prediction associated for that region
• The computation for the SSE of all training points is denoted as:

• We want to minimise the difference between the predicted value and the actual value / ground truth
  • For this reason, it makes sense to derive some sort of function that compares them
  • This is precisely what the The Sum of Squared Errors (SSE) metric measures - low value if similar, high value if different.
• Using the SSE metric, the training problem is given as trying to minimise the SSE.

• To solve the machine learning problem using decision trees, we compute the split in between each of the values (all possible splits as shown by the dotted lines in the figure below) and compute its SSE
  • We first compute the splits for the root node of the tree and choose the one with the lowest SSE
  • Then "fix" that split and compute the splits for its children.
  • Repeat until the depth of the tree reaches the desired limit $L$.
  • We can alternatively stop the algorithm's growth by setting a minimum number of data-points per leaf node (e.g., each leaf node must contain a minimum of 5 data-points.)

Regression Tree Algorithm

Goal: Learn a decision tree using recursive binary splitting:
Data: Training data $\mathcal{T}=\{\bf{x}_i, y_i\}_{i=1}^n$
Result: Decision tree with regions $R_1, \cdots, R_L$ and corresponding predictions $\hat{y}_1, \cdots, \hat{y}_L$

1. Let $R$ denote the whole input space
2. Compute the regions $(R_1, \cdots, R_L)=\text{Split}(R, \mathcal{T})$
3. Compute the predictions $\hat{y}_\ell$ for $\ell\in\{1, \cdots, L\}$ as:
   $$\hat{y}_\ell=\begin{cases} \text{Average}\{y_i:\bf{x}_i \in R \ell\} & \text{(Regression Problems)}\\ \text{MajorityVote}\{y_i : \bf{x}_i\in R\ell \} &\text{(Classification Problems)}\\ \end{cases}$$

Function Split(R, 𝒯):
    if (stopping criterion fulfilled) 
        return r
    else
        Go through all possible splits xⱼ < s for all input variables
          j = 1, ..., p
        Pick the pair (j,s) that minimises the loss function for regression/classification problems
        Split the region R into R₁ and R₂ according to Eq 2.4
        Split data 𝒯 into 𝒯₁ and 𝒯₂ respectively 
        return Split(R₁, 𝒯₁), Split(R₂, 𝒯₂)

Classification Tree Algorithm

• This is very much the same for the application of the decision tree to the classification problem
• However, we must derive a new "loss function" for this categorical data
  • Popular approaches include the misclassification rate, Gini index and entropy.

• For example, we can construct a classification tree using the misclassification rate as our loss function

• We introduce the following equation to denote the proportion of training observations in the $\ell$th region to belong to the $m$th class

  • Note that in this equation $\mathbb{I}$ denotes the identity function, which returns 1 if $y_i=m$ and otherwise returns 0.
    $$\hat{\pi}_{\ell m}=\frac{1}{n_\ell} \sum_{i : \bf{x}_i\in R_\ell} \mathbb{I}\{y_i=m\}$$

• We can then define the splitting criterion $Q_\ell$ based on these class proportions

  • Misclassification rate (proportion of data-points in region $R_\ell$ which do not belong to the most common class)
    $$Q_\ell = 1 -\max_m \hat{\pi}_{\ell m}\tag{2.7a}$$
  • Gini index:
    $$Q_\ell = \sum_{m=1}^{M} \hat{\pi}_{\ell m} (1 - \hat{\pi}_{\ell m}) \tag {2.7b}$$
  • Entropy Criterion
    $$Q_\ell=-\sum_{m=1}^M \hat{\pi}_{\ell m} \ln \hat{\pi}_{\ell m} \tag{2.7c}$$

• Why are there these different classification algorithms?
  • Introduce the concept of node purity
  • A node is more pure if it assigns more of the same class to every data point that "comes its way"
  • That is, nodes with higher purity can lead to fewer total splits in total
  • The Gini Index and Entropy loss functions tend to be better than the misclassification rate.

• We can "stop training" a model by limiting the depth of the tree
• The effect of implementing this limit below for the classification problem

• We can also show the effects on a regression problem