NYU K12 STEM Education: Machine Learning (Day 2)

Statistics ReviewPermalink

In machine learning, a solid understanding of basic statistical concepts is essential for analyzing data and interpreting model results.

MeanPermalink

The mean, or average, is the sum of all the values in a dataset divided by the number of values. It provides a measure of the central tendency of the data.

Formula: $$\text{Mean}(\mu )=\frac{1}{N}\sum _{i=1}^N{x}_i$$

Example:

For the dataset [2, 4, 6, 8, 10]: $$\mu =\frac{2+4+6+8+10}{5}=6$$

VariancePermalink

Variance measures the spread of the data points around the mean. It indicates how much the data varies from the mean.

Formula: $$\text{Variance}({\sigma }^2)=\frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^2$$

Example:

For the dataset [2, 4, 6, 8, 10]: $${\sigma }^2=\frac{(2-6{)}^2+(4-6{)}^2+(6-6{)}^2+(8-6{)}^2+(10-6{)}^2}{5}=8$$

Mean and Variance VisualizationPermalink

Standard DeviationPermalink

Standard deviation is the square root of the variance. It provides a measure of the spread of the data points in the same units as the data itself.

Formula: $$\text{Standard Deviation}(\sigma )=\sqrt{\text{Variance}}$$

Example:

Using the variance calculated above: $$\sigma =\sqrt{8}\approx 2.83$$

Standard Deviation VisualizationPermalink

CovariancePermalink

Covariance measures the degree to which two variables change together. If the covariance is positive, the variables tend to increase together; if negative, one variable tends to increase when the other decreases.

Formula: $$\text{Covariance}(\text{Cov}(X,Y))=\frac{1}{N}\sum _{i=1}^N(x_i-{\mu }_X)(y_i-{\mu }_Y)$$

Example: For the datasets X = [2, 4, 6] and Y = [3, 6, 9]: $$\text{Cov}(X,Y)=\frac{(2-4)(3-6)+(4-4)(6-6)+(6-4)(9-6)}{3}=6$$

Linear RegressionPermalink

In a Nutshell…Permalink

• Consider a function $y=2x+1$
• Here we introduce a new notation $f(x)=2x+1$
• What this means is that we have a function $f(x)$ which has $x$ as its variable.
• If we have different $x$ values we will have different values of $f(x)$. Example:
  • For $f(x)=2x+1$ and setting $x=1$ we have $f(x)=3$
  • For $f(x)=2x+1$ and setting $x=0$ we have $f(x)=1$
  • For $f(x)=2x+1$ and setting $x=-1.5$ we have $f(x)=-2$
• We believe that dataset are representation of underlying models which can be represented as functions of features.
• For example, we can build a model to forecast weather, we can use the features humidity, current temperature and wind speed to estimate what the temperature will be tomorrow.
• Here we have $f(x)$ representing the tomorrow’s temperature and $x$ being a vector containing humidity, current temperature and wind speed.
• But many times we do have $f(x)$ available, our task here is to figure out what $f(x)$ is using the data available to us.
• Here $f(x)$ is called a model.
• In other words, we want to find a model that fits the data.
• It would be easier to have a ”framework” of the model ready and find the model parameters using the data.
  • $f(x)=w_{1}x+w_0$
  • $f(x)=w_2{x}^2+w_{1}x+w_0$
  • $f(x)=\frac{1}{e^{-(w_{1}x+w_0)}+1}$
• The numbers $w_0$, $w_1$ and $w_2$ are called model parameters.
• We often write the model as $f(x;w)$, stacking all parameters to a vector $w$.

Structure of a datasetPermalink

• In a dataset we have many data.
• We can represent each piece of data as $(x_i,y_i)$, $i=1,2,3,\cdots$
• $x_i$ is called the feature and $y_i$ is called the label.
• The relationship between $x_i$ and $y_i$ and the model $f$ is $f(x_i)=y_i$
• For example, if the weather forecast says it will be 21${}^{\circ }$C (69.8${}^{\circ }$F) if it turns out to be 22${}^{\circ }$C (71.6${}^{\circ }$F) you won’t be yelling at the TV.

How would you fit a line?Permalink

Can you find a line that passes through (0, 0) and (1, 1)?

• The ”framework” of the model is $f(x)=w_{1}x+w_0$
• The data is ($x=0$, $f(x)=y=0$) and ($x=1$, $f(x)=y=1$).
• The process of finding a model to fit the data is to find the values of $w_1$ and $w_0$.

How would you fit a quadratic curve?Permalink

Can you find a quadratic curve that passes through (0, 0), (1, 1) and (−1, 1)?

• The ”framework” of the model is $f(x)=w_2{x}^2+w_{1}x+w_0$
• The data is $(x=0,f(x)=y=0)$, $(x=1,f(x)=y=1)$ and $(x=-1,f(x)=y=1)$
• The process of finding a model to fit the data is to find the values of $w_2$, $w_1$ and $w_0$.

Is Your Model a Good Fit?Permalink

• How would you determine if your model is a good fit or not?
  • How will you determine this?
  • Is there a quantitative way?
• We now introduce a new notation $f(x_i)=\widehat{y_i}$ here the $\hat{\cdot }$ represents $f(x_i)$ is a prediction of $y_i$.

Error FunctionsPermalink

• An error function quantifies the discrepancy between your model and the data.
  • They are non-negative, and go to zero as the model gets better.
• Common Error Functions:
  • Mean Squared Error: $$\text{MSE}=\frac{1}{N}\sum _{i=1}^N||y_i-\widehat{y_i}|{|}^2$$
  • Mean Absolute Error: $$\text{MAE}=\frac{1}{N}\sum _{i=1}^N|y_i-\widehat{y_i}|$$
• In later units, we will refer to these as cost functions or loss functions.

Linear RegressionPermalink

• Linear models: For scalar-valued feature $x$, this is $f(x)=w_{1}x+w_0$
• One of the simplest machine learning model, yet very powerful.

Least Square SolutionPermalink

• Model: $f(x)=w_{1}x+w_0$
• Loss: $$J(w_0,w_1)=\frac{1}{N}\sum _{i=1}^N||y_i-f(x_i)|{|}^2$$
• Optimization: Find $w_0$, $w_1$ such that $J(w_0,w_1)$ is the least possible value (hence the name “least square”).

Using Pseudo-InversePermalink

• For $N$ data points ($x_i,y_i$) we have, $$\begin{array}{rl}\widehat{y_1}& =w_0+w_1{x}_1\\ \widehat{y_2}& =w_0+w_1{x}_2\\ & ⋮\\ \widehat{y_N}& =w_0+w_1{x}_N\end{array}$$
• In matrix form we have, $$\left[\begin{array}{c}\widehat{y_1}\\ \widehat{y_2}\\ ⋮\\ \widehat{y_N}\end{array}\right]=\left[\begin{array}{cc}1& x_1\\ 1& x_2\\ ⋮& ⋮\\ 1& x_N\end{array}\right]\left[\begin{array}{c}{w}_0\\ w_1\end{array}\right]$$
• We can write it as $\hat{Y}=X\times w$. We call $X$ the design matrix.
• We can put the desired labels in matrix form as well: $$Y=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_N\end{array}\right]$$
• Our goal is to minimize the error between $Y$ and $\hat{Y}$ which can be written as $||Y-\hat{Y}|{|}^2$

Multilinear RegressionPermalink

• What if we have multivariate data with $x$ being a vector?

• Example: $$x_i=\left[\begin{array}{c}{x}_{i1}\\ x_{i2}\end{array}\right]$$

  $$\begin{array}{rl}\widehat{y_1}& =w_0+w_1{x}_{11}+w_2{x}_{12}\\ \widehat{y_2}& =w_0+w_1{x}_{21}+w_2{x}_{22}\\ & ⋮\\ \widehat{y_N}& =w_0+w_1{x}_{N1}+w_2{x}_{N2}\end{array}$$

• The model can be written as: $\widehat{y_i}=\left[\begin{array}{ccc}1& x_{i1}& x_{i2}\end{array}\right]\left[\begin{array}{c}{w}_0\\ w_1\\ w_2\end{array}\right]$
• In matrix-vector form: $\left[\begin{array}{c}\widehat{y_1}\\ \widehat{y_2}\\ ⋮\\ \widehat{y_N}\end{array}\right]=\left[\begin{array}{ccc}1& x_{11}& x_{12}\\ 1& x_{21}& x_{22}\\ ⋮& ⋮& ⋮\\ 1& x_{N1}& x_{N2}\end{array}\right]\left[\begin{array}{c}{w}_0\\ w_1\\ w_2\end{array}\right]$
• Solution remains the same $w=(X^{T}X{)}^{-1}{X}^{T}Y$

DemosPermalink

1. Vectorized Programming
2. Plotting Functions
3. Ice-breaker Dataset
4. Linear Regression
5. Multivariable Linear Regression

ReferencesPermalink