Manual Gradient Descent with Linear Regression

---

Work with your assigned partner to fit linear regression to this MPG dataset using gradient descent. We will be continuing the activity from last time, modifying your code to handle multiple features. This time we will be using all the features rather than just simply the horsepower feature. Make a copy/backup of your code from last time for safe keeping. You should be able to minimally modify your code from last time to complete this activity.

Setup

The setup is similar to last time, but notice that w_1 has been replaced.

1. Load the data with pandas and split out the features (just horsepower) from mpg
2. Create a variable alpha = .00005, this the "learning rate" i.e. step size of the update to the parameters.
3. Create a variable w_0 set it to 0. This the model intercept, it function the same as last time.
4. Create a variable w_vec and set it to a one dimensional numpy array (a vector) of 0's. The length of the array should match the number of features in the data. Take a look at numpy.zeros().

Predictions & Error

The linear regression model is \( \hat{y}_i = w_0 + w_1 x_{i,i} + w_2 x_{i,2} + ... + w_c x_{i,c} \). Note that \( c \) represents the number of spread sheet columns/features/explanatory variables. Also the subscript \( i \) indicates the row/instance index i.e. a prediction is made for one example/row.

The model can be rewritten using a linear algebra operation, the dot product. \( \hat{y}_i = w_0 + w_1 x_{i,1} + w_2 x_{i, 2} + ... + w_c x_{i, c} = w_0 + \vec{w} \cdot \vec{x}_i \). Recall that vectors are just lists of numbers, so the dot product between two vectors means to multiply first values out of each list/vector, the second pair, the third pair, and so on, then sum up all the products.

But this can be further optimized with linear algebra! The model can be applied to the entire dataset with one operation. The entire dataset (all rows and columns) can be contained in a 2D numpy array which is also called a matrix. Let's call the matrix of all the data \( X \) (note that it is capitalized to denote it is a matrix). When a matrix is multiplied by a vector (in that order) the result is the application of the dot product of the first row and the vector, the dot product of second row and the vector, and so on. If the prediction for a single row/example is \( \hat{y}_i = \vec{x}_i \vec{w} +w_0 \) then this can be generalized with matrix multiplication to \( \hat{\vec{y}} = X \vec{w} + w_0 \). Note that \( \hat{\vec{y}} \) is the vector (list) of all the predictions of MPG for the whole dataset. Look at numpy.matmul to do matrix multiplication, but pay attention to the order of the arguments because \( X\vec{w} \not = \vec{w}X \). Also note that you don't technically have to use matmul, you could use a for-loop, numpy will automatically parallelize this operation for you!

Calculating the residuals is still straight forward, if you called the predictions y_hat then you can measure the error (residuals):
res = y - y_hat.

Manual SGD

Start with weights of one, then update the weights manually by using the gradient descent update:
\( w_{\text{new}} = w_{\text{old}} - \alpha \nabla(w) \)

Where the gradient (first derivatives) are
\( \nabla(w_0) = \frac{-2}{n} \sum\limits_{i=1}^{n} [y_i - \hat{y}_i] = \frac{-2}{n} \sum\limits_{i=1}^{n} res_i \)

\( \nabla(w_1) = \frac{-2}{n} \sum\limits_{i=1}^{n} x_{i,1} [y_i - \hat{y}_i] = \frac{-2}{n} \sum\limits_{i=1}^{n} x_{i,1} res_i \)

...

\( \nabla(w_c) = \frac{-2}{n} \sum\limits_{i=1}^{n} x_{i,c} [y_i - \hat{y}_i] = \frac{-2}{n} \sum\limits_{i=1}^{n} x_{i,c} res_i \)

Notice that \( x_{i,m} res_i \) is the same for every column (m). The numpy operation X * res will multiply all the values in a row in X by the corresponding value in res e.g. the new first row will be \( x_{1,1}r_1, \quad x_{2,1} r_1, \quad ..., \quad x_{c,1} r_1 \), the second row will be \( x_{1,2}r_2, \quad x_{2,2} r_2, \quad ..., \quad x_{c,2} r_2 \)

\( \nabla(w_m) \) effectively averages out the entire row for column \( m \). Use np.mean() but with axis=0 to implement this.

Test this code by running the program to calculate the gradient, then manually update the weights (variables), and repeat.

Gradient Descent

---

Replace the manually fitting process with a while loop. Try different termination conditions for the loop e.g. fixed number of updates, or if the weights do not change much from the previous iteration.

Compare

Compare your implementation versus sci-kit learn's version.

---

1. GFU
2. CSIS
3. J. Walker Orr
4. CSIS 441