Manual Gradient Descent with Linear Regression

---

Work with your assigned partner to fit linear regression to this MPG dataset using gradient descent. Only use the horsepower feature.

Setup

We are going building up to implement gradient descent for Linear Regression but first we need to do a few steps:

1. Load the data with pandas and split out the features (just horsepower) from mpg
2. Create variables w_0 and w_1 and set them both to 1. These are the parameters of the model, the intercept and slope respectively
3. Create a variable alpha = .00005, this the "learning rate" i.e. step size of the update to the parameters.

Predictions & Error

Make predictions for all the vehicles (rows) in the dataset. To do this, y_hat = w_1 x + w_0 where x is the vector of vehicle horsepower. After this, measure the error (residuals):
res = y - y_hat.

Manual SGD

Start with weights of one, then update the weights manually by using the gradient descent update:
\( w_{\text{new}} = w_{\text{old}} - \alpha \nabla(w) \)

Where the gradient (first derivatives) are
\( \nabla(w_0) = \frac{-2}{n} \sum\limits_{i=1}^{n} [y_i - w_0 - w_1 x_i] = \frac{-2}{n} \sum\limits_{i=1}^{n} [y_i - \hat{y}_i] = \frac{-2}{n} \sum\limits_{i=1}^{n} res_i \)

\( \nabla(w_1) = \frac{-2}{n} \sum\limits_{i=1}^{n} x_i [y_i - w_0 - w_1 x_i] = \frac{-2}{n} \sum\limits_{i=1}^{n} x_i [y_i - \hat{y}_i] = \frac{-2}{n} \sum\limits_{i=1}^{n} x_i res_i \)

To implement this, try np.mean().

Test this code by running the program to calculate the gradient, then manually update the weights (variables), and repeat.

Gradient Descent

---

Replace the manually fitting process with a while loop. Try different termination conditions for the loop e.g. fixed number of updates, or if the weights do not change much from the previous iteration.

Compare

Compare your implementation versus sci-kit learn's version.

---

1. GFU
2. CSIS
3. J. Walker Orr
4. CSIS 441