MachineLearning-MultiLinearRegressionWithGradientDescent

Description:

• Python script to estimate coefficients for multilinear regression using gradient descent algorithm.
• Linear regression implemented from scratch.
• Using simulated data of job prospects given AI and machine learning skills.

Given,

$$ y = W^T X + \epsilon $$

Where:
y is the target,

$w_0$ is the intercept (bias value)

W is a vector parameters (weights) to be estimated.

$$ \begin{align*} W &= \begin{bmatrix} w_{0} \\ w_{1} \\ w_{2} \\ \vdots \\ w_{N} \end{bmatrix} \end{align*} $$

X is a matrix of 1's and K feature weights and N data points of given inputs

$$ \begin{align*} X &= \begin{bmatrix} 1&x_{12}&\cdots &x_{1K} \\ 1&x_{22}&\cdots &x_{2K} \\ \vdots & \vdots & \ddots & \vdots\\ 1&x_{N2}&\cdots &x_{NK} \end{bmatrix} \end{align*} $$

and $\epsilon$ is a vector of estimation errors denoted

$$ \begin{align*} \epsilon &= \begin{bmatrix} \epsilon_{1} \\ \epsilon_{2} \\ \vdots \\ \epsilon_{N} \end{bmatrix} \end{align*} $$

• The loss function chosen is minimum mean square error given by:

$$ \begin{equation*}\label{eq:MultipleLinearRegressionCostFunction} C(W) = \frac{1}{N} \sum^{N}_{n=1}(( W^T X_n)-y_n)^2 \end{equation*} $$

• With partial derivatives

$$ \begin{align*} \frac{\partial C}{\partial w_0} &= -\frac{2}{N} \sum_{n=1}^{N} ((W^T X_n ) - y_n) \\ \frac{\partial C}{\partial w_i} &= -\frac{2}{N} \sum_{n=1}^{N} x_i((W^T X_n - y_n)) \end{align*} $$

• With weight updates given by:

$$ \begin{equation*} w_{n} = w_{n-1} - \alpha \frac{\partial C}{\partial w_{n-1}} \end{equation*} $$

• Where $\alpha$ is the "learning weight".

How to use

python mulitpleLinearRegression.py

Expected Output

=======================================================================
MULTI LINEAR REGRESSION USING GRADIENT DESCENT TERMINATION RESULTS
=======================================================================
Initial Weights were:             0.0, 0.0, 0.0.
   With initial cost:          3281.9.
        # Iterations:       2,500,000.
       Final weights:    w0:+24.94, w1:+0.32, w2:+0.483.
          Final cost:            +8.1.
                RMSE:            +4.0, R-Squared:         +0.7
=======================================================================
Finished

Requirements

Python (>2.7), Numpy and Pandas.