Hi all,
I thought it would be a good exercise for me to summarize what I have learned in MLS Course 1. You might find this helpful to so I encourage you to share your own summary. Maybe we can give each other feedback.
I will post this as the first post in the topic and then begin working on my summary.
it looks good so far, @jesse
thanks for summarising @jesse
I would add a bit on improving the model:
underfit (high bias) → train with larger network (more nodes or layers), or try different algorithms, or train longer, etc…
overfit (high variance) → add regularizations, or more data can help
Well, looks like I cannot edit my first post each time since it times out so I will delete and repost as needed?
In MLS Course 1 I learned about two kinds of supervised machine learning problems, (1) linear regression and (2) classification. Here, supervised means that the learning algorithm had labels provided.
univariate linear regression means only one input feature x
General Notation — f_{w,b}(x)=wx+b
Training Set Prediction Notation — f_{w,b}(x^{(i)})=wx^{(i)}+b
multiple linear regression means multiple input features \begin{bmatrix} x_1 & x_2 & \dots & x_n \end{bmatrix} where n is the number of features
General Notation — f_{w,b}(x_1, x_2, ..., x_n)=w_1x_1+w_1x_1+...+w_nx_n+b
Vectorized Notation — f_{\vec{w},b}(\vec{x})=\vec{w} \cdot \vec{x} + b
Training Set Prediction Notation — f_{\vec{w},b}(\vec{x^{(i)}})=\vec{w} \cdot \vec{x^{(i)}} + b
The goal here is to fit a line or curve to a dataset (i.e., model) that can be used to predict “reasonable” values for new values of x (in the context of univariate) or \vec{x} (in the context of multiple).
With this goal in mind, the question is, “How do we accomplish this goal?”
Note: From here on out I will only reference multiple linear notations.
In the context of our linear regression problem, we use the squared error cost function which looks like this:
J(\vec{w},b) = \frac{1}{2m} \sum\limits_{i = 1}^{m} (f_{w,b}(x^{(i)}) - y^{(i)})^2
This can be summarized as:
The cost of our model given specific values for the parameters w and b
It is important to note that this cost is in the context of all outcomes of the model on the training set.
Minimizing the cost of our model means minimizing our cost function J(\vec{w},b). In other words, trying to get our model to do the very best it can at predicting “acceptable” values for new values of \vec{x}
Gradient descent is a method of iterating over values for the parameters of the model until a specific goal is satisfied. That goal is convergence. Convergence here means “at a local minimum cost” where the parameters no longer change much with each additional step.
Gradient descent works well with convex functions. The squared error cost function produces this shape.
The parts of the gradient descent equation that stick out to me are:
Once the goal of gradient descent is satisfied (i.e., convergence) then that model with those specific parameters is considered ready to produce “acceptable” predictions.
When your features \vec{x} are of different scale (e.g., x^{(i)}_1=1000 while x^{(i)}_2=.01) gradient descent can take a long time to converge. Scaling these features can help overcome this problem. The three common ways to do this are:
So you have a model, you have a cost function, you minimize the cost function using gradient descent, and your predictions are not that great… Now what?
is when a model has high bias
is when a model has high variance
The process of reducing the impact of lesser input features on predictions without removing any of them from your learning algorithm. This is applied at the level of the cost function which in turn has an impact on the calculation of your gradient during gradient descent.
Note: You do not need to apply regularization on the parameter b.
\frac{\lambda}{2m}\sum\limits_{j = 1}^{m}w_j^{2}
where if,
\lambda= close to 0, the model will underfit (basically eliminating the parameters \vec{w}
or if,
\lambda= large number, the model will overfit
J(\vec{w},b) = \frac{1}{2m} \sum\limits_{i = 1}^{m} (f_{w,b}(x^{(i)}) - y^{(i)})^2+\frac{\lambda}{2m}\sum\limits_{j = 1}^{m}w_j^{2}
\frac{\partial J(\vec{w},b)}{\partial w_j} = \frac{1}{m} \sum\limits_{i = 1}^{m} (f_{\vec{w},b}(\vec{x}^{(i)}) - y^{(i)})x_{j}^{(i)} + \frac{\lambda}{m} w_j
This is a valid approach when your y is binary, 0 or 1. This methodology uses the sigmoid function which produces values between 0 and 1.
Sigmoid function → g(z)=\frac{1}{1+e^{-z}}
We use this function by applying it to our existing linear model where z=f_{\vec{w},b}(\vec{x}) = \vec{w}\vec{x} + b or vectorized as \vec{w} \cdot \vec{x} + b. So logistic regression looks like,
Logistic regression → f_{\vec{w},b}(\vec{x}) = g(\vec{w} \cdot \vec{x} + b) = \frac{1}{1+e^{-(\vec{w} \cdot \vec{x} + b)}}
The above equation produces values between 0 and 1 which can be thought of as:
The probability y will be 1
This reality is not really binary though, at least not until we apply something called the decision boundary which is a heuristic threshold (a custom rule). The most common rule is for values of f_{\vec{w},b}(\vec{x}) \geq 0.5 predict 1 else 0.
We happen to know that f_{\vec{w},b}(\vec{x}) \geq 0.5 when z \geq 0 (i.e., \vec{w} \cdot \vec{x} + b \geq 0). This is better understood in a picture which won’t be drawn here.
Something important to consider here is that we need a cost function that captures the unique characteristics of our prediction model outputting only 0 or 1 AND satisfies that results of the cost function create a “convex” function. Unfotunately, sqaured error cost function does not work here (it creates many local minima). This introduces a new concept of loss function which will be combined with a similar (not same) cost function as squared error cost function.
The new cost function looks like this,
J(\vec{w},b) = \frac{1}{m} \sum\limits_{i = 1}^{m} \left[ loss(f_{\vec{w},b}(\vec{x}^{(i)}), y^{(i)}) \right]
The loss function is applied for each iteration i of the training set and looks like this,
loss(f_{\vec{w},b}(\vec{x}^{(i)}), y^{(i)}) = -y^{(i)} \log\left(f_{\vec{w},b}\left( \vec{x}^{(i)} \right) \right) - \left( 1 - y^{(i)}\right) \log \left( 1 - f_{\vec{w},b}\left( \vec{x}^{(i)} \right) \right)
While this looks overwhelming, think about our specific cases for y, that is, when it is equal to 0 or 1. This function penalizes the model heavily when it is further from the truth (e.g., y^{(i)}=1 and the model predicts 0).
Other concepts from linear regression carry over like feature scaling, regularization, and even gradient descent is represented the same for parameters w_j. However, while gradient descent is represented the same way, remember that the cost function behind the scenes and f_{\vec{w},{b}}(\vec{x}^{(i)}) is actually the sigmoid function.
I will come back and finish this.
Hi @jesse,
I think it would be better to say, in an originally overfit model, setting lambda close to zero won’t help defeating overfitting and the model stays overfit, and with a lambda that is too large, the model would become underfit.
Well said.
I enjoy reading this summary. Thank you for sharing it.
Cheers,
Raymond
Hi I am new to machine learning.
“is when a model has high bias”
Explain the above line.
Hi @jesse,
This comment is wrong. No summation is missing. I have removed it from my above reply.
Raymond
It looks like you left out the 3rd method of Feature Scaling - z-score normalization:
https://www.coursera.org/learn/machine-learning/lecture/akapu/feature-scaling-part-2
Thanks for sharing your summary, @jesse ! Very helpful.
I am the same like you – I made a summary, was also thinking of sharing it here with others in case they’d find it helpful, and then I saw your post!
The summary touches on key topics from the course. Outside of serving as reminders, they will also help me never lose sight of the fundamentals. You can find my summary here.
The notes are nice and pretty practical oriented. Thank you for sharing them with us, @christiney! 
Cheers,
Raymond
Thanks, @rmwkwok ! Excited to make progress with the second course and keep learning!
