C2_W2_Relu Lab - "Why Non-Linear Activations?"

Course Q&A Machine Learning Specialization Advanced Learning Algorithms
1

Quick question:

I’ve been playing with functions that look like this:
f(x) = g(w1*x + b1) + g(w2*x + b2) + g(w3*x + b3)
g(x) = max(0,x) [ReLU function]

(I’m adding 3 functions as an example, think of “one function per segment”)

Is there a way to model the example function given in the sub-section “Why Non-Linear Activations?” in the lab using ReLU?

image

I’m specifically referring to the part where the slope becomes negative, I don’t think it can be done because you’re adding, and ReLU makes all functions that you’re adding always >= 0, it makes me think that you can only increase the slope, or keep it constant, with the consecutive functions that you’re adding, am I missing something?

Thanks!

1 Like
2

Remember each unit has a weight value.
Weights can be negative.
That will switch the slope of that segment to a negative value.

Here’s what I’d try first:
Input the critical points in the target (each endpoint and each change of slope) as the training data.

Train a small NN with linear output using ReLU as the hidden layer activation.

1 Like
3

Hi TMosh, thanks for the reply, and sorry for the late answer!

Yes, weights can be negative but ReLU will ultimately make the entire operation g(w*x + b) >= 0 again so I don’t see a way to subtract from the function to make its slope negative, it seems like you can just:

  • add to the slope if the function that you’re adding (let’s say g(w3*x + b3)) is > 0, or
  • keep it as it is if w3*x + b3 < 0, because g(w3*x + b3) will return:
    max(0, (w3*x + b3))

Also I’m sorry, I’m not sure I understood the exercise you suggested to me, could you explain a bit more? Thank you for taking your time to answer!

1 Like
4

We are not talking about the weight inside g(...) to be negative, we are talking about the one that is outside, meaning the weight in the next layer. For example, we can have -3 * g(w*x + b) where w is a weight in the previous layer and -3 is a weight in this layer.

Cheers,
Raymond