C2_W2_Relu Lab - "Why Non-Linear Activations?"

Quick question:

I’ve been playing with functions that look like this:
`f(x) = g(w1*x + b1) + g(w2*x + b2) + g(w3*x + b3)`
`g(x) = max(0,x) [ReLU function]`

(I’m adding 3 functions as an example, think of “one function per segment”)

Is there a way to model the example function given in the sub-section “Why Non-Linear Activations?” in the lab using ReLU?

I’m specifically referring to the part where the slope becomes negative, I don’t think it can be done because you’re adding, and ReLU makes all functions that you’re adding always `>= 0`, it makes me think that you can only increase the slope, or keep it constant, with the consecutive functions that you’re adding, am I missing something?

Thanks!

1 Like

Remember each unit has a weight value.
Weights can be negative.
That will switch the slope of that segment to a negative value.

Here’s what I’d try first:
Input the critical points in the target (each endpoint and each change of slope) as the training data.

Train a small NN with linear output using ReLU as the hidden layer activation.

1 Like

Hi TMosh, thanks for the reply, and sorry for the late answer!

Yes, weights can be negative but ReLU will ultimately make the entire operation `g(w*x + b) >= 0` again so I don’t see a way to subtract from the function to make its slope negative, it seems like you can just:

• add to the slope if the function that you’re adding (let’s say `g(w3*x + b3)`) is `> 0`, or
• keep it as it is if `w3*x + b3 < 0`, because `g(w3*x + b3)` will return:
`max(0, (w3*x + b3))`

Also I’m sorry, I’m not sure I understood the exercise you suggested to me, could you explain a bit more? Thank you for taking your time to answer!

1 Like

We are not talking about the weight inside `g(...)` to be negative, we are talking about the one that is outside, meaning the weight in the next layer. For example, we can have `-3 * g(w*x + b)` where `w` is a weight in the previous layer and `-3` is a weight in this layer.

Cheers,
Raymond