Relu Optional Lab Course 2 Week 2

Dear Mentor,

Please help to clarify the below doubts. We are struck and please help.

Doubt 1: How we can arrive values for W1,b1,W2,b2 to match the target ? We are not able to understand how the values actually derived for it ?

Doubt 2: Can someone provide explanation for below ?

The new function is added at transition point but does not contribute to the output prior to that point. The non-linear activation function is responsible for disabling the input prior to and sometimes after the transition points.

Hello @Anbu,

  1. Just by trial-and-error. A key is to look at the small plots on the right hand side for each individual unit.

    Now, only focus on W1 and b1. Make sure we change one thing at a time ;). If we increase W1, the slope of the line will increase. If we increase b1, the position of the transition point will be shifted to the left, however, when b1 is too large, the transition point is disappeared because it is beyond the plot’s frame.

    By now, we should already know adjusting W1 and b1 has what effect. Then we can look at the main plot. The goal here is that we want to adjust W1 and b1 so that orange line overlaps on the blue line. There are 3 sections in the blue lines. The first section is done for us, so we only need to adjust 2 sections out. W1 and b1 will take care of the second section, and W2 and b2 will take care of the third section.

    Try to adjust W1 first, then you should see the slope of some section of the orange line to change accordingly. Adjust W1 until it is parallel to the second section of the blue line. Then adjust b1 until it levels with the second section. Do the same to W2 and b2 to match the third section.

  2. I have shown you the meaning of a transition point. Instead of answering your question directly, I hope to see your understanding first. Here is my suggestion:

    • Try out my steps in (1) to fit the target
    • After getting the perfect fit, further adjust the parameters to understand how they affect each other
    • read your 2nd question again and try to come up with some explanation based on your understanding.

Thanks,
Raymond

Thanks Sir I gone through worked out. I understand from the right graph plot Unit 0,1 and 2 that, for Unit 0 in the x-axis, from 0 till 1, there is slope so we should adjust W0 and adjusted bias for cut-off. Then for Unit 1, from 1 till 2 in the x-axis there is slope so we should adjust W1 and adjust bias for cut off. Again for Unit 2, in the x-axis from 2 till 3, there is slope so increase W2 and adjust bias negative for cutoff.

But what does it mean the following statement says sir ? The slope of the unit, 𝑤2, must be set so that the sum of unit 1 and 2 have the desired slope

Hello @Anbu,

Now that you have the correct set of w1 b1 w2 b2, then from the small plots in the right hand side, you will see this:

I added a table on top to show in which region do each of the units “activated” (i.e. the pink line is > 0, or a > 0).

In the table, the third is the region ( z > 2 ) where both unit 1 and 2 are activated (both with a > 0). Since they are both activated, and they are added to give the final result, the final result is affected by both unit 1 and 2. To get the desired slope in the final result, we therefore need to set w2 correctly.

Cheers,
Raymond

Thanks for the reply sir. I’m able to understand the diagram u drawn but im unable to relate it and cannot understand to the statement ( slope of the unit W2 must be set so that sum of the unit 1 and unit 2 have the desired slope). can you put it in different context pls to understand ?

@Anbu Which unit(s) affect the slope in the circled region? The blue slope is the DESIRED slope.

image

Hi Sir,

Both unit 1 and 2. But what does it mean W2 must be set so that sum of unit 1 unit 2 have the desired slope… can u pls ?

When you open the lab, w2 is initialized to 0. If you do not set w2 to the right value, the result will not match with the desired target. Therefore, w2 must be set (to the right value).

You can understand it as

“I need to set the value of w2 to the correct value, so that unit 1 and unit 2 will give the slope in the circled region as desired.”

1 Like

Thanks Sir i got it now. Below is the picture captured for Unit 1 parameters adjusted only . Here in this pic, Unit 1 somewhat bit of close to the desired slope (third segment in blue colour) To achieve the slope for third segment, W2 must be set correctly which w2 =2 so that Unit 1 & 2 together achieves the desired slope (third segment blue colour). Am I right sir ?

It is close, but it does not achieve the desired slope when x > 2. (The region I have circled in my previous post)

No matter how much time you spend on w1 and b1, you will never get the desired slope in the region when x > 2 (The region I have circled in my previous post). You can only get it if you set something else to w2 and b2. I will not share what w2 and b2 should be.

Raymond

Thanks Sir but my understanding correct ?

yes, except that I am not sure what w2 should be. If your orange curve overlaps with the blue curve, then you have the right w1, b1, w2, and b2.

1 Like

confused: about the third segment and the statement: ."The slope of the unit, 𝑤[1]2, must be set so that the sum of unit 1 and 2 have the desired slope.
"the second segment starts at (1,0) and ends at (2,1): from these two points we get eqn: y=x-1 which does match with the 2nd line segment with w=1 and b=-1
the third segment starts at (2,1) (i guess) and ends at (3,4): if we find an equation it turns out to be y=3x-5 but the slope(w) must actually be 2 and b must be -2 as per the lab
thank you in advance

@Zishan_Siddique

How do you understand the equation in the green box? What is its relation to the target curve?

thank you so much ! think i get it now: please do correct me if i’m wrong
that function is something like a2_0=max(0,-2x+2)+max(0,x-1)+max(0,2x-4)
the reason second segment is not affected by first ReLU function is because it becomes negative after the interval [0,1] but i still don’t understand why does the a1_0(unit 0) affect a1_w(unit 2) [ as unit 0 become 0 in interval (1,infinity)

Hello @Zishan_Siddique

For “a1_w”, I assume you meant “a1_2”.

I agree with you that unit 0 does not affect the third line segment. The third line segment is only affected by the sum of unit 1 and unit 2. Agree?

Raymond

i agree:
reasoning: the third line segment is affected by the sum of unit 1 and unit 2 because unit 1 has positive value in the interval [2,infinity)
thank you for your help

You are welcome, @Zishan_Siddique!

1 Like