Input and output mapping the relu function assignment is not clear to me

[tbhaxor]

Lets start with second segment here

z_1 = w_1 x+ b_1 \\ a_1 = \begin{cases} z_1, \quad z_1 > 0 \\ 0, \quad z_1 \le 0 \end{cases}

Based on this if x=1 then the z_1 = 1 \cdot 1 + (-1) \implies 0, \quad \therefore a_1 = 0, so the function line would roughly be like this

Now in case of a_2

z_2 = w_2 x+ b_2 \\ a_2 = \begin{cases} z_2, \quad z_2 > 0 \\ 0, \quad z_2 \le 0 \end{cases}

let takes x=1 , z_2 = 1 \cdot 2 - 4 \implies a_2 = 0, the line would roughly be

Now here comes the doubt, the black dotted line works fine where cutoff happens a t x=1, but in the red dotted line the cutoff at x=2 gives us y =1, it is not setting it to 0. Am i missing something?

Just to confirm, the first layer will divide data like this

[tbhaxor]

@rmwkwok @paulinpaloalto I believe in you help me answer this question

[rmwkwok]

@tbhaxor

image913×647 31.7 KB

Hello @tbhaxor,

The above graph is the result of a(x) = a_0(x) + a_1(x) + a_2(x), so you need to consider all three of them for every x. Give it a try and you should get consistent answers with the graph!

Cheers,
Raymond

[tbhaxor]

What are weightes of a_0 unit? I need to try this practically

[rmwkwok]

the weight is -2, and the bias is 2

[tbhaxor]

But please read my question again,i think this not what I asked

[rmwkwok]

are you asking for w0 and b0?

[tbhaxor]

No I am confused at cut off of w2 and b2

[rmwkwok]

You didn’t compute it correctly because you were only computing a_2(x=2), instead of a(2) = a_0(2) + a_1(2) + a_2(2). What is a_0(2) + a_1(2)?

[tbhaxor]

why this, a^{[1]}_2 will be just \vec X \cdot \vec W_2 + b_2, weights of first layer always take input from the data

[rmwkwok]

I don’t understand your last post.

This is from the notebook:

The following graph is a_0^{[2]}

@tbhaxor

[tbhaxor]

Ohk, before this i have another question.

is a^{[2]} = \sum_{i=0}^2 a^{[i]} mean composition of function? According to me yes, because it is taking some functions and returning another function?

I could be wrong.

[rmwkwok]

Hello @tbhaxor,

If I expand the summation sign in this equation from you:

This is what I will get: a^{[2]} = a^{[0]} + a^{[1]} + a^{[2]}, do you agree? If so, I notice that the subject in the left hand side appears in the third term of the right hand side. Therefore, I do not understand this equation.

Would you mind to rephrase your question but using the symbols defined in this screenshot, if we are still discussing this I assume? If so, this can save your time defining the meaning of each symbol you use in your question.

Cheers,
Raymond

[tbhaxor]

I tried to create the neural net with the following code

import matplotlib.pyplot as plt
import numpy as np

X = np.linspace(-50, 50, (600)).reshape(-1, 1)
w = np.array([ [-2 , 1, 2]])
b = np.array([2, -1, -4])

Y = X @ w + b
A1 = np.maximum(Y, 0)
c = ["blue", "red", "orange"]
plt.subplots_adjust(hspace=0.8, wspace=0.4)
plt.figure(figsize=(20, 8))
for i in range(Y.shape[1]):
    ax = plt.subplot2grid((2, 6), (0, i))
    ax.plot(X, A1[:, i], color=c[i])
    ax.set_title("Plot of $a^{[1]}_{%d}$" % i)

ax = plt.subplot2grid((2, 6), (0, 3), colspan=3)
ax.set_title("Plot of all $\\mathbf{a^{[1]}}$")
for i in range(Y.shape[1]):
    ax.plot(X, A1[:, i], color=c[i])

plt.subplots_adjust(hspace=0.5, wspace=0.4)
A2 = np.sum(A1, axis=1)
ax = plt.subplot2grid((2, 6), (1, 0), colspan=6)
ax.set_title("Plot of $\\mathbf{a^{[2]}}$")
ax.plot(X, A2)

which got me this graph, I can not obtain the third abrupt pivot at x=2 could you look the code above?

[rmwkwok]

Hello @tbhaxor,

They won’t look similar because the following graph

has an x-axis scaled from 0 to 3 and y-axis from 0 to 4, but yours is totally different.

I found that the lab uses this np.linspace(0, 3, 3*100) to initialize X. To reproduce the lab’s result, it is easier if sticking with the lab’s setting. The lab’s notebook doesn’t show a lot of source code, but you might always click “File” > “Open” (on the menu bar of the notebook) to browse and open the source code which is imported by the lab.

Cheers,
Raymond

[tbhaxor]

Oh yeah I totally forgot that decision boundary depends on input feature as well

With np.linspace(0, 3, 3*100) I am able to create the same graph. Thanks for your guidance @rmwkwok

[rmwkwok]

No problem. It is good to see that you can reproduce the results!

Raymond

[tbhaxor]

So we can say that the weights in a^{[2]} is 1 for all the inputs it received from a^{[1]}

[rmwkwok]

That is a yes!

Raymond