with an input x = (x_1, x_2) and output y (one-dimensional) it is easy to understand/visualize, how the decision boundary looks like, when using logistic regression.
→ its a straight line
Is there a way to understand/predict of how the decision boundaries would look like, when using a deeper network (lets start with one hidden layer with two neurons)?
→ curved line
Can you point me to a good webpage that explains this… would like to get a good intuition of how these hyperplanes arise and form.
Let’s take your example: a single hidden layer with two neurons, followed by an output layer with one more neuron.
To simplify more, let’s use ReLU activations.
Each of those two first-level neurons is going to split the input space by the straight line where the ReLU goes from 0 to positive. One side of the line is zero, the other side is positive.
In the second layer we combine those values, and so in terms of the input space we have two lines, that split the input space into four regions. In one region both inputs are zero. In two regions one input is zero, the other positive, and in the final region both inputs are positive.
With more first-level neurons, there are (many) more regions that combine.
With ReLU activations, the final value that we get to is a piecewise linear (and continuous) function of the inputs. That can help you visualize what it can do.
Week 3 uses planar classification as the assignment to help build some of this intuition. I’d also recommend playing on the Tensorflow Playground to be able to explore more network options and how they split the input area.
Then I like the following (stupid-sounding) exercise. Take a planar classification problem as the one to solve, with the target being a single triangle shape. Now design by hand (no learning algorithm) a neural network (using ReLU I suggest) that will classify the inside vs outside of that triangle. What lines do you you use as the first layer dividing lines? How many first layer neurons do you need?
Try that exrecise: I’m sure you will gain some insight: then you can try to think or code through some variations (more complicated, non-convex shapes; different activations; does a network learn parameters like the ones you designed; does learning handle noisy training data well?)
Another good place to play with different network architectures is definitively TensorFlow playground, where you can change the number and shape different layers and other major hyperparameters. It provides nice illustration that may help you with getting a better intuition on how decision boundaries look like as network complexity changes.
Your triangle example is a good analogy. Just a follow up question. I understand the concept of classifying a triangle’s inside and outside by drawing three (3) boundaries at each side of the triangle. The 3 planes will then enclose the “inside” of the triangle. I’m assuming each layer of the neural net represents one (1) boundary plane or side of the triangle however what is the intuition behind how many neurons we need to create that boundary? Could this be achieved with three layers with one neuron for each layer? Also couldn’t the classification occur with just three (3) linear regression lines? Thank you a ton!
