# Binary classification - Simplest classification task

**URL:** <https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676>\
**Category:** Neural Networks and Deep Learning\
**Tags:** coursera-platform\
**Created:** [July 23, 2023, 5:48pm UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676 "2023-07-23T17:48:09Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Michal\_Majk\_Ritcherd](https://avatars.discourse-cdn.com/v4/letter/m/5daacb/32.png) [@Michal\_Majk\_Ritcherd](https://community.deeplearning.ai/u/Michal_Majk_Ritcherd)\
**Post date:** [July 23, 2023, 5:48pm UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/1 "2023-07-23T17:48:09Z")

</div>

I’ve created for myself the most basic classification task.

The input has 2 training examples with one feature as [[-1], [1]] and the outputs are [1, 0].  
I’ve created Neural Network with 1 neuron with Linear activation function and the model’s uses Binary crossentropy loss.

Here’s the code of the described example.

```auto
features = np.array([[-1], [1.]], dtype=np.float16)
labels = np.array([1., 0.], dtype=np.float16)

model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(1,)),
    tf.keras.layers.Dense(1, activation='linear', dtype=tf.float16)
])

model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-2),
    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
    metrics='accuracy'
)

model.fit(features, labels, None, 100000)

```

If I am correct, this task should be fairly easy to train to 100%, meaning I can get loss equal to 0 (altough accuracy is 100%). But this model is unable to overfit the training data to have 0 loss (or predict probability exactly 1 for ‘-1’ and 0 for ‘1’).  
Am I missing something? Or why doesn’t it converge to 0?

If I make from this task Linear regression task by switching the loss from BinaryCrossentropy to MeanSquaredError, the model then converges to zero.

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [July 24, 2023, 12:15am UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/3 "2023-07-24T00:15:37Z")

</div>

A single dense layer with no hidden layer isn’t really an NN. It is just regression.

Try it without the logits, try different learning rate or iterations.

---

<div class="post-metadata">

**Author:** ![Michal\_Majk\_Ritcherd](https://avatars.discourse-cdn.com/v4/letter/m/5daacb/32.png) [@Michal\_Majk\_Ritcherd](https://community.deeplearning.ai/u/Michal_Majk_Ritcherd)\
**Post date:** [July 24, 2023, 4:59am UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/4 "2023-07-24T04:59:54Z")

</div>

Yeah I did try a lot of things, but I wasn’t able to learn the regression.

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [July 25, 2023, 12:11am UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/5 "2023-07-25T00:11:32Z")

</div>

Sorry, but i am currently on leave, and do not have ability to try your experiment.

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [July 31, 2023, 11:57pm UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/6 "2023-07-31T23:57:20Z")

</div>

@Michal_Majk_Ritcherd, were you able to make progress on this experiment?

---

<div class="post-metadata">

**Author:** ![Michal\_Majk\_Ritcherd](https://avatars.discourse-cdn.com/v4/letter/m/5daacb/32.png) [@Michal\_Majk\_Ritcherd](https://community.deeplearning.ai/u/Michal_Majk_Ritcherd)\
**Post date:** [August 2, 2023, 5:06pm UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/7 "2023-08-02T17:06:49Z")

</div>

Yes, I was.  
I needed way higher learning rate (2.55).

But this did not work using tf.float32 and tf.float64, there I needed some changes as follows:  
I’ve created new model with 1 hidden layer with 6 neurons (ReLU activation) and set pretty high learning rate (1.29) and was able to train the NN. Although it depends on the parameter’s initialization.  
In one run I an able to train the NN in just 10 epochs, in another I wasn’t able (Tried 300 000 epochs and possibly it would learn with far more).

But still wondering why I am not able to train these 2 scenarios the same way as with tf.float16, meaning with just output layer…

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [August 2, 2023, 9:19pm UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/8 "2023-08-02T21:19:00Z")

</div>

> [@Michal\_Majk\_Ritcherd](#):
>
> Although it depends on the parameter’s initialization.

That’s not unusual for neural networks. Their cost function is not convex, so you can get a local minimum.

What weight and bias values did you get when it did converge?

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [August 2, 2023, 9:45pm UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/9 "2023-08-02T21:45:22Z")

</div>

Just FYI, I used your data set of two examples, using both logistic regression and an NN with one hidden layer, implemented in a different toolset, and it converged very quickly.

---

<div class="post-metadata">

**Author:** ![Michal\_Majk\_Ritcherd](https://avatars.discourse-cdn.com/v4/letter/m/5daacb/32.png) [@Michal\_Majk\_Ritcherd](https://community.deeplearning.ai/u/Michal_Majk_Ritcherd)\
**Post date:** [August 3, 2023, 7:08am UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/10 "2023-08-03T07:08:26Z")

</div>

When it converged, I got weight = [[-70.1]] and bias [-16.4]. With another run I get another different params, which is expected.

---

<div class="post-metadata">

**Author:** ![Michal\_Majk\_Ritcherd](https://avatars.discourse-cdn.com/v4/letter/m/5daacb/32.png) [@Michal\_Majk\_Ritcherd](https://community.deeplearning.ai/u/Michal_Majk_Ritcherd)\
**Post date:** [August 3, 2023, 7:12am UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/11 "2023-08-03T07:12:13Z")

</div>

> implemented in a different toolset, and it converged very quickly

What toolset did you use? Still I am not sure why I can’t train on the Logistic regression with different data type such as tf.float32 and tf.float64. I know that the updates will be small, but in my cases it gets stuck and don’t change at all…

Anyway thanks for your time.

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [August 3, 2023, 5:07pm UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/12 "2023-08-03T17:07:37Z")

</div>

> [@Michal\_Majk\_Ritcherd](#):
>
> Still I am not sure why I can’t train on the Logistic regression with different data type such as tf.float32 and tf.float64.

I don’t think the size of the float data has anything to do with your issue.

---

<div class="post-metadata">

**Author:** ![Michal\_Majk\_Ritcherd](https://avatars.discourse-cdn.com/v4/letter/m/5daacb/32.png) [@Michal\_Majk\_Ritcherd](https://community.deeplearning.ai/u/Michal_Majk_Ritcherd)\
**Post date:** [August 3, 2023, 5:23pm UTC](https://community.deeplearning.ai/t/binary-classification-simplest-classification-task/394676/13 "2023-08-03T17:23:32Z")

</div>

I agree with you, maybe I have bug somewhere in my code 😃
