# Size of the expected output does not make sense

**URL:** <https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711>\
**Category:** Neural Networks and Deep Learning\
**Tags:** week-module-2, coursera-platform\
**Created:** [September 6, 2024, 1:50pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711 "2024-09-06T13:50:49Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![chay\_guez](https://avatars.discourse-cdn.com/v4/letter/c/ccd318/32.png) [@chay\_guez](https://community.deeplearning.ai/u/chay_guez)\
**Post date:** [September 6, 2024, 1:50pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/1 "2024-09-06T13:50:49Z")

</div>

Hi!  
I have an issue in this function :

```auto
def optimize(w, b, X, Y, num_iterations=100, learning_rate=0.009, print_cost=False):
    for i in range(num_iterations):
.
.
.
        # Record the costs
        if i % 100 == 0:
            costs.append(cost)
.
.
.
    return params, grads, costs

```

Logically, the costs array should be of size 1 because parameter i go from 0 to 99  
But the error I get implies otherwise :

AssertionError: Wrong values for costs. [array(5.80154532), array(nan)] != [5.80154532, 0.31057104]

---

<div class="post-metadata">

**Author:** ![balaji.ambresh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/balaji.ambresh/32/128697_2.png) [@balaji.ambresh](https://community.deeplearning.ai/u/balaji.ambresh)\
**Post date:** [September 6, 2024, 1:59pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/2 "2024-09-06T13:59:48Z")

</div>

Please take a look at `optimize_test` which invokes `optimize` function with 101 iterations like this:

```python
params, grads, costs = target(w, b, X, Y, num_iterations=101, learning_rate=0.1, print_cost=False)

```

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [September 6, 2024, 4:47pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/3 "2024-09-06T16:47:00Z")

</div>

That error message is telling you that you got the correct cost on the first iteration, but you got NaN for the cost after 100 iterations. Most likely that means that your logic for updating the parameters is incorrect somehow. E.g. did you add the gradient values, instead of subtracting them?

---

<div class="post-metadata">

**Author:** ![chay\_guez](https://avatars.discourse-cdn.com/v4/letter/c/ccd318/32.png) [@chay\_guez](https://community.deeplearning.ai/u/chay_guez)\
**Post date:** [September 8, 2024, 9:53am UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/4 "2024-09-08T09:53:02Z")

</div>

thank you everybody you’ve been very helpful!  
(yes I added them how did you know 😂)

---

<div class="post-metadata">

**Author:** ![chay\_guez](https://avatars.discourse-cdn.com/v4/letter/c/ccd318/32.png) [@chay\_guez](https://community.deeplearning.ai/u/chay_guez)\
**Post date:** [September 8, 2024, 9:54am UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/5 "2024-09-08T09:54:47Z")

</div>

do you know why adding them result in a Nan? instead of just resulting in an incorrect number

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [September 8, 2024, 2:51pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/6 "2024-09-08T14:51:30Z")

</div>

The NaN is caused by getting 0 \* log(0) from either the y \* log(\hat{y}) term or the (1 - y) \* log(1 - \hat{y}) term. log(0) is -\infty, but 0 \* -\infty gives you NaN (not a number) in floating point.

Now that I think a bit harder about it, I’m not sure why adding will cause that, but I’ve seen this syndrome a number of times before and that’s usually the cause. 🤓

Let me run the experiment myself and look at the actual numbers I get. Stay tuned!

---

<div class="post-metadata">

**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\
**Post date:** [September 8, 2024, 3:37pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/7 "2024-09-08T15:37:10Z")

</div>

@chay_guez starting to wonder what an ‘incorrect number’ would mean 😳

And presuming, though it is not stated, one understands ‘NaN’ stands for ‘Not a number’.

Reminds me of my Harvard Biology Professor who once stated this case:

[How NASA Lost Its Mars Climate Orbiter From a Metric Error](https://www.simscale.com/blog/nasa-mars-climate-orbiter-metric/#:~:text=In%20September%20of%201999%2C%20after,%2C%20i.e.%2C%20the%20metric%20units)!

Personally, I’d rather have the computer tell me ‘you have no idea what you are doing !’-- Rather than getting an answer that is wrong 😉

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [September 8, 2024, 4:17pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/8 "2024-09-08T16:17:23Z")

</div>

Before I dig into the details of what actually happens in this case, here is my general experiment to show the behavior of things like divide by zero and how `Inf` and `NaN` propagate in floating point computations:

```auto
v = 42. * np.ones((1,4), dtype = 'float64')
print(f"type(v) = {type(v)}")
print(f"v = {v}")
w = np.zeros((1,4), dtype = 'float64')
print(f"type(w) = {type(w)}")
print(f"w = {w}")
z = v / w
print(f"z = {z}")
a = -1. * z
print(f"a = {a}")
b = z + 42.
print(f"b = {b}")
c = z - 42.
print(f"c = {c}")
d = z + z
print(f"d = {d}")
e = z - z
print(f"e = {e}")
f = z / z
print(f"f = {f}")
g = np.log(w)
print(f"g = {g}")
h = np.log(-1 * v)
print(f"h = {h}")

```

Running the above gives this:

```auto
type(v) = <class 'numpy.ndarray'>
v = [[42. 42. 42. 42.]]
type(w) = <class 'numpy.ndarray'>
w = [[0. 0. 0. 0.]]
z = [[inf inf inf inf]]
a = [[-inf -inf -inf -inf]]
b = [[inf inf inf inf]]
c = [[inf inf inf inf]]
d = [[inf inf inf inf]]
e = [[nan nan nan nan]]
f = [[nan nan nan nan]]
g = [[-inf -inf -inf -inf]]
h = [[nan nan nan nan]]

```

Note that this behavior is specified by the [IEEE 754 spec](https://en.wikipedia.org/wiki/IEEE_754), so it it not specific to numpy. It is implemented in the lower level math libraries and in the FP hardware. I have another cell that runs the same experiments in TF instead and the results are exactly the same.

---

<div class="post-metadata">

**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\
**Post date:** [September 8, 2024, 4:36pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/9 "2024-09-08T16:36:09Z")

</div>

@paulinpaloalto okay, Paul, looking at this I have a weird question I don’t know the answer to…

How exactly (or even mathematically), does negative infinity differ from positive infinity ?

\*And, at least someone here has been reading Douglas Adams.

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [September 8, 2024, 6:12pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/10 "2024-09-08T18:12:18Z")

</div>

It’s negative instead of positive. 😆 But seriously folks, it’s a good question. From a functional standpoint, it probably doesn’t make a lot of difference, since (as demonstrated above) you can’t do arithmetic with it.

But to be slightly more serious, look at the graph of log(z) for 0 \< z \leq 1:

 ![image](https://global.discourse-cdn.com/dlai/original/3X/3/6/36d17018ffe63ed68a14a26a6ec1f066fed78fbe.jpeg)

So you can see that as z \rightarrow 0 the value of log(z) gets larger in magnitude, but the sign is negative for the entire domain 0 \< z \< 1. The closer z gets to 0, the bigger a negative number we get for log(z).

So from a mathematical standpoint, you’d express that using limits:

\displaystyle \lim\_{z \rightarrow 0}log(z) = -\infty

By contrast, if you look at the graph of e^z, then it’s clear that:

\displaystyle \lim\_{z \rightarrow \infty}e^z = +\infty

---

<div class="post-metadata">

**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\
**Post date:** [September 8, 2024, 7:23pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/11 "2024-09-08T19:23:46Z")

</div>

😁 Okay, I get that part; But recently I am thinking about ‘race conditions’-- So if the equation ‘heads south’ (or to ‘true north’) is this telling us something?

Or the direction (i.e the sign) does not matter ?

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [September 8, 2024, 7:27pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/12 "2024-09-08T19:27:41Z")

</div>

I’m not sure what you are really asking. In my experience the term “race conditions” means you are doing parallelization and you have done things in a way that is non-deterministic. That’s generally ok, although “race conditions” can also be bugs in your code, meaning you need to have locking to protect the update of certain shared “state” variables that are used by all the parallel threads.

> [@Nevermnd](#):
>
> Or the direction (i.e the sign) does not matter ?

Not sure how that applies to race conditions, but the point of my previous post about the difference between -\infty and +\infty was that the direction _does_ matter, right?

I think I must just be missing your point.

---

<div class="post-metadata">

**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\
**Post date:** [September 8, 2024, 7:39pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/13 "2024-09-08T19:39:10Z")

</div>

Paul, I’ve been playing around with this CUDA stuff and it gets hard once you start to implement custom streams. At least the default is ‘always blocking’, so you can rely on that. At that point it is ‘not so easy’ and you can kind of have all sorts of values floating in at any time.

At first I kind of scoffed when they included an algorithm checker at the end of every routine; I get it now though, it is not that hard to have the GPU produce you results… But they could also be horribly wrong.

I guess what I was wondering, from a ‘practical’ point of view, -Inf = Inf ? Or, no…

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [September 8, 2024, 7:49pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/14 "2024-09-08T19:49:12Z")

</div>

When you implement parallel algorithms, correctness matters, as it always does. But the point is it’s a lot harder to get right. E.g. consider the classic “single queue, multi server” model for parallelizing a task. You break the work to be done up into chunks and create a queue of those chunks. Then you unleash your parallel worker threads and they each do:

```auto
Fetch next item from queue
Complete task
Record results
Repeat

```

There are (at least) two points in that process where all the threads are accessing a shared data structure: the work queue and the results. If you don’t provide proper locking so that the updating of the queue and the results are actually single threaded, then your results are garbage.

Of course that’s just one simple example. So maybe you can argue that it doesn’t matter whether your wrong results are too negative or too positive. Wrong is wrong, right? 😄

I’m about to get busy with “real life” for the next 4 or 5 hours, so I won’t be able to make any further progress for a while on the original question here.

---

<div class="post-metadata">

**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\
**Post date:** [September 8, 2024, 7:49pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/15 "2024-09-08T19:49:24Z")

</div>

So you know… I am not afraid to ask for help, when I feel I need it:

> **[Max threads/blocks](https://forums.developer.nvidia.com/t/max-threads-blocks/305437/11)**
>
> A kernel call within a stream is always one stream operation. All blocks and threads finish before the stream continues. It does not matter, if the same or a different kernel is called as next operation in the stream. Within a kernel call the...

---

<div class="post-metadata">

**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\
**Post date:** [September 8, 2024, 7:55pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/16 "2024-09-08T19:55:21Z")

</div>

And no-- We replied at the same time, but that is just chance. Paul, go out and enjoy yourself 😁.

Best,  
-A

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [September 8, 2024, 8:00pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/17 "2024-09-08T20:00:36Z")

</div>

Sorry, I won’t be able to answer that question from the CUDA thread without basically taking that course myself, since you are using a bunch of technical terms that are not generic but are very specific to how their APIs work.

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [September 8, 2024, 8:28pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/19 "2024-09-08T20:28:37Z")

</div>

> [@Nevermnd](#):
>
> does negative infinity differ from positive infinity ?

Yes, they’re on opposite ends of the number line.

---

<div class="post-metadata">

**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\
**Post date:** [September 8, 2024, 8:37pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/20 "2024-09-08T20:37:10Z")

</div>

@TMosh Damn it. For a short time everyone benefits (but I did not include the solution you need to pass);

And I get the positive/negative part. Perhaps I should be more blunt-- Why does it slide one way or another ? I think that has to be saying ‘something’… But I don’t know what it is…

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [September 8, 2024, 8:46pm UTC](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711/21 "2024-09-08T20:46:48Z")

</div>

Paul’s examples provide results for both +Inf and -Inf. You can study those to see what causes each.

[Next page](https://community.deeplearning.ai/t/size-of-the-expected-output-does-not-make-sense/688711.md?page=2)