Hello,

Do Bellman equations still apply if I set the left reward at zero? I would have expected the values encircled in the generated graph to be zero in this case.

Hello,

Do Bellman equations still apply if I set the left reward at zero? I would have expected the values encircled in the generated graph to be zero in this case.

Hello @Mo_Okasha,

I believe you are setting the `misstep_prob`

to `0.4`

which means that even if we decide to go LEFT, there is still a chance that the robot will go RIGHT and finally get some rewards. This is why you see those encircled positive rewards.

Cheers,

Raymond

Hello @rmwkwok ,

No that was just me trying to play with the misstep_prob for another reason. However, even with misstep_prob is set to zero, i still see values for the left direction. Below is an updated screenshot. Thanks!

Hello @Mo_Okasha,

Perhaps try to re-run all the cells after setting 0 for `misstep_prob`

?

I did the tests myself with `misstep_prob`

be 0 or 0.4. I can reproduce your result with 0.4, and as for `misstep_prob = 0`

, the LEFT value in state s is always half the RIGHT value in state s-1 because every time it takes the LEFT action in state s and reach s-1, it will then turn to go RIGHT to achieve the biggest value. It is “halved” because of the `gamma`

, it will turn RIGHT because Q is defined as followed:

My screenshots below:

`misstep_prob = 0`

`misstep_prob = 0.4`

Raymond

But in your solution, I see that some LEFT values aren’t zero with misstep_prob = 0. Please see encircled values below. Thanks!