# Q function training

**URL:** <https://community.deeplearning.ai/t/q-function-training/680403>\
**Category:** Unsupervised Learning, Recommenders, Reinforcement\
**Tags:** week-module-3\
**Created:** [August 20, 2024, 6:10am UTC](https://community.deeplearning.ai/t/q-function-training/680403 "2024-08-20T06:10:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ryan106001020](https://avatars.discourse-cdn.com/v4/letter/r/e79b87/32.png) [@ryan106001020](https://community.deeplearning.ai/u/ryan106001020)\
**Post date:** [August 20, 2024, 6:10am UTC](https://community.deeplearning.ai/t/q-function-training/680403/1 "2024-08-20T06:10:27Z")

</div>

![螢幕擷取畫面 2024-08-20 140556](https://global.discourse-cdn.com/dlai/original/3X/f/c/fcc7cba103d1374404cf92d25efb655e00ca144c.png)  
I have a question about Q function. At first, we randomly guess Q, and we get y from this random Q function. We tried 10000 examples and paired (x,y) trying to update q to fit y, but we use original random q to get y. how is that possible we can get other q function. what is the meaning? please explained clearly thank you! really confused.

---

<div class="post-metadata">

**Author:** ![Alireza\_Saei](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/alireza_saei/32/85175_2.png) [@Alireza\_Saei](https://community.deeplearning.ai/u/Alireza_Saei)\
**Post date:** [August 20, 2024, 6:36am UTC](https://community.deeplearning.ai/t/q-function-training/680403/2 "2024-08-20T06:36:44Z")

</div>

Hey there @ryan106001020

Even though the initial Q function is random and inaccurate, but it slowly improves in the process of iteratively updating the Q function.

Here’s how it works:  
First, You use a random Q to estimate the target values (`y`) based on the rewards and future Q-values. Then, you train the Q function to better approximate these targets and update it, the Q function gets closer to the true Q values, making the agent smarter in the environment.

Hope it helps! Feel free to ask if you need further assistance.

---

<div class="post-metadata">

**Author:** ![ryan106001020](https://avatars.discourse-cdn.com/v4/letter/r/e79b87/32.png) [@ryan106001020](https://community.deeplearning.ai/u/ryan106001020)\
**Post date:** [August 20, 2024, 7:03am UTC](https://community.deeplearning.ai/t/q-function-training/680403/3 "2024-08-20T07:03:31Z")

</div>

Hi~  
the point i don’t understand is that:  
since i use inaccurate Q to make assumption for target y, what am I updating for?

𝑄(𝑠,𝑎;𝑤)=y hat  
𝑅+𝛾max𝑎′𝑄̂ (𝑠′,𝑎′;𝑤−)=y target  
we are calculating the loss between y hat and y target and updating Q function

is that correct?  
thank you for replying.

---

<div class="post-metadata">

**Author:** ![Alireza\_Saei](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/alireza_saei/32/85175_2.png) [@Alireza\_Saei](https://community.deeplearning.ai/u/Alireza_Saei)\
**Post date:** [August 20, 2024, 7:12am UTC](https://community.deeplearning.ai/t/q-function-training/680403/4 "2024-08-20T07:12:53Z")

</div>

Yes, you are correct! You’re updating the Q by minimizing the difference (loss) between the predicted Q value and the target value (y\_hat and y). The target y is based on the reward and the best estimate of future Q values (using γ max𝑎′𝑄̂(𝑠′,𝑎′)) and over n iterations, the updates improve the Q function’s ability to predict accurate values.