Reinforcement Learning

Course Q&A Machine Learning Specialization Unsupervised Learning, Recommenders, Reinforcement
1

When creating a post, please add:

  • Week #3.
  • Description:
  • I am currently learning reinforcement learning and its hard to understand the concept. I was able to understand state-value-function, Bellman equation but I could not understand how Neural network can be used to make predictions. I understood like supervised learning NN tries to determine optimal Q Value but without training data how is that possible. Its also said NN creates training data but are those referring to tuples or experiences which are saved?
1 Like
2

In reinforcement learning (RL), neural networks are used in a way that’s different from traditional supervised learning, where we usually have a fixed dataset. Here’s how it works:

  1. Experience Replay: Instead of using a static dataset, RL algorithms often use experience replay. This involves storing the agent’s experiences (tuples of state, action, reward, next state) in a replay buffer. These experiences are then randomly sampled to train the neural network. This helps break the correlation between consecutive experiences and stabilizes training.
  2. Q-Learning and Deep Q-Networks (DQN): In Q-learning, the goal is to learn the optimal action-value function (Q-value) that tells us the maximum expected reward for taking a particular action in a given state. In DQN, a neural network is used to approximate this Q-value. The training data for the neural network comes from the agent’s own interactions with the environment.
  3. Generating Training Data: The training data (tuples of state, action, reward, next state) is generated by the agent as it explores the environment. The agent uses its current policy (which might be a neural network) to take actions and collect rewards. These interactions are stored and later used to update the neural network.
  4. Updating the Neural Network: The neural network is updated by minimizing the difference between the predicted Q-values and the target Q-values. The target Q-values are calculated using the Bellman equation:
1 Like