We have over-fitting concept in supervised learning, in the same way do we have concept of over-fitting in reinforcement learning?
Please check this paper out for analysis and discussion of overfitting (and underfitting) in RL. The main content is 11-page long, and while I encourage you to read them all, the abstract and section 5 might already provide you some very good insights. There, you will come across some techniques that we have learnt in the MLS too.
PS: I moved this topic to the MLS Course 3 Week 3 category.