thanks for sharing this, I actually wanted to point this overfitting problem.
I notice in your next state steps you are state space size, Why didn’t you approach this from (window size-1) where each previous step, detect the next step. it will actually help your model network to also helps understand and compare the previous pattern with the current pattern, helping you in long term trends and short term fluctuations, and provide you insight on the data points flat minima at the end of episodic iterative training.