Hi, @mickey.
Yes, it could spiral around the minimum if Beta is too large.
Here is a nice simulator you can play with. You can download the source code and make any adjustments you want. E.g., Momentum with Beta set to 0.99:
Have fun experimenting 
1 Like