Did try looking at older threads for similar issue? @hj320
there are more similar threads, when your loss is not reducing meaning it is not learning any new variation, so is the box predictor being assigned or recalled correctly?
Check these two threads!! there are more threads which could explain more, explore those thread, will learn alot going through comments.