Figured out what happened, I forgot about the ignore class when we do the masked loss, glad that i figured it out by myself!