One need to understand more importantly about the training and testing data they are working on to check overfitting and undercutting of any ai model, be it a small model or larger model.
Smaller model it can be easier to detect the issue as you can review the data with linear approach where as in larger data like your mentioned when you need to train on U-Net, you need to make sure to check if the data you are working on is not highly imbalanced, like if it is classification model, it should one class more often than other class as the model tends to adapt to the presence of class which is higher in presence..
This can also happen if the model was created on a highly imbalanced data, where the validation data had more number of one class than compare to other, but in the training if that class were less in number than model would very less accuracy due to under presentation of other class in the training data.
Here is a link
Feel free to ask if any question.