U-Nets and transfer learning

Hi, I was wondering if it is possible to apply the same type of transfer learning that we saw in Week2 of the course to U-Nets. In case it is, how can you implement such an idea? I mean, how can you decide which layers to freeze and which layers to train?

Great question!

I am not an expert on the matter, but you are onto something:

The common practice in transfer learning is to keep the shallow layers unchanged and to modify deeper layers according to the new data set. This approach may not work when using a U-Net and when moving from a different domain to ultrasound (US) images due to their drastically different appearance.


The results showed that choosing which layers to fine-tune is a critical task. In particular, they demonstrated that fine-tuning the last layers of the network, which is the common practice for classification networks, is often the worst strategy. It may therefore be more appropriate to fine-tune the shallow layers rather than deep layers in US image segmentation when using a U-Net. Shallow layers learn lower level features which are critical in automatic segmentation of medical images. Even when a large US data set is available, we observed that fine-tuning shallow layers is a faster approach compared to fine-tuning the whole network.

I would say you should treat which layers to train as a hyperparameter and you evaluate the performance on your val data set. Just like any other hyperparameter tuning we do :slight_smile: