I agree with you, @tbhaxor.
Transfer Learning works well, e.g. in applications of:
- if you learn a transfer of style
- if you have (implicit) representative features learned in the pre-trained model which help in the fine-tuning, see also @shanup’s excellent example above!
In the end a deep learning model can learn what is contained in the data. If the data is sufficiently representative, compared to the new specialised tasks, transfer learning should be worth a look! Then you can leverage the pre-trained model (which is usually also super expensive and difficult to create in the first place).
Best regards
Christian