Transfer learning and hyperparameters

Hello,

I was wondering if transfer learning could help finding a good approximation on some key hyperparameters or initialization of the network when the number of pre tuning samples at hand is not much greater than the fine tuning ones. Has it been tried already ?

Say you are transferring a model that learns from dataset A to learn on your new dataset B.
This is useful if features from dataset A have been well captured by the model.

In short, what are you transferring via a bad model?