@TMosh Thanks for the response. Actually I missed defining weights from imagenet. That property was set to None and I was getting 0.4466 accuracy before and after fine-tuning. When I fixed this, everything worked fine.
An interesting thing to note here is that I got 100 points even with incorrect implementation.