I could get it working with langchain.embeddings.azure_openai.AzureOpenAIEmbeddings which I couldn’t figure out earlier because of its different namespace.
There is a great post on this around this Utilizing Azure OpenAI for Completing Deeplearning.ai Generative AI Courses
Sharing it here since it has much more information than my answer and gives a very good insight into parity between Open AI and Azure Open AI in general.