Is it a practical to hope for developing ML Skills in parallel to a job in Software Engineering

I am about to join a job as a software engineer. Primarily for Mixed-reality application development (C#, Unity3D). However, I am very fascinated by computer vision and ML-Ops. For personal reasons, I might have to stick to this software engineering position for at least the next 3-4 years. It is in a startup that partially resulted from my MS research and I’ll be the first employee there. I want to help the startup.
As the initial product requires most effort in software engineering, I’ll not get any professional chance to practice ML/DL. However, I am thinking about continuing learning on and doing side projects to one day shift into an ML position.

I wanted to ask here, whether my plan or hope is impractical. This will be my first industry job so I don’t have a lot of practical experience to guess whether or not my plan is pragmatic. I would really appreciate any comments or suggestions.