Data Loading Issues in Unet Image Segmentation

I am working on this layout segmentation model with unet. And my results are always so bad. I believe it is happening due to some mistake I am making in my dataloading process. There are at max 7 different colors in my ground truth files. And this it how I am loading it.

And here is an example from my ground truth files

But after training the model it changes colors.

Please let me know if you know what might be the issue.
Thank you in advance!