I am trying to build a project that takes the dataset of images captured by an autonomous car during Daylight and convet those images into Nightime. It is an unsupervised learning task.
Since this project is to be made using core Deep Learning, i.e. CNNs and LSTMs, I cannot use any type of GANs or tools that are based upon GANs.
My initial idea was to use CNN with Neural Style Transfer to convert daytime images to nightime, but that would require a separate dataset with just nightime images, and then output a mixed image. That is, however, not the goal of the project.
I’m running out of ideas. All advices and suggestions are appreciated.
@K-Aditya25 In all honesty this even seems to me like it would possibly be an easier task to accomplish by building your own script ‘Action’ in Photoshop and batch processing the images. No NN required.
@TMosh I went through the paper. Though they’re using Neural ISPs to transform daytime to nightime images, they’re using a supervised learning approach to do so.
Their dataset comprises of explicitly labelled day and nightime images.
Would you know of any way to achieve this task through an unsupervised learning approach?
I have searched the internet for all possible ways to generate images through deep learning, and all I can find is GAN based approaches, VAE or diffusion based models.
Maybe this is a lack of imagination on my part, but I also can’t think of any way to do this that is unsupervised learning. How do you know what a nighttime image even means without examples?
You could try multiplying all pixels by a constant factor that moves them in the direction of darker. But then you’d be using your judgement to decide the right factor, so how is that ML?
Or maybe this is just a trick question and you’re supposed to come back and say that the request doesn’t really make sense.
Hi, @paulinpaloalto . Thanks for your reply. You are right, I had to get the problem statement right on my part, there must be, and in fact, there are night time images in the dataset as well.
However, the core of the problem statement remains the same, it has to be achieved only using core DL techniques.
I have decided to go ahead with Neural Style Transfer. My approach will be to take the night time images and day time images and try to combine them to produce some output. Let’s see how things go. I am simultaneously learning about Neural Style Transfer from the DL specialisation.
Any suggestions that you might have are highly appreciated.
Thanks
Aditya
Hi @TMosh. I want to thank you for referring this research paper here. While this paper didn’t actually help achieve the task at hand, it increased my knowledge about how cameras work and how those highly sought after night mode images are produced in flagship phones today.
For those reading this thread, here is a summary of the research paper referred by @TMosh :
The researchers take daytime images captured on Samsung S20 FE and transform them into night time images using a thorough pipeline mentioned in the paper. They achieve this by altering the metadata of the image. No deep learning is involved here.
The output of the pipeline are two images: a noisy night time image and a clean synthetic night time image.
Further, they use this pair of noisy-clean night time images to train a neural ISP (Image Signal Processor present in smartphone cameras) which is based U-Net architecture. The input for the neural ISP are the noisy night time images and the target output are the clean nighttime image. So, here, Deep Learning is involved.
