model_predict = unet(
noisy_model_input,
timesteps,
encoder_hidden_states,
)[0]
I’m trying to fine tune Dreambooth with my photos instead of Andrew’s. However, the code above is producing the argument of type 'NoneType' is not iterable error and I’m stuck.
First I was getting NaNs from the latent_dist.sample() and i upcasted the pixel values to torch.float32 and now i’m nomore having the NaNs. However, even after upcasting, the NoneType error is there!
I am having the same problem. I suspect there is something wrong with the default unet.config.addition_embed_type=“text_time” but when I reset that to None I get a matrix multiplication error
can you share a screenshot of the error.
regards
dp
print(unet.config.addition_embed_type) prints “text_time” which requires an additional directory to be passed (added_cond_kwargs) when calling unet. When I reset unet.config.addition_embed_type = None, then I get the matrix multiplication error.
“text_time” warning and error screen_shot
usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py:456: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at …/aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)
return F.conv2d(input, weight, bias, self.stride,
Setting unet.config.addition_embed_type = None here is the warning and error I get.
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py:456: UserWarning: Plan failed with an OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU (Triggered internally at …/aten/src/ATen/native/cudnn/Conv_v8.cpp:924.)
return F.conv2d(input, weight, bias, self.stride,
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py:456: UserWarning: Plan failed with an OutOfMemoryError: CUDA out of memory. Tried to allocate 8.00 GiB. GPU (Triggered internally at …/aten/src/ATen/native/cudnn/Conv_v8.cpp:924.)
return F.conv2d(input, weight, bias, self.stride,
that’s the exact same issue i got stuck at
1 Like
Can you confirm if you have downloaded all the necessary files also with the codes you are trying to run?
Yes, I believe so. Just to be extra cautious, I have included the entire utils.py file in my colab notebook. Fyi, there were three previous things I had to debug before getting to this bug. 1. create an xformers_is_available() function (mentioned by someone else in another thread) 2. fix a typo noise_scheduler.config_num_train_timesteps to noise_scheduler.config.num_train_timesteps 3. change return_dic =False to return_dict=False
can you share link to your colab with the codes you mentioned? let me check once!!
sharing a link here results in: An error occurred: Sorry, you can’t include links in your posts.
Hi @Mohsin_Ansari
The link your shared, tells me I do not have access, can you make the Google Colab link public, so I can have a look. Probably use some other random image than personal image for such assignments and then share the link.
Regards
DP
Hi @techsis
You did DM me but without the link to your Colab.
I have made the drive public and sent you a new link. I loaded the andrew pictures in my instance folder. The class folder is empty so upload the pictures of men used in the class into that directory.
Hi @Mohsin_Ansari . did you get a solution?
Hi @Mohsin_Ansari
-
Issues what I noticed, while pip installing comet_ml, there is an ERROR: Operation cancelled by user, this means you didn’t let the cell run completely.
-
The above issue cause next issue while you were setting new Comet experiment which gave you log info
COMET WARNING: To get all data logged automatically, import comet_ml before the following modules: torch.
COMET WARNING: As you are running in a Jupyter environment, you will need to call experiment.end() when finished to ensure all metrics and code are logged before exiting.
COMET INFO: Couldn’t find a Git repository in ‘/content’ nor in any parent directory. Set COMET_GIT_DIRECTORY if your Git Repository is elsewhere.
COMET INFO: Experiment is live on comet.com Comet.ml | Supercharging Machine Learning
- Can I know the significance of recalling unit configuration as None
unet.config.addition_embed_type = None
- Your next error
usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py:456: UserWarning: Plan failed with an OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU (Triggered internally at …/aten/src/ATen/native/cudnn/Conv_v8.cpp:924.)
return F.conv2d(input, weight, bias, self.stride,
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py:456: UserWarning: Plan failed with an OutOfMemoryError: CUDA out of memory. Tried to allocate 8.00 GiB. GPU (Triggered internally at …/aten/src/ATen/native/cudnn/Conv_v8.cpp:924.)
return F.conv2d(input, weight, bias, self.stride,
the above error is stating you do not enough GPU to run the model
-
Another similar error
OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU
-
RuntimeError Traceback (most recent call last)
in <cell line: 20>()
18
19 # Concatenate the tensors
—> 20 add_embeds = torch.cat([text_embeds, time_embeds], dim=-1) # Adjust dim as needed
21 add_embeds = add_embeds.to(encoder_hidden_states.dtype)
22
RuntimeError: Tensors must have same number of dimensions: got 3 and 2
The above error is giving an mismatch dimension error.
Honestly too many errors, I don’t why you still went ahead after error log, I cannot understand.
Another thing, to run numerous codes in a single cell also would be not a good choice.
If Google Colab is causing issue, try Jupyter notebook, but make sure you have enough memory to run the models.
Regards
DP
@techsis
You are suppose to make your Colab link public before you share a link.
When I click the link provided by you, it is stating I do not have access.
Also when you share the link, make sure you have run down the cell one by one, so I can access your error log as reflect in your system.
Regards
DP
