Help required to create "Text to Image" Gradio app in Spaces

[vsrinivas]

I am trying to create my own Text to Image Space using the concepts explained in the lesson. I am using the below code. However, even after several debugging efforts, I am unable to get rid of the error message. Hope someone can help here. By the way, while debugging I understood, negative_prompt does not exist in diffusionpipeline/ stablediffusionpipeline any more.

import gradio as gr
from transformers import pipeline
import torch
from diffusers import DiffusionPipeline
    
def get_completion(prompt,params):
    return pipeline(prompt=prompt, height=params['height'], width=params['width'], 
                    num_inference_steps=params['num_inference_steps'], 
                    guidance_scale=params['guidance_scale'])['sample'][0]

def generate(prompt,steps,guidance,width,height):
    params = {
        "num_inference_steps": steps,
        "guidance_scale": guidance,
        "width": width,
        "height": height
    }
    output = get_completion(prompt,params)
    return output
    
pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)

with gr.Blocks() as demo:
    gr.Markdown("# Image Generation with Stable Diffusion")
    with gr.Row():
        with gr.Column(scale=4):
            prompt = gr.Textbox(label="Your prompt") #Give prompt some real estate
        with gr.Column(scale=1, min_width=50):
            btn = gr.Button("Submit") #Submit button side by side!
    with gr.Accordion("Advanced options", open=False): #Let's hide the advanced options!
            # negative_prompt = gr.Textbox(label="Negative prompt")
            with gr.Row():
                with gr.Column():
                    steps = gr.Slider(label="Inference Steps", minimum=1, maximum=100, value=25,
                      info="In many steps will the denoiser denoise the image?")
                    guidance = gr.Slider(label="Guidance Scale", minimum=1, maximum=20, value=7,
                      info="Controls how much the text prompt influences the result")
                with gr.Column():
                    width = gr.Slider(label="Width", minimum=64, maximum=512, step=64, value=512)
                    height = gr.Slider(label="Height", minimum=64, maximum=512, step=64, value=512)
    output = gr.Image(label="Result") #Move the output up too

    btn.click(fn=generate, inputs=[prompt,steps,guidance,width,height], outputs=[output])

demo.launch()

The log shows following error message:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/gradio/queueing.py", line 388, in call_prediction
    output = await route_utils.call_process_api(
  File "/home/user/.local/lib/python3.10/site-packages/gradio/route_utils.py", line 219, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/user/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1437, in process_api
    result = await self.call_function(
  File "/home/user/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1109, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/user/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/user/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/home/user/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/home/user/.local/lib/python3.10/site-packages/gradio/utils.py", line 641, in wrapper
    response = f(*args, **kwargs)
  File "/home/user/app/app.py", line 21, in generate
    output = get_completion(prompt,params)
  File "/home/user/app/app.py", line 7, in get_completion
    return pipeline(prompt=prompt, height=params['height'], width=params['width'],
  File "/home/user/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 635, in __call__
    prompt_embeds, negative_prompt_embeds = self.encode_prompt(
  File "/home/user/.local/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 338, in encode_prompt
    prompt_embeds = self.text_encoder(
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 840, in forward
    return self.text_model(
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 745, in forward
    encoder_outputs = self.encoder(
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 656, in forward
    layer_outputs = encoder_layer(
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 385, in forward
    hidden_states = self.layer_norm1(hidden_states)
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/normalization.py", line 190, in forward
    return F.layer_norm(
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/functional.py", line 2515, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

[vsrinivas]

@elirod @nilosreesengupta hi, can you help on this?

[elirod]

@vsrinivas

"LayerNormKernelImpl" not implemented for 'Half'

I found a issue on the hugging face community regards this run time error and they advise to use two arguments on the stable diffusion model: --precision full and --no-half

Give it a try and let me know the results

[vsrinivas]

After a lot of struggle, I was able to resolve the issue . Below are some points that I have identified.

1. In my pipeline statement pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16) I have added the parameter torch_dtype=torch.float16. This works only with GPUs and not CPUs. Unfortunately the Huggingface free account for Spaces platform provide only CPU support. Hence, I was facing error during inference. If I remove that, I could successfully run.

2. While inference, I have subscripted pipeline object with ['sample'][0]. This surprisingly works in Colab environment but not in HF Spaces. In spaces, we shall use the .images[0] attribute as per stablediffusion pipeline documentation and as provided in the lesson. I remember .images[0] was not working in colab.

3. If I use negative_prompt parameter in the pipeline, it returns error with unknown parameter given, negative_prompt, though both in the lesson and in the stablediffusion, this parameter is provided.

Hope the above helps.

[elirod]

Well done! And thanks for sharing

[nilosreesengupta]

Hello @vsrinivas ,

Sorry for the delay in replying, I missed the notification email of my name mentioned in this thread among my bunch of emails.

Glad to see you yourself solved it. Great job!

Happy learning!

With regards,
Nilosree Sengupta