L2 section Captioning with gr.Interface() not work

[fabiodasilva]

Hi
In L2: Image captioning app, I can use correctly up to this point of code:

image_url = “https://free-images.com/sm/9596/dog_animal_greyhound_983023.jpg”
display(IPython.display.Image(url=image_url))
get_completion(image_url)

Output:
[{‘generated_text’: ‘a dog wearing a santa hat and a red scarf’}]

After launch the gradio interface code everything related with local ou remote access is ok, but I always receive a big and red ERROR inside the caption box:

import gradio as gr

def image_to_base64_str(pil_image):
byte_arr = io.BytesIO()
pil_image.save(byte_arr, format=‘PNG’)
byte_arr = byte_arr.getvalue()
return str(base64.b64encode(byte_arr).decode(‘utf-8’))

def captioner(image):
base64_image = image_to_base64_str(image)
result = get_completion(base64_image)
return result[0][‘generated_text’]

gr.close_all()
demo = gr.Interface(fn=captioner,
inputs=[gr.Image(label=“Upload image”, type=“pil”)],
outputs=[gr.Textbox(label=“Caption”)],
title=“Image Captioning with BLIP”,
description=“Caption any image using the BLIP model”,
allow_flagging=“never”,
examples=[“dog.jpeg”, “cow.jpeg”])

demo.launch(share=True)

Any idea of what I need to do to fix this ?

Tks,

Fábio

[HenryB]

Hi Fábio, I have the same problem

[fabiodasilva]

I decided to run the same code in Google colab and works fine !

This lead me to think that is something wrong with gradio and my local environment.

Following the output of print(os.environ()):
environ({‘__CFBundleIdentifier’: ‘com.apple.Terminal’, ‘TMPDIR’: ‘/var/folders/0b/k2rqhkbd14bf6rxgpm_ltnx40000gn/T/’, ‘XPC_FLAGS’: ‘0x0’, ‘LaunchInstanceID’: ‘0F5B1071-56D4-4897-A633-4E90B34E1C17’, ‘TERM’: ‘xterm-color’, ‘SSH_AUTH_SOCK’: ‘/private/tmp/com.apple.launchd.RplAdG6KGN/Listeners’, ‘SECURITYSESSIONID’: ‘186b3’, ‘XPC_SERVICE_NAME’: ‘0’, ‘TERM_PROGRAM’: ‘Apple_Terminal’, ‘TERM_PROGRAM_VERSION’: ‘447’, ‘TERM_SESSION_ID’: ‘EE1E36B0-8C9D-46F9-8CC6-4DF2AF1D663C’, ‘SHELL’: ‘/bin/zsh’, ‘HOME’: ‘/Users/fabiosilva’, ‘LOGNAME’: ‘fabiosilva’, ‘USER’: ‘fabiosilva’, ‘PATH’: ‘/Users/fabiosilva/miniforge3/envs/dsa_python/bin:/Users/fabiosilva/miniforge3/condabin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin’, ‘SHLVL’: ‘1’, ‘PWD’: ‘/Users/fabiosilva/Downloads/_python/DLAI’, ‘OLDPWD’: ‘/Users/fabiosilva/Downloads/_python’, ‘CONDA_EXE’: ‘/Users/fabiosilva/miniforge3/bin/conda’, ‘_CE_M’: ‘’, ‘CE_CONDA’: ‘’, ‘CONDA_PYTHON_EXE’: ‘/Users/fabiosilva/miniforge3/bin/python’, ‘CONDA_SHLVL’: ‘2’, ‘CONDA_PREFIX’: ‘/Users/fabiosilva/miniforge3/envs/dsa_python’, ‘CONDA_DEFAULT_ENV’: ‘dsa_python’, ‘CONDA_PROMPT_MODIFIER’: '(dsa_python) ', ‘CONDA_PREFIX_1’: ‘/Users/fabiosilva/miniforge3’, ‘GSETTINGS_SCHEMA_DIR_CONDA_BACKUP’: ‘’, ‘GSETTINGS_SCHEMA_DIR’: ‘/Users/fabiosilva/miniforge3/envs/dsa_python/share/glib-2.0/schemas’, ‘XML_CATALOG_FILES’: ‘file:///Users/fabiosilva/miniforge3/envs/dsa_python/etc/xml/catalog file:///etc/xml/catalog’, ‘LC_CTYPE’: ‘UTF-8’, '’: ‘/Users/fabiosilva/miniforge3/envs/dsa_python/bin/jupyter’, ‘PYDEVD_USE_FRAME_EVAL’: ‘NO’, ‘JPY_SESSION_NAME’: ‘/Users/fabiosilva/Downloads/_python/DLAI/L2_Image_captioning_app.ipynb’, ‘JPY_PARENT_PID’: ‘2643’, ‘CLICOLOR’: ‘1’, ‘FORCE_COLOR’: ‘1’, ‘CLICOLOR_FORCE’: ‘1’, ‘PAGER’: ‘cat’, ‘GIT_PAGER’: ‘cat’, ‘MPLBACKEND’: ‘module://matplotlib_inline.backend_inline’})

Any idea of what can I do to fix that ?

Tks
Fábio

[leastsquared]

Hi @fabiodasilva. If you are running locally, you have to replace the “captioner” on fn param by the function that you create to captione the image.

Try:

import gradio as gr

def captioner(image):
result = get_completion(image)
return result[0][‘generated_text’]

gr.close_all()
demo = gr.Interface(fn=captioner,
inputs=[gr.Image(label=“Upload image”, type=“pil”)],
outputs=[gr.Textbox(label=“Caption”)],
title=“Image Captioning with BLIP”,
description=“Caption any image using the BLIP model”,
allow_flagging=“never”,
examples=[“dog.jpeg”, “cow.jpeg”])

[Beejay]

I’m running into issues here too.

I’m running this in a Google Colab.
I already get an error in the cell with the test image.
The cell runs but I get an output: {‘error’: “a bytes-like object is required, not ‘str’”}

When I come to the actual Captioning with gr.Interface(), the blocks of the interface load, I can drag and drop an image but like @fabiodasilva I only get an error.

I’ve been playing around a bit.

• The first time I ran it, the error said ‘Unexpected end of JSON input’, now it just says ‘Error’.
• I also tried it with share=True but in a browser it gives the same result.
• I am also very confused with the Salesforce/blip-image-captioning-base model on HF. It seems you cannot just spin it up since it lacks a handler.py file. You have to provide it yourself (I found one on HF).

Any ideas what I’m missing here?