Mistake in the Lab 3

Short Course Q&A Evaluating AI Agents
,
1

In Lab 3 of Evaluating AI Agents course:

query = SpanQuery().where(
    # Filter for the `LLM` span kind.
    # The filter condition is a string of valid Python boolean expression.
    "span_kind == 'LLM'",
).select(
    question="input.value",
    tool_call="llm.tools"
)

# The Phoenix Client can take this query and return the dataframe.
tool_calls_df = px.Client().query_spans(query, 
                                        project_name=PROJECT_NAME, 
                                        timeout=None)
tool_calls_df = tool_calls_df.dropna(subset=["tool_call"])

tool_calls_df.head()

I think tool_call="llm.tools" is incorrect, this only retrieves all the available tools given to the LLM, but not the chosen tool returned by the LLM.

Can you have a check?

2

Yes you are right, I was going through the lab. I think the correct one is

tool_call="llm.output_messages"

I am happy someone spotted the same thing as me that made me scratch my head.

Whole cell here:

import pandas as pd

pd.set_option('display.max_colwidth', None) # Set to None to display full content

query = SpanQuery().where(
    # Filter for the `LLM` span kind.
    # The filter condition is a string of valid Python boolean expression.
    "span_kind == 'LLM'",
).select(
    question="input.value",
    tool_call="llm.output_messages"
)

# The Phoenix Client can take this query and return the dataframe.
tool_calls_df = px.Client().query_spans(query, 
                                        project_name=PROJECT_NAME, 
                                        timeout=None)
tool_calls_df = tool_calls_df.dropna(subset=["tool_call"])

tool_calls_df.head()
3

Hi @hawraa.salami , could you help us verify on this? :smiley: Wanna make sure future students could have accurate cells when doing the labs

4

@sheen_an’s answer is correct, but then you’ll also need to filter correctly in order to just keep rows where tool_call column contains “tool_call”. Something like this:

query = SpanQuery().where(
    # Filter for the `LLM` span kind.
    # The filter condition is a string of valid Python boolean expression.
    "span_kind == 'LLM'",
).select(
    question="input.value",
    tool_call="llm.output_messages"
)


# The Phoenix Client can take this query and return the dataframe.
tool_calls_df = px.Client().query_spans(query,
                                        project_name=PROJECT_NAME,
                                        timeout=None)


# tool_calls_df = tool_calls_df.dropna(subset=["tool_call"])

# Filter to keep rows where tool_call column contains "tool_call"
tool_calls_df = tool_calls_df[tool_calls_df['tool_call'].astype(str).str.contains('tool_call', na=False)]