Reflection Pattern

Course Q&A Agentic AI
,
1

Dear community,

In the M2 graded lab, we reflected on the generated draft in one step and then applied the feedback from that reflection in a subsequent step.

In the M3 graded lab, the reflection and the application of its feedback were performed within the same step.

I have a few questions about these approaches:

  1. Which pattern is generally more advisable when applying reflection:

    • separating the generation of the reflection and the application of that reflection into two steps (possibly using different LLMs), or

    • performing both within a single step using the same model to reduce cost and latency?

  2. Does it make sense to have the same LLM both perform a task (e.g., generate a report) and review the result (e.g., reflect on the generated report and optionally produce an improved version), assuming different prompts are used? Could the use of the same LLM introduce bias in the reflection process? For example, might a model be more inclined to favor its own outputs and be more critical of outputs produced by other models?

Curious to hear how others approach this.

Thanks!

2

Hi Ana_1,

Thanks for your questions. Here’s my two cents.

As for your first question, I would say it depends on the application. I guess this can be seen as a general comment on the use of LLMs. When an application is relatively straightforward and does not have critical implications, draft and reflection could happen in a single step. When things are more complex and/or have critical implications, separating the two allows for adding control through additional code or human-in-the-loop.

Regarding the second, this would seem to be an empirical question. In my experience, different models do tend to have different styles of output. But that does not necessarily mean they produce more critical output when prompted to reflect on the output of a different model than on their own output. It also depends on the kind of critique that is desired for a particular application. One way to approach this would be to just try and find out taking the use case as the starting point.

1 Like