Source_documents are returned even if the question has no context related

Just to understand I’m doing the right thing and how to manage this kind of scenario.

I’m playing with the code of the lesson 6 (the part without the memory) trying to ask for a question not related the information contained into pdf documents.

just the important part of the code for reference


  qa_chain = RetrievalQA.from_chain_type(
        llm,
        retriever=vectordb.as_retriever(),
        return_source_documents=True,
        chain_type_kwargs={"prompt": QA_CHAIN_PROMPT},
    )

    result = qa_chain({"query": question})

My question is : “What about SpaceX ?”
The answer I get back is correct :
“There is no mention of SpaceX in the given context, so I don’t know the answer to that question. Thanks for asking!”

What I don’t understand is why the “source_documents” property contains documents with metadata.

I expected to find the “source_document” property empty since these documents don’t contain information about the question.

So my questions are :

  • Am I missing something?
  • How to handle this scenario when you want show “source_documents” but in this case are not related to the question ?

Thanks
Luca