BM25 Retrieve / Exercise 1

Course Q&A Retrieval Augmented Generation
1

Is there an error in the initialisation code with the global BM25 definitions, in this line:

BM25_RETRIEVER = bm25s.BM25(corpus=corpus)

It appears that passing in the corpus to the constructor causes the retrieve() function to return the raw text of the documents, rather than the indices of the documents.

Calling the constructor with no arguments appears to fix this error.

2

As a followup, there seems to be some general problem with the (non-editable) cell labeled
# Use these as a global defined BM25 retriever objects

The grader does not appear to run the 4 lines of code in that cell, and therefore fails subsequent (graded) cells with errors such as:

There was a problem compiling the code from your notebook, please check that you saved before submitting. Details:
name 'TOKENIZED_DATA' is not defined
4

Hi!

Thanks for flagging this. Could you please send your solution file to me? I will take a look. You can send it via direct message.

Thanks,
Lucas

5

Hi, have you had a chance to look at the solution file which I sent to you as a DM a couple of weeks ago? I just tried to re-submit my solution and it is still getting auto-graded as 0% even though in the Jupyter environment all the tests pass.

I am wondering if the assignment / starter code has been tweaked now and if so, how do I wipe my work-in-progress and just get a fresh start on this assignment?

6

Hi @sinis,

I’m so sorry! I missed your direct message. I’ll answer you privately.

7

any one can help in assignment -C1M2 .i have tried a lot -def bm25_retrieve(query, top_k=5):
import hashlib

def hash_tokens(tokens):
    flat_tokens = [str(t) for sublist in tokens for t in (sublist if isinstance(sublist, list) else [sublist])]
    return hashlib.md5(" ".join(flat_tokens).encode()).hexdigest()

tokenized_query = bm25s.tokenize([query])[0]
retrieved_docs = BM25_RETRIEVER.retrieve(tokenized_query, k=top_k)

token_hash_to_index = {
    hash_tokens(tokens): idx for idx, tokens in enumerate(TOKENIZED_DATA)
}

indices = []
for tokens in retrieved_docs:
    tokens_list = tokens.tolist() if isinstance(tokens, np.ndarray) else tokens
    doc_hash = hash_tokens(tokens_list)
    index = token_hash_to_index.get(doc_hash)
    if index is not None:
        indices.append(index)

return indices

.Any one have another code Expected output

[752, 673, 289, 626, 43]