The formula in 'get_chunks_fixed_size_with_overlap' of C1M3_Ungraded_Lab_2.ipynb might not be correct

Course Q&A Retrieval Augmented Generation
, ,
1

Here is what I understand how chunks of fixed size with overlap is supposed to work. Take chunk_size = 100, chunk_overlap = 0.05 as an example.
The ending index is exclusive
chunk 0: 0 - 100
chunk 1: 95 - 195
chunk 2: 190 - 290
chunk 3: 285 - 385

It will keep advance. The range step size should (chunk_size - overlap_int). It’s 95 in this case. IMO, here is what the formula supposed to be:

for i in range(0, len(text_words), (chunk_size - overlap_int)):
chunk_words = text_words[i: i + chunk_size]

Let me know if that makes sense. Thanks.

2

Forgot one more comment.
I am seeing a lot of businesses are using langchain_text_splitters.RecursiveCharacterTextSplitter to chunk documents like

text_splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
separators=[“\n\n”, “\n”, " ", “”]
)


I am using `RecursiveCharacterTextSplitter` too. Is that good idea to just use `RecursiveCharacterTextSplitter` in place of our hand-made mix_chunker?   Thanks