Is the description Step 1 “Compute distances between all document vectors” correct? Calculating every pairwise distance would require roughly O(n^2) comparisons.
Should it be “Represent each document or document chunk as an embedding vector”?
Is my understanding corret?
it can be done both ways @Vivek_Muruganantham as per what I know, but the more preferred way is document chunks especially when creating a rag architecture as document chunking gives high precision to a specific topic matching, also if model have token limits document chunks will focus more on input to processing information and lastly any retrieval query against a small chunks gives a much higher similarity score than compared to matching against 100-page document.
Although document level chunking is also done in case of classification on a much global level, or generic level where query might be more general categories related to a specific task like retail, banking, stocking to economy, education to book vocabulary.