why does index give a different number of records/docs?
Loader gives 1000 (as we know, 1000 rows):
docs = loader.load()
len(docs)
Index gives 1210:
index = VectorstoreIndexCreator(
vectorstore_cls=DocArrayInMemorySearch
).from_loaders([loader])
index.vectorstore.doc_index.num_docs()
but db gives 1000 (like the number of rows):
db = DocArrayInMemorySearch.from_documents(
docs,
embeddings
)
db.doc_index.num_docs()
and retriever gives 1000 (as I would expect since it’s using db):
retriever = db.as_retriever()
retriever.vectorstore.doc_index.num_docs()