I dont quite get, how ColBERT is different from chunking on token level.
Can anyone explain this to me?
The problem with chunking on token level has the big drawback, that context is lost. Why is that not a problem with ColBERT?
@kuephi Because ColBERT uses dense vectors representation to contextualize the documents. Instead of generating one semantic vector for the entire document, you generate one semantic vector for each token in the document. If a vectored token appears more than one time in the same document, this process ill take advantage, by reducing the time of search for this specific token. In other hand if a document has a thousand tokens that needs to be converted to a thousand dense vectors, this process can make token search feel slow (depends on the size of your document), and have deeper interactions between the text of the prompt and each document will give you a higher precision on token search. It’s a trade-off with pros and cons, that you maybe need to combine with other techniques to fit on your application.