Majority of examples in the course look like they’re within the 4096 token limit. How would this work if I wanted to give GPT a large piece of text, chunked up into paragraphs, where the model retains context from each paragraph. Specific example I am thinking of is converting long internal operating procedures into customer facing guides.
I can think of a few different implementations but it all kind boils down to iterative summarization of paragraphs while using get_completion_from_messages() for context. You could build various stages or levels e.g. summarize paragraphs, then every 5 paragraphs - get a summary of those, etc
Hello @nyclh ,
Welcome to the community!!
There is a word limit of 4096 tokens for the “Davinci” and “Curie” models.
So what you can do is :
For complex files, I would suggest going for data pre-processing.
Then, Chunking : breaking up into smaller chunks.
Then after it’s done performing operations, concatenate the result.
You can use continuation prompts such that chatgpt can understand the workflow of your code and inputs.
With regards,
Nilosree Sengupta
As a follow-up (sorry for hijacking conversation)…
Let’s say I wanna buy a condo from the chatbot. I’ll ask tons of questions with all kinds of negotiations. There will be a lot of key points to be kept in the context. Eventually token limit will be reached no matter how condensed it has been made. Thoughts on this?
Generating a running summary, as pointed by @Eric_Townes is a good place to start. Here are 2 more approaches where summarization may not be required:
- If quality of responses should be high, switch to a bigger model once the token limit is reached. After switching to the largest model your company can pay for, get a human involved.
- If earlier parts of the conversation are unimportant, remove them from the chat history to get more space for more recent sentences.