Provide a way to continue chatting with a thread after hitting context length limit possibly by dropping/consolidating old message.

[sangee2004]

Once i hit a context length limit in a thread , then the whole thread will become unusable . Even after a restart script , i keep getting the same error.

There should be way to get the thread out of this failing mode and to start succeeding again.

Can this possibly be done by dropping/consolidating old messages?

[cjellick]

I spoke with Darren about this. Gptscript already garbage collects but it can't catch all scenarios.

• It definitely won't catch scenarios where a single message would blow out the context.
• It probably wont catch/handle secearnios where two or three large messages in succession cause a blowout.

Darren: Yeah, so there is code that will drop older messages. I believe it's only for chat: true tools. But it's also hard coded to 128k window. It's super dumb logic. Like I just do number of chars/3 as a rough guess at the token count and then if that is like >100k we start dropping messages. It's weird and not at all exact, because it's impossibly hard to be exact.

Me: would it be dumb to have an "after the fact" explicit cleanup? Like we detect when we get a context limit error and then go through a routine to more agressively prune?

Darren: That would probably be reasonable, the downside would be that once you fill up the window basically every new request after that will be two requests (fail, prune)

So, something like that seems like a good enhancement, but needs thought through.

[cjellick]

I'd also just like an escape hatch if we do get into this. Having the restart script button there and it not work is the worst scenario.

[g-linville]

This is ready for testing. I recommend reading the PR description so you can understand what changes I made that need to be tested: gptscript-ai/gptscript#832

[sangee2004]

Tested with latest electron build - 9d4616691 and using gptscript version - v0.0.0-dev+eaaf0cdb

I have not been able to hit the context length limit exceeded error anymore.

Following test was done to validate the fact that we will now trying to prevent the context length limit exceeded error by providing some directive to users.

1. Chat with Tildy
2. Upload a large json file to workspace.

ls -ltr ~/Downloads/mytestdocuments/test1.json 
-rw-r--r--  1 sangeethahariharan  staff  2480380 Sep  4 11:36 /Users/sangeethahariharan/Downloads/mytestdocuments/test1.json

3. Ask to read this large json file
4. We get the following response from LLM to prevent users from getting into the context length limit exceeded error

The contents of test1.json are too long to display all at once. How about I extract specific information for you? Let me know what you're looking for, and I'll get it for you!