Persist chats through an appendable store - #7554
Draft
candrewlee14 wants to merge 2 commits into
Draft
candrewlee14 wants to merge 2 commits into
candrewlee14 wants to merge 2 commits into
Conversation
`Chat.Persisted` re-encoded the entire conversation on every save, making persistence quadratic in the number of turns and stalling other fibers while it ran. Encode each message once and reuse that encoding across saves. Message identifiers are now stamped by replacing messages rather than mutating them in place, so messages in history are immutable values and the reused encodings cannot go stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: a7ef42f The changes in this PR will be included in the next version bump. This PR includes changesets to release 30 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
`Chat.Persisted` stored a conversation as one value and rewrote all of it on every save. Persisting a chat was therefore quadratic in the number of turns, and a chat could only be loaded whole - no last-N, no listing, no retention. Introduce `Chat.ChatStore`: write a range, read a range, read backwards, list, remove, clean up. The interface is deliberately narrow so that every backend can implement all of it; richer querying belongs in a backend's own client. `layerStoreMemory` implements it in memory, and `layerStoreBacking` implements it over an existing `BackingPersistence` so current applications keep working. Message identifiers are now stamped by replacing messages rather than mutating them in place, which is also what lets a save find the first message it has to write by comparing identity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
candrewlee14
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Chat.Persistedstores a conversation as a single value under one key and rewrites all of it on every save. Two consequences:BackingPersistenceStoreisget/set/remove/clearand cannot append or enumerate.Change
Chat.Persistedis now backed by a newChat.ChatStore:writereplaces a chat's messages fromfromonwards. Appending isfromequal to the number of messages already stored; rewriting history — what summarizing or redacting a conversation produces — is a smallerfrom. Afrompast the end fails rather than leaving a hole, and afromthat does not match what the caller last saw is how two writers racing on one chat is caught instead of silently losing one.Values are opaque to the store:
Chatencodes and decodes them, so a store never sees aPrompt. This is the same splitPersistedQueueuses withPersistedQueueStore.The interface is deliberately narrow, so that every backend can implement all of it. Anything richer — searching message content, filtering by metadata — belongs in a backend's own client, where SQL can be SQL, rather than in an abstraction only some backends could honour.
A save now finds the first message it has to write by comparing history against what it last stored, by identity. An ordinary turn writes the two messages it added. That requires messages in history to be immutable, which they nearly were: the exception was
saveChat, which stamped message identifiers by mutatingmessage.optionsin place ((message.options as any)[Persistence.key] = ...) and then wrote the object it had just mutated back into theRef. That is now done by replacement, so a message already handed to a caller never changes underneath them, and theas anyis gone.Layers
Chat.layerStoreMemory— in-memory, process-local.Chat.layerStoreBacking— aChatStoreover an existingBackingPersistence. Every write reads, applies, and writes the whole chat back, so it costs what today costs; it exists so an application already onBackingPersistencekeeps working. Listing is served from an index kept alongside the chats, since the backing interface cannot enumerate keys.Chat.layerPersistednow requiresChatStoreinstead ofBackingPersistence. Existing wiring becomes:Not in this PR
Native
layerStoreSqlandlayerStoreRedis. Those are where the byte amplification actually goes away — a messages table with an index on(chat_id, seq), or a Redis list withLRANGE— and they are a few hundred lines each, followingmakeStoreSql/makeStoreRedisinPersistedQueue. I would rather agree on the interface before writing them; I am happy to follow up with both.Verification
Chattests now run against both stores, unchanged in intent.Chattests cover reloading a chat from the store and continuing it, and truncating a chat.Chat.test.ts: 26 tests.packages/effect/test/unstable/ai: 24 files, 1076 passed / 44 skipped. Doctests forChat.tspass.tsc -b tsconfig.json,oxlint, anddprint checkare clean.