feat: add IBigSegmentStore interface + Redis and DynamoDB stores#536
feat: add IBigSegmentStore interface + Redis and DynamoDB stores#536beekld wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default mode and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a3dfc38. Configure here.
| it != item.end()) { | ||
| for (auto const& ref : it->second.GetSS()) { | ||
| excluded.emplace_back(ref); | ||
| } |
There was a problem hiding this comment.
Unvalidated GetSS() silently drops membership data on type mismatch
Medium Severity
The GetSS() calls on the included and excluded attributes are not validated for type correctness. AWS SDK's GetSS() silently returns an empty vector when the attribute is a non-SS type (S, N, BOOL, etc.), just as GetS() returns "" for non-String types. Since DynamoDB enforces that SS attributes must have at least one element, an empty result from GetSS() when the attribute key exists in the item indicates a type mismatch. The GetN() call for synchronizedOn at line 131 correctly validates for empty, but these GetSS() calls do not, causing silent membership data loss instead of surfacing an error.
Triggered by learned rule: DynamoDB source: validate AttributeValue type before using GetS()
Reviewed by Cursor Bugbot for commit a3dfc38. Configure here.


Ticket: SDK-2363 · Follows #534
Summary
Adds the public
IBigSegmentStoreinterface, theMembership/StoreMetadatavalue types, and concrete DynamoDB + Redis implementations. Schema strings match what the Relay Proxy writes.Design notes
synchronizedOnparsing. Stored as DynamoDB N / Redis string. Both stores reject malformed values (non-numeric strings, wrong DynamoDB attribute type) rather than silently returning 0, matching the existingdynamodb_source.cpprow validation.Not in scope
BigSegmentsBuilderconfig plumbing.BigSegmentStoreWrapper(LRU cache + staleness polling) and the hashing path.rules.cppbig-segments TODO.Test plan
Membershipunit tests pass.RedisBigSegmentStoreintegration tests pass against Redis 7 (docker).DynamoDBBigSegmentStoreintegration tests pass against DynamoDB Local.Note
Medium Risk
Introduces new public Big Segments store API and two new persistence integrations (Redis/DynamoDB), which could affect consumers and runtime behavior if schema/connection handling is incorrect. Risk is moderated by being additive and covered by new unit/integration tests, but it touches external storage and parsing/validation paths.
Overview
Adds a new public Big Segments persistence API via
IBigSegmentStoreplus value typesMembershipandStoreMetadata, including semantics like inclusion wins and explicit error propagation.Implements concrete read-only Big Segments stores for DynamoDB (
DynamoDBBigSegmentStore) and Redis (RedisBigSegmentStore), wires them into their respective CMake builds, and adds schema constants/validation (notably strictsynchronizedOnparsing and prefix-scoped keys/namespaces) with new unit and integration tests covering empty stores, include/exclude resolution, prefix isolation, malformed metadata, and unreachable Redis errors.Reviewed by Cursor Bugbot for commit a3dfc38. Bugbot is set up for automated code reviews on this repo. Configure here.