Skip to content

Commit 76a99d3

Browse files
committed
Unify SQL JSON and CSharp queries through a validated AST
1 parent cabf439 commit 76a99d3

17 files changed

Lines changed: 695 additions & 42 deletions

File tree

‎README.md‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,23 @@ page.ThrowIfFail();
131131

132132
Q1 supports projections and aliases, scalar parameters, comparisons, `AND`/`OR`/`NOT`, `IN`, `IS NULL`, `IS MISSING`, `ORDER BY`, `LIMIT` and `EXPLAIN`. Identifiers containing dots need double quotes. `id` and `revision` refer to canonical document identity/revision. Parameters use `@name`. JSON numbers use the decimal scalar policy. SQL is read-only; unsupported statements fail explicitly.
133133

134-
Queries require a matching point/equality index or explicit `AllowFullScan`. Scans, parser depth, tokens, rows, bytes and execution time have budgets. Cursor tokens bind the principal, policy epoch, schema, query, node identity, read generation and current storage cut. Writes, policy changes or snapshot installation can expire a cursor; compaction preserves its cut. Sensitive predicates and sorting require a field-use grant. Returned documents omit protected paths, including classified values nested in arrays.
134+
SQL, version 1 JSON AST and the C# expression builder normalize into the same typed AST and use the same planner, permissions and execution path. `GET /v1/query/capabilities` (SDK: `QueryCapabilitiesAsync`) reports that contract and the configured limits. JSON requests use `POST /v1/query/ast` or `QueryAstAsync`. See the [Q1 protocol and expression subset](docs/design/query-q1.md).
135+
136+
```csharp
137+
var query = KeyLoadQuery<Order>.From(partition, "orders")
138+
.Where(order => order.Number == 1m)
139+
.OrderBy(order => QueryFunctions.DocumentId(order))
140+
.Select(order => new { Id = QueryFunctions.DocumentId(order), order.Status })
141+
.Take(20);
142+
var typedPage = await client.QueryAsync(query);
143+
typedPage.ThrowIfFail();
144+
145+
public sealed record Order(decimal Number, string Status);
146+
```
147+
148+
The builder supports scalar comparisons, Boolean composition, constant-array/list `Contains`, null/missing markers, field projections and ordering. It translates expression trees without invoking application delegates or getters. Unsupported methods and lossy casts fail explicitly. It has independent immutable query branches and provides no implicit client evaluation.
149+
150+
Queries require a matching point/equality index or explicit `AllowFullScan`. Scans, parser depth, nodes, parameters, rows, bytes and execution time have budgets. Cursor tokens bind the principal, policy epoch, schema, normalized query, node identity, read generation and persisted collection data version. Equivalent SQL/JSON/C# forms can continue the same cursor. A write to that collection, a principal policy change or snapshot installation can expire it; catalog heartbeats, unrelated collection writes and compaction preserve its cut. Sensitive predicates and sorting require a field-use grant. Returned documents omit protected paths, including classified values nested in arrays.
135151

136152
Search accepts typed vector spaces and explicit text/vector fields. Both branches use one authorized read cut. Exact vector scores and BM25 ranks are combined with weighted RRF using one-based ranks. The managed ANN and graph retrieval extensions are tracked separately.
137153

‎docs/design/query-q1.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Q1 query contract
2+
3+
Protocol 1 exposes read-only queries over one collection in one atomic partition. SQL and the C# expression builder lower into AST version 1. JSON supplies that AST directly. All three adapters enter the same admission gate, structural validator, authorization binder, access-path planner and executor. Unsupported constructs produce an explicit error.
4+
5+
`GET /v1/query/capabilities` requires an authenticated caller and reports the dialect, AST version, supported predicate families and configured budgets. `POST /v1/query` accepts SQL; `POST /v1/query/ast` accepts `AstQueryRequest` from `KeyLoad.Abstractions`.
6+
7+
```json
8+
{
9+
"partition": {
10+
"tenantId": "acme",
11+
"databaseId": "shop",
12+
"transactionDomainId": "order-processing",
13+
"partitionKey": "customer-42"
14+
},
15+
"query": {
16+
"collection": "orders",
17+
"alias": null,
18+
"projection": [{ "path": "/@id", "alias": "id" }, { "path": "/status", "alias": "status" }],
19+
"filter": {
20+
"kind": "comparison",
21+
"left": { "kind": "field", "path": "/number" },
22+
"operator": "=",
23+
"right": { "kind": "value", "value": 1 }
24+
},
25+
"order": [{ "path": "/@id", "descending": false }],
26+
"limit": 20,
27+
"explain": false
28+
},
29+
"parameters": null,
30+
"allowFullScan": false,
31+
"cursor": null,
32+
"astVersion": 1
33+
}
34+
```
35+
36+
Fields use bounded JSON pointers; `/@id` and `/@revision` address canonical metadata. SQL aliases are resolved before normalization and do not change the normalized plan. Scalars are strings, booleans, decimal-policy numbers or JSON null. Arrays and objects are not scalar parameters or operands. Parameter operands use `{"kind":"parameter","name":"status"}` and bind through the request's scalar parameter dictionary.
37+
38+
| Construct | Semantics and cost | Required permission |
39+
| --- | --- | --- |
40+
| `=`, `!=`, `<>`, `<`, `<=`, `>`, `>=` | Same scalar types; null or missing yields unknown. Equality can select a point/composite equality index; remaining checks cost one comparison per candidate. | `Query` and `DocumentsRead`; protected input needs its field-use grant. |
41+
| `AND`, `OR`, `NOT` | Three-valued logic; only true enters the result. Cost follows the bounded predicate tree. | Every input field is authorized before candidate access. |
42+
| `IN` / `NOT IN` | Bounded scalar list, with comparison's null/missing behavior. | Field-use permission on the input and any field operands. |
43+
| `IS NULL`, `IS MISSING` | Explicit distinction between a present JSON null and an absent path. Negation negates the corresponding test. | Field-use permission on the tested field. |
44+
| Projection | Canonical identity/revision plus selected fields or `*`; omitted classified values remain omitted or null in a selected alias. | Row scope and current field-read projection. |
45+
| `ORDER BY` | Ordered-key scalar policy, followed by canonical ID as a deterministic tie break; bounded in-memory sort. | Protected sort fields need field-use grants. |
46+
| `LIMIT` | Positive result count within configured bounds. | Ordinary query permission. |
47+
| `EXPLAIN` | Reports the authorized access path, atomic partition and scan budget. | The same binding permissions as execution. |
48+
49+
Full scans require explicit opt-in. Candidate count, result bytes, request bytes, predicate depth/nodes, parameter count, projection count, sort keys, execution time and simultaneous queries are bounded. This profile still requires the broader tenant memory/CPU governor and batch executor qualification recorded in the implementation tracker.
50+
51+
Each page uses one consistent storage read gate. Its cursor is signed and binds the normalized query, principal/policy epoch, schema, node/read generation, original cut and collection data version. Document changes atomically advance that version, including ACL changes. Unrelated catalog or collection writes preserve the cursor; a change to its source rejects continuation with `CursorExpired`. Principal revocation/policy change is rechecked before every page. Replica installation changes the read generation; compaction preserves it. A cursor is a short-lived continuation contract, not an indefinitely retained MVCC snapshot.
52+
53+
The C# builder accepts mapped properties, constant scalar comparisons, Boolean composition, field ordering, anonymous/member projections, and `Contains` over captured constant arrays or plain lists. `QueryFunctions.DocumentId` and `DocumentRevision` expose metadata. `QueryFunctions.IsNull` and `IsMissing` preserve Q1's explicit null/missing distinction; equality against a null literal lowers to `IS NULL`. Predicates follow canonical Q1 semantics, rather than executing a compiled CLR delegate over deserialized objects.
54+
55+
Property names follow the protocol's JSON naming policy and `JsonPropertyName` attributes. Captured fields may supply constants; application property getters, arbitrary method calls, custom operators and lossy numeric casts are rejected. No user assembly is executed by the server. Builder branches hold independent query contexts. More SQL features, query functions, vector attachments, session tokens, distributed plans and the PostgreSQL differential baseline remain tracked extensions.

‎docs/implementation/kernel-qualification.json‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
"restore": "locked-mode passed",
88
"build": "passed with zero warnings",
99
"unitTests": {
10-
"passed": 42,
10+
"passed": 52,
1111
"failed": 0
1212
},
1313
"recoveryTests": {
@@ -29,7 +29,7 @@
2929
"evidence": "CI retains artifacts/qualification/crash-trials-*.jsonl"
3030
},
3131
"rf3Integration": {
32-
"passed": 3,
32+
"passed": 4,
3333
"failed": 0,
3434
"scenarios": [
3535
"leader process kill",
@@ -42,7 +42,9 @@
4242
"unsigned peer RPC rejection",
4343
"empty-replica native snapshot catch-up",
4444
"command outcome preserved across snapshot installation",
45-
"topic events, group checkpoints and inbox receipts preserved across snapshot installation"
45+
"topic events, group checkpoints and inbox receipts preserved across snapshot installation",
46+
"SQL, JSON AST and C# query parity through authenticated HTTP",
47+
"SQL-to-JSON cursor continuation and protected predicate rejection"
4648
]
4749
},
4850
"durability": [
@@ -52,9 +54,9 @@
5254
"powerLoss": "not qualified",
5355
"endurance72Hours": "pending",
5456
"crossPlatformCI": {
55-
"qualifiedCommit": "2bc7ccc93997a36555c71553403d5539d64b90d7",
56-
"run": "https://github.com/managedcode/KeyLoad/actions/runs/36885624099",
57+
"qualifiedCommit": "cabf43986dfbf6b8f78e7aaa5fbfa5e4fc62321f",
58+
"run": "https://github.com/managedcode/KeyLoad/actions/runs/36893476976",
5759
"platforms": ["Linux", "macOS", "Windows"],
58-
"currentTopicSubscriptionChanges": "pending"
60+
"currentQueryAdapterChanges": "pending"
5961
}
6062
}

‎docs/implementation/status.json‎

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -333,6 +333,8 @@
333333
"title": "Query capability manifest і AST version",
334334
"status": "in_progress",
335335
"evidence": [
336+
"src/KeyLoad.Abstractions/QueryAst.cs",
337+
"docs/design/query-q1.md",
336338
"src/KeyLoad.Query/QueryAst.cs",
337339
"src/KeyLoad.Query/SqlParser.cs",
338340
"src/KeyLoad.Query/QueryEngine.cs",
@@ -364,6 +366,8 @@
364366
"status": "in_progress",
365367
"evidence": [
366368
"src/KeyLoad.Query/QueryAst.cs",
369+
"src/KeyLoad.Query/QueryValidation.cs",
370+
"src/KeyLoad.Client/KeyLoadQuery.cs",
367371
"src/KeyLoad.Query/SqlParser.cs",
368372
"src/KeyLoad.Query/QueryEngine.cs",
369373
"tests/KeyLoad.UnitTests/SecurityAndQueryTests.cs"
@@ -391,28 +395,44 @@
391395
},
392396
"KL-051": {
393397
"title": "SQL/JSON/C# equivalence",
394-
"status": "pending",
395-
"evidence": []
398+
"status": "in_progress",
399+
"evidence": [
400+
"src/KeyLoad.Abstractions/QueryAst.cs",
401+
"src/KeyLoad.Query/QueryValidation.cs",
402+
"src/KeyLoad.Client/KeyLoadQuery.cs",
403+
"tests/KeyLoad.UnitTests/QueryAdapterTests.cs",
404+
"tests/KeyLoad.IntegrationTests/ClusterTests.cs"
405+
]
396406
},
397407
"KL-052": {
398408
"title": "Ранній admission control",
399409
"status": "in_progress",
400410
"evidence": [
401411
"src/KeyLoad.Query/QueryAst.cs",
412+
"src/KeyLoad.Query/QueryValidation.cs",
413+
"tests/KeyLoad.UnitTests/QueryAdapterTests.cs",
402414
"src/KeyLoad.Query/SqlParser.cs",
403415
"src/KeyLoad.Query/QueryEngine.cs",
404416
"tests/KeyLoad.UnitTests/SecurityAndQueryTests.cs"
405417
]
406418
},
407419
"KL-053": {
408420
"title": "Session facade та independent query contexts",
409-
"status": "pending",
410-
"evidence": []
421+
"status": "in_progress",
422+
"evidence": [
423+
"src/KeyLoad.Client/KeyLoadQuery.cs",
424+
"src/KeyLoad.Query/QueryEngine.cs",
425+
"tests/KeyLoad.UnitTests/QueryAdapterTests.cs"
426+
]
411427
},
412428
"KL-054": {
413429
"title": "SQL differential і metamorphic suite",
414-
"status": "pending",
415-
"evidence": []
430+
"status": "in_progress",
431+
"evidence": [
432+
"tests/KeyLoad.UnitTests/QueryAdapterTests.cs",
433+
"tests/KeyLoad.UnitTests/SecurityAndQueryTests.cs",
434+
"docs/design/query-q1.md"
435+
]
416436
},
417437
"KL-055": {
418438
"title": "GraphScope / GraphRetriever / GraphExpansion",
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
using System.Text.Json;
2+
using System.Text.Json.Serialization;
3+
4+
namespace KeyLoad.Query;
5+
6+
public sealed record SelectQuery(string Collection, string? Alias, Selection[] Projection, Predicate? Filter,
7+
Ordering[] Order, int Limit, bool Explain = false);
8+
public sealed record Selection(string Path, string Alias);
9+
public sealed record Ordering(string Path, bool Descending);
10+
[JsonPolymorphic(TypeDiscriminatorPropertyName = "kind")]
11+
[JsonDerivedType(typeof(FieldOperand), "field")]
12+
[JsonDerivedType(typeof(ValueOperand), "value")]
13+
[JsonDerivedType(typeof(ParameterOperand), "parameter")]
14+
public abstract record Operand;
15+
public sealed record FieldOperand(string Path) : Operand;
16+
[method: JsonConstructor]
17+
public sealed record ValueOperand(JsonElement Value) : Operand
18+
{
19+
public ValueOperand(object? value) : this(JsonSerializer.SerializeToElement(value, JsonDefaults.Options)) { }
20+
}
21+
public sealed record ParameterOperand(string Name) : Operand;
22+
[JsonPolymorphic(TypeDiscriminatorPropertyName = "kind")]
23+
[JsonDerivedType(typeof(Comparison), "comparison")]
24+
[JsonDerivedType(typeof(Logical), "logical")]
25+
[JsonDerivedType(typeof(Negation), "not")]
26+
[JsonDerivedType(typeof(NullTest), "nullTest")]
27+
[JsonDerivedType(typeof(InPredicate), "in")]
28+
public abstract record Predicate;
29+
public sealed record Comparison(Operand Left, string Operator, Operand Right) : Predicate;
30+
public sealed record Logical(Predicate Left, string Operator, Predicate Right) : Predicate;
31+
public sealed record Negation(Predicate Inner) : Predicate;
32+
public sealed record NullTest(Operand Value, bool Negated, bool Missing) : Predicate;
33+
public sealed record InPredicate(Operand Value, Operand[] Values, bool Negated) : Predicate;
34+
35+
public sealed record AstQueryRequest(PartitionRef Partition, SelectQuery Query, Dictionary<string, JsonElement>? Parameters = null,
36+
bool AllowFullScan = false, string? Cursor = null, int AstVersion = 1);
37+
public sealed record QueryCapabilityManifest(int ProtocolVersion, int AstVersion, string SqlDialect, string Scope,
38+
string NumericPolicy, string MissingPolicy, string[] Adapters, string[] Predicates, int MaxRows, int MaxCandidates,
39+
int MaxBytes, int MaxDepth, bool FullScanRequiresOptIn, bool ReadOnly);

‎src/KeyLoad.Client/KeyLoadClient.cs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
using System.Net.Http.Headers;
22
using System.Net.Http.Json;
33
using System.Text.Json;
4+
using KeyLoad.Query;
45
using ManagedCode.Communication;
56

67
namespace KeyLoad.Client;
@@ -63,6 +64,12 @@ public Task<Result<CommitReceipt>> CommitProcessingAsync(ProcessingRequest reque
6364
=> Send<MessageInspection?>("/v1/queues/inspect", request, false, null, cancellationToken);
6465
public Task<Result<QueryPage>> QueryAsync(QueryRequest request, CancellationToken cancellationToken = default)
6566
=> Send<QueryPage>("/v1/query", request, false, null, cancellationToken);
67+
public Task<Result<QueryPage>> QueryAstAsync(AstQueryRequest request, CancellationToken cancellationToken = default)
68+
=> Send<QueryPage>("/v1/query/ast", request, false, null, cancellationToken);
69+
public Task<Result<QueryPage>> QueryAsync<T>(KeyLoadQuery<T> query, bool allowFullScan = false, string? cursor = null, CancellationToken cancellationToken = default)
70+
=> QueryAstAsync(query.ToRequest(allowFullScan, cursor), cancellationToken);
71+
public Task<Result<QueryCapabilityManifest>> QueryCapabilitiesAsync(CancellationToken cancellationToken = default)
72+
=> Send<QueryCapabilityManifest>("/v1/query/capabilities", null, false, null, cancellationToken);
6673
public Task<Result<RankedDocument[]>> SearchAsync(SearchRequest request, CancellationToken cancellationToken = default)
6774
=> Send<RankedDocument[]>("/v1/search", request, false, null, cancellationToken);
6875
public Task<Result<GraphTraversal>> TraverseAsync(TraverseRequest request, CancellationToken cancellationToken = default)

0 commit comments

Comments
 (0)