|
| 1 | +# ADR-052: bounded TimeSeries latest and raw aggregates |
| 2 | + |
| 3 | +Status: Accepted; source joined, exact-SHA qualification and remaining gates pending. |
| 4 | +Owner: KeyLoad lead integration owner. Date: 2026-10-02. |
| 5 | +Related: KL-025; REQ/AC-SERIES-008..012; REQ-STORAGE-013 / |
| 6 | +AC-RANGE-REV-001..003; ADR-004/005/010/014/035/036/041/050. |
| 7 | + |
| 8 | +## Decision and rationale |
| 9 | + |
| 10 | +Add typed read-only latest, whole-range statistics and dense fixed-window |
| 11 | +statistics over the canonical persisted samples. Keep the existing inclusive |
| 12 | +ReadSamples operation unchanged. Every new public request keeps its separate |
| 13 | +Orleans request/read actors; physical node-local storage and the existing RF3 |
| 14 | +authority/consistent-cut behavior remain the owners of records and reads. |
| 15 | + |
| 16 | +Use the published centrally pinned ManagedCode.TimeSeries 10.0.0 inside Core as |
| 17 | +a bounded temporary numeric accumulator. It does not replace KeyLoad storage, |
| 18 | +ordering, event deduplication, persisted authorization or replication. The owning |
| 19 | +source at published commit23632b9f8d49d5a7feb337caa6cbeb5b22d7575a was inspected; |
| 20 | +no dependency defect has been established and no consumer workaround is approved. |
| 21 | + |
| 22 | +Three native DoubleTimeSeriesSummer instances use Strategy.Sum/Min/Max, |
| 23 | +TimeSpan.MaxValue and maximum bucket count1. Pass each real sample timestamp and |
| 24 | +value in the canonical UTC/sequence order. Every representable UTC date tick is |
| 25 | +less than half TimeSpan.MaxValue, so native RoundUtc puts all values into one |
| 26 | +bucket. This avoids unordered multi-bucket summation, retains actual dates and |
| 27 | +uses native raw DataCount. Raw average is Sum/DataCount; the library's documented |
| 28 | +bucket Average operation is a different contract and is not used here. Empty |
| 29 | +windows need no native accumulator, and live accumulators do not retain samples. |
| 30 | + |
| 31 | +## Public contract |
| 32 | + |
| 33 | +| Operation | Input | Output / rules | |
| 34 | +|---|---|---| |
| 35 | +| Latest | ReadLatestSampleRequest(Partition,Set,SeriesId,AtOrBefore=null) | LatestSampleResult(SampleRecord? Sample); greatest UTC timestamp then greatest persisted sequence at/before optional inclusive cut; absent is null Sample; tags are projected by current persisted policy | |
| 36 | +| Aggregate | AggregateSamplesRequest(Partition,Set,SeriesId,From,UntilExclusive=null,MaxSamples=10000) | SampleAggregate(Count,Sum,Minimum,Maximum,Average); complete half-open [From,UntilExclusive), UTC comparisons; null end includes the representable maximum timestamp | |
| 37 | +| Windows | AggregateSampleWindowsRequest(Partition,Set,SeriesId,From,UntilExclusive,Width,MaxSamples=10000,MaxWindows=1000) | SampleAggregateWindowsResult(ImmutableArray<SampleAggregateWindow> Windows); each window has From, nullable UntilExclusive and SampleAggregate; dense ascending fixed-width UTC windows anchored at From, final window clamped | |
| 38 | + |
| 39 | +Empty aggregate: Count=0, Sum=0, extrema/average=null. Equal range bounds are valid |
| 40 | +empty ranges. Inverted range and nonpositive width give Validation. Finite input |
| 41 | +whose accumulated sum is nonfinite gives Validation with a fixed safe message, |
| 42 | +no partial response or mutations; a smaller following read remains healthy. |
| 43 | +Raw average uses event counts, including multiple samples at equal timestamps. |
| 44 | +Late committed data is visible on a new cut; event deduplication stays upstream. |
| 45 | + |
| 46 | +The internal null-end tick is DateTimeOffset.MaxValue.UtcTicks+1, which fits long; |
| 47 | +it is never constructed or serialized as a DateTimeOffset. Only a window ending |
| 48 | +after the greatest representable tick returns null UntilExclusive. Bucket count |
| 49 | +uses span/width and remainder, not overflowing span+width-1 arithmetic. End |
| 50 | +advance clamps by remaining span before addition. Width may exceed the range. |
| 51 | + |
| 52 | +Both caller caps must be positive and no larger than server MaxScanRecords / |
| 53 | +MaxResults; invalid caps give BudgetExceeded. Check the calculated window count |
| 54 | +against both caller and server bounds before allocation. A live matching sample |
| 55 | +beyond MaxSamples is charged lookahead and makes the whole aggregate fail |
| 56 | +BudgetExceeded; truncated statistics never count as success. |
| 57 | + |
| 58 | +One ReadExecutionBudget.CreateView wraps the original gated view. Principal and |
| 59 | +resource lookups, sample bytes and lookahead are charged exactly once. All new |
| 60 | +operations require current persisted SeriesRead, including empty results. Check |
| 61 | +cancellation/deadline during metadata, traversal, window fill and result writing. |
| 62 | +Check the exact complete JSON output under the same configured MaxBatchBytes. |
| 63 | +All result collections are immutable and retain ADR-041 array serialization. |
| 64 | +The new HTTP and MCP canonical routes reserve the existing heavy-read ingress |
| 65 | +working set under AC-SERIES-011, sharing capacity with query/search/range reads. |
| 66 | +HttpAdmissionGovernor and its new TimeSeriesHttpAdmissionTests have the same root |
| 67 | +integration owner; existing admission ceilings and release semantics are unchanged. |
| 68 | + |
| 69 | +## Shared reverse storage contract |
| 70 | + |
| 71 | +Add IKeyValueView.VisitReverseRange with the identical prefix, exclusive lower |
| 72 | +afterKey, exclusive upper untilKey, observer, cancellation and borrowed visitor |
| 73 | +parameters as VisitRange. Only order changes to descending encoded keys. |
| 74 | +Parameterize the existing private range merge and its native cursors. Preserve |
| 75 | +forward defaults, staged replacement/insert/tombstone precedence and counters. |
| 76 | + |
| 77 | +The baseline uses pinned ZoneTree's CreateReverseIterator(NoRefresh); seek to |
| 78 | +the lesser of caller upper bound and binary prefix successor. Empty/all-FF |
| 79 | +prefix has no successor and uses native reverse start. Filter exclusive bounds |
| 80 | +explicitly and stop below the lower/prefix boundary. Staged data uses the native |
| 81 | +SortedSet.GetViewBetween(...).Reverse enumerator, not a buffered LINQ reverse. |
| 82 | +Do not buffer/copy the full baseline, transaction changes or series history. |
| 83 | +If seek throws, dispose the acquired native iterator before rethrowing. |
| 84 | + |
| 85 | +Charge matching examined key/value bytes before callback or live lookahead. |
| 86 | +Visitor=false stops before further cursor advance. Every failure releases the |
| 87 | +native cursors and original storage gate. Latest stops after its first live |
| 88 | +committed sample; its real-store logical baseline-entry delta is one, independent |
| 89 | +of history size. Counters are logical work, not physical disk/RSS measurements. |
| 90 | + |
| 91 | +```mermaid |
| 92 | +flowchart LR |
| 93 | + Client[Typed SDK or official MCP client] --> Request[Unique Orleans request grain] |
| 94 | + Request --> Read[Authorized consistent read grain] |
| 95 | + Read --> Gate[Node local store gate and one budgeted view] |
| 96 | + Gate --> Latest[Native descending first live sample] |
| 97 | + Gate --> Raw[Ascending bounded raw samples] |
| 98 | + Raw --> Native[Published ManagedCode native statistics] |
| 99 | + Native --> Windows[Dense capped UTC windows] |
| 100 | + Latest --> Result[Exact bounded typed JSON output] |
| 101 | + Native --> Result |
| 102 | + Windows --> Result |
| 103 | +``` |
| 104 | + |
| 105 | +## Ordered implementation and ownership contract |
| 106 | + |
| 107 | +1. TASK-SERIES-CONTRACT-R8, root: accept this ADR, TimeSeries/StorageRecovery/ |
| 108 | + ResourceExecution maps, acceptance and task graph before source. Root alone |
| 109 | + owns Abstractions new TimeSeries DTOs and shared IKeyValueView, Core csproj, |
| 110 | + BudgetedReadView, ZoneTreeStore/ZoneTreeReadView/ZoneTreeTransaction forwarding |
| 111 | + and new public Core/Features/TimeSeries/TimeSeriesReadOperations.cs. Public |
| 112 | + static extension operations ReadLatestSample/AggregateSamples/ |
| 113 | + AggregateSampleWindows accept engine, principal, typed request, cancellation; |
| 114 | + they do not enlarge the existing oversized DatabaseEngine partial type. |
| 115 | +2. TASK-SERIES-REVERSE-W8, cheaper capable worker: author genuine new |
| 116 | + UnitTests/Features/StorageRecovery/ReverseRangeTests, |
| 117 | + ReverseTransactionRangeTests and ReverseRangeResourceTests first. Own only |
| 118 | + ZoneTreeRangeReader, ZoneTreeBaselineCursor, ZoneTreeStagedCursor and |
| 119 | + ZoneTreeRangeBounds private files. Extend Visit with final optional |
| 120 | + bool reverse=false. No shared forwarding/interface or forward test edits. |
| 121 | +3. TASK-SERIES-CORE-W8, disjoint cheaper capable worker: author new real |
| 122 | + UnitTests/Features/TimeSeries/SampleLatestTests, SampleAggregateTests, |
| 123 | + SampleAggregateWindowTests, SampleAggregateBudgetTests, |
| 124 | + SampleAggregateAuthorizationTests and private SampleAggregateTestData first. |
| 125 | + Own only new private Core/Features/TimeSeries/SampleReadScope, |
| 126 | + SampleReadKeys, SampleLatestReader, SampleAggregateAccumulator, |
| 127 | + SampleAggregateReader, SampleAggregateWindowReader and SampleWindowAccumulator |
| 128 | + files. Reader signatures are Read(DatabaseEngine,IKeyValueView,string |
| 129 | + principalId,typed request,ReadExecutionBudget); supplied view is already |
| 130 | + budgeted, so never wrap/charge it a second time. Root owns public facade. |
| 131 | +4. TASK-SERIES-SURFACE-R8, root: add matching Client TimeSeries SDK slice, HTTP |
| 132 | + routes /v1/series/latest,/aggregate,/windows, append GrainReadKind values27..29 |
| 133 | + preserving0..26, typed read dispatch, official MCP names |
| 134 | + keyload_series_latest/keyload_series_aggregate/keyload_series_windows, |
| 135 | + descriptions, input/output schema registrations and discovery expectations. |
| 136 | +5. TASK-SERIES-RF3-W8, cheaper capable worker after a slot opens: own only new |
| 137 | + IntegrationTests/Features/TimeSeries files. Use existing genuine RF3 fixture, |
| 138 | + persisted credentials, real .NET and official MCP clients, parity/error cases |
| 139 | + and a real isolated follower kill/restart. No shared-fixture, topology, |
| 140 | + timeout, fake transport, retry-policy or authority changes. |
| 141 | +6. TASK-SERIES-JOIN-R8, root: independently inspect every diff and acceptance |
| 142 | + oracle; join all completed work; full development restore/Release build, |
| 143 | + canonical format/static governance, all-source current-main commit/push, |
| 144 | + exact-SHA GitHub unit/recovery/Docker RF3/MCP/native report inspection and |
| 145 | + honest durable source/qualification evidence. Blocked/partial workers cannot |
| 146 | + unblock this join. No local tests, container or benchmark execution. |
| 147 | + |
| 148 | +Existing e8d1a9eb1 CI run37049469093 is the full relevant baseline; all unit, |
| 149 | +recovery/analyzer and RF3 jobs pass, comparison-smoke has two genuine Aspire |
| 150 | +Waiting failures. Preserve them as failures and keep comparisons/measurements |
| 151 | +distinct. Native report counts and hashes are retained in the runtime ledger. |
| 152 | + |
| 153 | +## Verification, migration and rollback |
| 154 | + |
| 155 | +Each criterion maps to test-owned independent raw oracles and real provider/ |
| 156 | +SDK/MCP/fault operations in the TimeSeries feature. Preserve all existing |
| 157 | +forward/inclusive/security/replication tests. No doubles, weakened assertions, |
| 158 | +global suppression, new retries or increased bounds. Native seek-error disposal |
| 159 | +requires source/lifetime review because no real provider failure-injection API |
| 160 | +is available; it is not claimed as an executed environmental fault branch. |
| 161 | + |
| 162 | +No persisted format or data migration. Public changes are additive read APIs; |
| 163 | +roll back the additive contracts/routes/read catalog/reader slice together while |
| 164 | +retaining old numeric read kinds and storage formats. The separate ADR-050 |
| 165 | +comparison continues declaring its own measured operations; client-folded |
| 166 | +statistics must not be relabeled measured server aggregation. |
| 167 | + |
| 168 | +This ADR remains Accepted until all implementation, tests, docs and verification |
| 169 | +exist. Numeric coverage, full comparison, endurance, power-loss, maximum speed, |
| 170 | +activation movement, retention/rollups, SQL and compressed chunks remain open |
| 171 | +gates/work. SMID waits for owner mapping; native Orleans Streams remains an |
| 172 | +independent first-priority architecture workstream under the root policy. |
0 commit comments