Skip to content

fix(ci): validate benchmarks and patch dependency advisories - #261

Merged
ovasylenko merged 2 commits into
mainfrom
fix/benchmark-workloads-oct23
Oct 23, 2026
Merged

ovasylenko merged 2 commits into
mainfrom
fix/benchmark-workloads-oct23

Conversation

@ovasylenko

@ovasylenko ovasylenko commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

The evaluator benchmark named evaluate_flat_10_steps configured one step with ten unrelated execution nodes and discarded its result. Failed workflows could therefore produce apparently valid timings. CI also omitted the standalone fuzz lockfile from dependency audits, allowing that graph to drift and retain vulnerable TLS dependencies.

This change makes benchmark workloads verify their intended work and audits every tracked Rust lockfile:

  • Configure ten matching steps and require successful evaluator completion for flat and parallel workloads. Scheduler cases assert their expected instance/step counts.
  • Give flatten fixtures unique IDs and accurate 31/23-node labels, with size/uniqueness assertions outside timing. Verify saved storage output is retrieved.
  • Run both explicit Criterion targets in functional smoke mode after workspace tests; the scheduled workflow continues collecting timings.
  • Upgrade ctutils 0.4.3 to 0.4.4 in all four tracked lockfiles for RUSTSEC-2026-0385.
  • Patch the fuzz graph's rustls-webpki to 0.103.16 and webpki-root-certs to 1.0.10 for RUSTSEC-2026-0382/0383. Synchronize its existing jsonschema 0.59 and serde_ignored manifest requirements.
  • Add the fuzz advisory gate and require --locked in every dependency audit. No advisory exception is added.

Validation on current head 89747e94529b5178563c1624e30ca11d6f76196d:

  • Full Linux workspace suite: 9,910 passed, zero failed, ten existing ignored across 238 reported groups.
  • New CI Criterion smoke step: all 40 cases passed (33 engine + seven storage), including evaluator completion and scheduler counts.
  • Full Benchmarks: passed.
  • Mobile: all 12 jobs passed, including iOS simulator tests and Android packaging.
  • Strict workspace/all-target Clippy, native binding checks, generated client contract, docs and all five locked audit configurations passed in Check&Lint.
  • Both nightly fuzz smoke targets passed. Local 806 types/push library regressions also passed against patched ctutils; these overlap the full suite, not an additional independent test count.
  • Formatting and whitespace checks passed. The original incorrect ten-step fixture was reproduced before correction (actual configured size 1).

All ten CI jobs, all twelve Mobile jobs and the full Benchmarks workflow passed before merge. PR #261 merged at 09:41:46 UTC as 77c60aeb9d4aeabd0ace5a66348831c68bb65b40; its tree ae988ae56d6d1896add23421b863f02781b42506 exactly matches the reviewed head. New-main CI and Mobile both completed successfully: all 22 individual jobs passed on exact merge commit 77c60ae. The final main log confirms 9,910 workspace passes, zero failures, ten existing ignored cases and all 40 Criterion smoke cases passing. Additional all-feature/all-target compilation passed without warnings on the identical source tree.

The 31/23-node labels replace misleading 100/500 labels, so those timing series start fresh. No throughput improvement is claimed.

@ovasylenko ovasylenko changed the title fix(bench): validate workloads and run functional smoke in CI fix(ci): validate benchmarks and patch dependency advisories Oct 23, 2026
@ovasylenko
ovasylenko merged commit 77c60ae into main Oct 23, 2026
23 checks passed
@ovasylenko
ovasylenko deleted the fix/benchmark-workloads-oct23 branch October 23, 2026 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant