Skip to content

Round-trip every remaining e2e JSON response through its schema - #1332

Merged
jviotti merged 1 commit into
mainfrom
tests-round-trip
Nov 19, 2026
Merged

jviotti merged 1 commit into
mainfrom
tests-round-trip

Conversation

@jviotti

@jviotti jviotti commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Signed-off-by: Juan Cruz Viotti jv@jviotti.com

View guided diff

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 7 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="enterprise/e2e/html/hurl/schemas-fragment.all.hurl">

<violation number="1" location="enterprise/e2e/html/hurl/schemas-fragment.all.hurl:10">
P2: The fragment-bearing `POST /schemas/evaluate` response is still not validated against its advertised schema, so this test does not cover every JSON response as intended. Add a capture block after that response and evaluate it through the captured error schema before continuing.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Link, ETag
Link: </self/v1/schemas/api/error>; rel="describedby"
[Captures]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The fragment-bearing POST /schemas/evaluate response is still not validated against its advertised schema, so this test does not cover every JSON response as intended. Add a capture block after that response and evaluate it through the captured error schema before continuing.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At enterprise/e2e/html/hurl/schemas-fragment.all.hurl, line 10:

<comment>The fragment-bearing `POST /schemas/evaluate` response is still not validated against its advertised schema, so this test does not cover every JSON response as intended. Add a capture block after that response and evaluate it through the captured error schema before continuing.</comment>

<file context>
@@ -7,6 +7,9 @@ Content-Type: application/problem+json
 Access-Control-Allow-Origin: *
 Access-Control-Expose-Headers: Link, ETag
 Link: </self/v1/schemas/api/error>; rel="describedby"
+[Captures]
+last_response: body
+schema_path: header "Link" regex "<([^>]+)>"
</file context>

@augmentcode

augmentcode Bot commented Nov 19, 2026

Copy link
Copy Markdown
🤖 Augment PR Summary

Summary: Extends schema round-trip coverage in enterprise end-to-end tests.
Changes:

  • Captures additional JSON response bodies and schema paths from Link headers.
  • Validates authenticated vault directory listings.
  • Validates login endpoint method-not-allowed errors.
  • Validates canonical MCP dependency and metadata response envelopes.
  • Covers MCP protocol revisions 2025-11-25 and 2026-07-28.
  • Validates fragment-rejection errors across schema REST endpoints.
  • Submits captured bodies to the evaluation API and requires valid: true.
Technical notes: Adds evaluator response-header assertions without changing production code.

🤖 Was this summary useful? React with 👍 or 👎

@augmentcode augmentcode Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. No suggestions at this time.

Comment augment review to trigger a new review at any time.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark (community)

Details
Benchmark suite Current: 3c85722 Previous: f5b6e93 Ratio
Add one schema (0 existing) 264 ms 284 ms 0.93
Add one schema (100 existing) 43 ms 54 ms 0.80
Add one schema (1000 existing) 88 ms 98 ms 0.90
Add one schema (10000 existing) 1139 ms 755 ms 1.51
Update one schema (1 existing) 35 ms 39 ms 0.90
Update one schema (101 existing) 43 ms 51 ms 0.84
Update one schema (1001 existing) 89 ms 102 ms 0.87
Update one schema (10001 existing) 864 ms 899 ms 0.96
Cached rebuild (1 existing) 9 ms 18 ms 0.50
Cached rebuild (101 existing) 15 ms 18 ms 0.83
Cached rebuild (1001 existing) 29 ms 51 ms 0.57
Cached rebuild (10001 existing) 243 ms 359 ms 0.68
Index 100 schemas 506 ms 570 ms 0.89
Index 1000 schemas 1079 ms 1489 ms 0.72
Index 10000 schemas 11543 ms 13359 ms 0.86
Index 10000 schemas (custom meta-schema) 15096 ms 16653 ms 0.91
Index 10000 schemas ($ref fan-out) 13822 ms 15709 ms 0.88
test/e2e/html: Schema Fetch (p50) 304 us 379 us 0.80
test/e2e/html: Schema Fetch (p99) 391 us 538 us 0.73

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark (enterprise)

Details
Benchmark suite Current: 3c85722 Previous: f5b6e93 Ratio
Add one schema (0 existing) 574 ms 378 ms 1.52
Add one schema (100 existing) 121 ms 92 ms 1.32
Add one schema (1000 existing) 170 ms 145 ms 1.17
Add one schema (10000 existing) 772 ms 1121 ms 0.69
Update one schema (1 existing) 111 ms 86 ms 1.29
Update one schema (101 existing) 127 ms 90 ms 1.41
Update one schema (1001 existing) 177 ms 157 ms 1.13
Update one schema (10001 existing) 803 ms 1006 ms 0.80
Cached rebuild (1 existing) 15 ms 12 ms 1.25
Cached rebuild (101 existing) 20 ms 16 ms 1.25
Cached rebuild (1001 existing) 45 ms 39 ms 1.15
Cached rebuild (10001 existing) 304 ms 316 ms 0.96
Index 100 schemas 802 ms 623 ms 1.29
Index 1000 schemas 2016 ms 1371 ms 1.47
Index 10000 schemas 16706 ms 12169 ms 1.37
Index 10000 schemas (custom meta-schema) 16733 ms 12490 ms 1.34
Index 10000 schemas ($ref fan-out) 22195 ms 15729 ms 1.41
enterprise/e2e/auth: Schema Anonymous (p50) 385 us 212 us 1.82
enterprise/e2e/auth: Schema Anonymous (p99) 506 us 334 us 1.51
enterprise/e2e/auth: Schema API Key Identity (p50) 390 us 216 us 1.81
enterprise/e2e/auth: Schema API Key Identity (p99) 497 us 316 us 1.57
enterprise/e2e/auth: Schema API Key SHA256 (p50) 398 us 215 us 1.85
enterprise/e2e/auth: Schema API Key SHA256 (p99) 517 us 295 us 1.75
enterprise/e2e/auth: Schema JWT (p50) 518 us 313 us 1.65
enterprise/e2e/auth: Schema JWT (p99) 666 us 460 us 1.45
test/e2e/html: Schema Fetch (p50) 391 us 219 us 1.79
test/e2e/html: Schema Fetch (p99) 478 us 389 us 1.23

This comment was automatically generated by workflow using github-action-benchmark.

@jviotti
jviotti merged commit 2685e21 into main Nov 19, 2026
6 checks passed
@jviotti
jviotti deleted the tests-round-trip branch November 19, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant