diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index e4ff3c2..0000000 --- a/AGENTS.md +++ /dev/null @@ -1,371 +0,0 @@ -# Http11Probe — AI Agent Contribution Guide - -This file is designed for LLM/AI agent consumption. It contains precise, unambiguous instructions for adding a new test or a new framework to the Http11Probe platform. - -## Project overview - -Http11Probe is an HTTP/1.1 compliance and security tester. It sends raw TCP requests to servers and validates responses against RFC 9110/9112. The codebase is C# / .NET 10. Documentation is a Hugo + Hextra static site under `docs/`. - ---- - -## TASK A: Add a new test - -Adding a test requires changes to **5 locations** (sometimes 4 if URL mapping is automatic). - -### Step 1 — Add the test case to the suite file - -Choose the correct suite file based on category: - -| Category | File path | -|----------|-----------| -| Compliance | `src/Http11Probe/TestCases/Suites/ComplianceSuite.cs` | -| Smuggling | `src/Http11Probe/TestCases/Suites/SmugglingSuite.cs` | -| Malformed Input | `src/Http11Probe/TestCases/Suites/MalformedInputSuite.cs` | -| Normalization | `src/Http11Probe/TestCases/Suites/NormalizationSuite.cs` | -| Cookies | `src/Http11Probe/TestCases/Suites/CookieSuite.cs` | - -Append a `yield return new TestCase { ... };` inside the `GetTestCases()` method. Here is the full schema: - -```csharp -yield return new TestCase -{ - // REQUIRED fields - Id = "COMP-EXAMPLE", // Unique ID. Prefix conventions below. - Description = "What this test checks", // One-line human description. - Category = TestCategory.Compliance, // Compliance | Smuggling | MalformedInput | Normalization - PayloadFactory = ctx => MakeRequest( // Builds the raw HTTP bytes to send. - $"GET / HTTP/1.1\r\nHost: {ctx.HostHeader}\r\n\r\n" - ), - Expected = new ExpectedBehavior // How to validate the response. See below. - { - ExpectedStatus = StatusCodeRange.Exact(400), - }, - - // OPTIONAL fields - RfcLevel = RfcLevel.Must, // Must (default) | Should | May | OughtTo | NotApplicable - RfcReference = "RFC 9112 §5.1", // Use § not "Section". Omit if no RFC applies. - Scored = true, // Default true. Set false for MAY/informational tests. - AllowConnectionClose = false, // On Expected. See validation rules below. - BehavioralAnalyzer = (response) => ..., // Optional Func for analysis notes. -}; -``` - -**Test ID prefix conventions:** - -| Prefix | Suite | -|--------|-------| -| `COMP-` | Compliance | -| `SMUG-` | Smuggling | -| `MAL-` | Malformed Input | -| `NORM-` | Normalization | -| `COOK-` | Cookies | -| `RFC9112-X.X-` or `RFC9110-X.X-` | Compliance (maps directly to an RFC section) | - -**Validation patterns — choose ONE:** - -Pattern 1 — Exact status, no alternatives: -```csharp -Expected = new ExpectedBehavior -{ - ExpectedStatus = StatusCodeRange.Exact(400), -} -``` -Use for strict MUST-400 requirements (e.g. SP-BEFORE-COLON, MISSING-HOST, DUPLICATE-HOST, OBS-FOLD, CR-ONLY). - -Pattern 2 — Status with connection close as alternative: -```csharp -Expected = new ExpectedBehavior -{ - ExpectedStatus = StatusCodeRange.Exact(400), - AllowConnectionClose = true, -} -``` -Use when close is acceptable instead of a status code. - -Pattern 3 — Custom validator (takes priority over ExpectedStatus): -```csharp -Expected = new ExpectedBehavior -{ - CustomValidator = (response, state) => - { - if (state == ConnectionState.ClosedByServer && response is null) return TestVerdict.Pass; - if (response is null) return TestVerdict.Fail; - if (response.StatusCode == 400) return TestVerdict.Pass; - if (response.StatusCode >= 200 && response.StatusCode < 300) return TestVerdict.Warn; - return TestVerdict.Fail; - }, - Description = "400 or close = pass, 2xx = warn", -} -``` -Use for pass/warn/fail logic, timeout acceptance, or multi-outcome tests. - -**Available StatusCodeRange factories:** -- `StatusCodeRange.Exact(int code)` — single status code -- `StatusCodeRange.Range(int start, int end)` — inclusive range -- `StatusCodeRange.Range2xx` — 200-299 -- `StatusCodeRange.Range4xx` — 400-499 -- `StatusCodeRange.Range4xxOr5xx` — 400-599 - -**Available TestVerdict values:** `Pass`, `Fail`, `Warn`, `Skip`, `Error` - -**Available ConnectionState values:** `Open`, `ClosedByServer`, `TimedOut`, `Error` - -**Helper method available in all suites:** -```csharp -private static byte[] MakeRequest(string request) => Encoding.ASCII.GetBytes(request); -``` - -**RfcLevel values:** -- `RfcLevel.Must` — (default) RFC says MUST / MUST NOT. Only set explicitly if you want to be clear. -- `RfcLevel.Should` — RFC says SHOULD / SHOULD NOT / RECOMMENDED. -- `RfcLevel.May` — RFC says MAY / OPTIONAL. Both behaviors are compliant. -- `RfcLevel.OughtTo` — RFC uses "ought to" (weaker than SHOULD). -- `RfcLevel.NotApplicable` — No single RFC 2119 keyword applies (best-practice / defensive tests). - -Check the [RFC Requirement Dashboard](docs/content/docs/rfc-requirement-dashboard.md) for classification guidance and to verify your assignment matches the existing pattern. - -**Critical rules:** -- NEVER set `AllowConnectionClose = true` for MUST-400 requirements where the RFC explicitly says "respond with 400". -- Set `RfcLevel` to match the RFC 2119 keyword in the relevant RFC quote. Default is `Must` — only set explicitly for non-Must tests. -- Set `Scored = false` only for MAY-level or purely informational tests. -- Always use `ctx.HostHeader` (not a hardcoded host) in payloads. -- Tests are auto-discovered — no registration step needed. The `GetTestCases()` yield return is sufficient. - -### Step 2 — Add docs URL mapping (conditional) - -**File:** `src/Http11Probe.Cli/Reporting/DocsUrlMap.cs` - -This step is **only needed** for `COMP-*` and `RFC*` prefixed tests. The following prefixes are auto-mapped: -- `SMUG-XYZ` → `smuggling/xyz` (lowercased) -- `MAL-XYZ` → `malformed-input/xyz` (lowercased) -- `NORM-XYZ` → `normalization/xyz` (lowercased) -- `COOK-XYZ` → `cookies/xyz` (lowercased) - -For compliance tests, add an entry to the `ComplianceSlugs` dictionary: -```csharp -["COMP-EXAMPLE"] = "headers/example", -``` - -If the doc filename doesn't match the auto-mapping convention, add to `SpecialSlugs` instead: -```csharp -["MAL-CHUNK-EXT-64K"] = "malformed-input/chunk-extension-long", -``` - -### Step 3 — Create the documentation page - -**File:** `docs/content/docs/{category-slug}/{test-slug}.md` - -Category slug mapping: - -| Category | Slug | -|----------|------| -| Compliance (line endings) | `line-endings` | -| Compliance (request line) | `request-line` | -| Compliance (headers) | `headers` | -| Compliance (host header) | `host-header` | -| Compliance (content-length) | `content-length` | -| Compliance (body) | `body` | -| Compliance (upgrade) | `upgrade` | -| Smuggling | `smuggling` | -| Malformed Input | `malformed-input` | -| Normalization | `normalization` | -| Cookies | `cookies` | - -Use this exact template: - -```markdown ---- -title: "EXAMPLE" -description: "EXAMPLE test documentation" -weight: 1 ---- - -| | | -|---|---| -| **Test ID** | `COMP-EXAMPLE` | -| **Category** | Compliance | -| **RFC** | [RFC 9112 §X.X](https://www.rfc-editor.org/rfc/rfc9112#section-X.X) | -| **Requirement** | MUST | -| **Expected** | `400` or close | - -## What it sends - -A request with [description of the non-conforming element]. - -\```http -GET / HTTP/1.1\r\n -Host: localhost:8080\r\n -[malformed element]\r\n -\r\n -\``` - -## What the RFC says - -> "Exact quote from the RFC with the MUST/SHOULD/MAY keyword." -- RFC 9112 Section X.X - -Explanation of what the quote means for this test. - -## Why it matters - -Security and compatibility implications. Why this matters for real-world deployments. - -## Sources - -- [RFC 9112 §X.X](https://www.rfc-editor.org/rfc/rfc9112#section-X.X) -``` - -**Requirement field values:** `MUST`, `SHOULD`, `MAY`, `"ought to"`, `Implicit MUST (grammar violation)`, `Unscored`, or a descriptive phrase like `MUST reject or replace with SP`. - -### Step 4 — Add a card to the category index - -**File:** `docs/content/docs/{category-slug}/_index.md` - -Find the `{{}}` block and add a new card entry. Place scored tests before unscored tests. - -``` -{{}} -``` - -The `link` value is the filename without `.md`. - -### Step 5 — Add a row to the RFC Requirement Dashboard - -**File:** `docs/content/docs/rfc-requirement-dashboard.md` - -This page classifies every test by its RFC 2119 requirement level. You must: - -1. **Add a row** to the correct table based on the test's requirement level: - - `MUST` / `MUST NOT` → "MUST-Level Requirements" table (use the "Reject with 400" sub-table if the RFC explicitly mandates 400, otherwise the "Reject (400 or Connection Close Acceptable)" sub-table) - - `SHOULD` / `SHOULD NOT` → "SHOULD-Level Requirements" table - - `MAY` → "MAY-Level Requirements" table - - `Scored = false` → "Unscored Tests" table (regardless of RFC keyword) - -2. **Update the counts** in: - - The summary table at the top (increment the matching requirement level) - - The total test count in both the `description` frontmatter and the "Total: N tests" line - - The "Requirement Level by Suite" section (increment the matching suite + level) - - The "RFC Section Cross-Reference" table (increment existing section count or add a new row) - -3. **Include** the test ID, suite name, RFC link, and an exact RFC quote with the keyword bolded (e.g., `**MUST**`). - -### Verification checklist - -After making all changes: - -1. `dotnet build Http11Probe.slnx -c Release` — must compile without errors. -2. The new test ID appears in the output of `dotnet run --project src/Http11Probe.Cli -- --host localhost --port 8080`. -3. Hugo docs render: `cd docs && hugo server` — the new page is accessible and linked from its category index. - ---- - -## TASK B: Add a new framework - -Adding a framework requires creating **3 files** in a new directory, plus a **documentation page** on the website. - -### Step 1 — Create the server directory - -Create a new directory: `src/Servers/YourServer/` - -### Step 2 — Implement the server - -Your server MUST listen on **port 8080** and implement these endpoints: - -| Endpoint | Method | Behavior | -|----------|--------|----------| -| `/` | `GET` | Return `200 OK` | -| `/` | `POST` | Read the full request body and return it in the response body | -| `/echo` | `GET`, `POST` | Return all received request headers in the response body, one per line as `Name: Value` | -| `/cookie` | `GET`, `POST` | Parse the `Cookie` header and return each cookie as `name=value` on its own line | - -HEAD and OPTIONS are handled automatically by virtually all frameworks — do not implement them explicitly. - -The `/echo` endpoint is critical for normalization tests. It must echo back all headers the server received, preserving the names as the server internally represents them. - -Example `/echo` response body: -``` -Host: localhost:8080 -Content-Length: 11 -Content-Type: text/plain -``` - -The `/cookie` endpoint is used by the Cookies test suite. It must split the `Cookie` header on `;`, trim leading whitespace from each pair, find the first `=`, and output `name=value\n` for each cookie. - -Example — given `Cookie: foo=bar; baz=qux`, the response body should be: -``` -foo=bar -baz=qux -``` - -### Step 3 — Add a Dockerfile - -Create `src/Servers/YourServer/Dockerfile` that builds and runs the server. - -Key requirements: -- The container runs with `--network host`, so bind to `0.0.0.0:8080`. -- Use `ENTRYPOINT` (not `CMD`) for the server process. -- The Dockerfile build context is the repository root, so paths like `COPY src/Servers/YourServer/...` are correct. - -Example: -```dockerfile -FROM python:3.12-slim -WORKDIR /app -RUN pip install --no-cache-dir flask -COPY src/Servers/YourServer/app.py . -ENTRYPOINT ["python3", "app.py", "8080"] -``` - -### Step 4 — Add probe.json - -Create `src/Servers/YourServer/probe.json` with exactly one field: - -```json -{"name": "Your Server Display Name"} -``` - -This name appears in the leaderboard and PR comments. - -### Step 5 — Create the server documentation page - -**File:** `docs/content/servers/{server-name-lowercase}.md` - -Use this template: - -```markdown ---- -title: "Server Name" -toc: false -breadcrumbs: false ---- - -**Language:** Language · [View source on GitHub](https://github.com/MDA2AV/Http11Probe/tree/main/src/Servers/YourServer) - -## Dockerfile - -\```dockerfile -[Complete Dockerfile contents] -\``` - -## Source — `filename.ext` - -\```language -[Complete source file contents] -\``` -``` - -Rules: -- Include one `## Source — \`filename\`` section per source file (exclude `probe.json`). -- Use the correct syntax-highlight language for each code block (e.g., `python`, `javascript`, `text`, `html`). -- The Dockerfile section always comes first, followed by source files. - -### Verification checklist - -1. Build the Docker image: `docker build -f src/Servers/YourServer/Dockerfile -t yourserver .` -2. Run: `docker run --network host yourserver` -3. Verify endpoints: - - `curl http://localhost:8080/` returns 200 - - `curl -X POST -d "hello" http://localhost:8080/` returns "hello" - - `curl -X POST -d "test" http://localhost:8080/echo` returns headers - - `curl -H "Cookie: foo=bar; baz=qux" http://localhost:8080/cookie` returns `foo=bar` and `baz=qux` on separate lines -4. Run the probe: `dotnet run --project src/Http11Probe.Cli -- --host localhost --port 8080` - -No changes to CI workflows, configs, or other files are needed. The pipeline auto-discovers servers from `src/Servers/*/probe.json`. diff --git a/docs/content/_index.md b/docs/content/_index.md index 429ea83..a210161 100644 --- a/docs/content/_index.md +++ b/docs/content/_index.md @@ -60,8 +60,8 @@ Every new framework added makes the comparison more useful for the entire commun
{{< cards >}} - {{< card link="add-a-framework" title="Add a Framework" subtitle="Three steps to add your framework — Dockerfile, probe.json, and open a PR." icon="plus-circle" >}} + {{< card link="add-a-framework" title="Add a Framework" subtitle="Four steps to add your framework — server, Dockerfile, probe.json, and open a PR." icon="plus-circle" >}} {{< card link="add-a-test" title="Add a Test" subtitle="How to define a new test case, write its documentation, and wire it into the platform." icon="beaker" >}} - {{< card link="add-with-ai-agent" title="Add with AI Agent" subtitle="Use an AI coding agent to add a test or framework using the machine-readable AGENTS.md guide." icon="chip" >}} + {{< card link="add-with-ai-agent" title="Add with AI Agent" subtitle="Point an AI coding agent at the contribution guides to add a test or framework." icon="chip" >}} {{< /cards >}} diff --git a/docs/content/add-a-framework/_index.md b/docs/content/add-a-framework/_index.md index 41ba1d6..808c361 100644 --- a/docs/content/add-a-framework/_index.md +++ b/docs/content/add-a-framework/_index.md @@ -16,7 +16,10 @@ Your server must listen on **port 8080** and implement three endpoints: | `/` | `HEAD` | Return `200 OK` with no body. Used by smuggling tests that check body handling on HEAD requests. | | `/` | `POST` | Read the full request body and return it in the response. Used by body handling and smuggling tests. | | `/` | `OPTIONS` | Return `200 OK`. Used by smuggling tests that check body handling on OPTIONS requests. | -| `/echo` | `POST` | Return all received request headers in the response body, one per line as `Name: Value`. Used by normalization tests. | +| `/echo` | `GET`, `POST` | Return all received request headers in the response body, one per line as `Name: Value`. Used by normalization tests. | +| `/cookie` | `GET`, `POST` | Parse the `Cookie` header and return each cookie as `name=value` on its own line. Used by the Cookies test suite. | + +HEAD and OPTIONS are handled automatically by virtually all frameworks — a catch-all route on `/` is usually enough, and you should not need to implement them explicitly. What matters is that they return `200` rather than `405`, so the smuggling tests can evaluate body handling instead of getting a method-not-allowed response. ### Why `/echo`? @@ -34,19 +37,83 @@ Content-Type: text/plain The order does not matter. Include all headers the server received (framework-added headers like `Connection` are fine). +### Response format for `/cookie` + +Split the `Cookie` header on `;`, trim leading whitespace from each pair, find the first `=`, and output `name=value` on its own line. Given `Cookie: foo=bar; baz=qux`, the response body is: + +``` +foo=bar +baz=qux +``` + +The Cookies tests use this to see how the server parses cookie pairs it receives, so echo back what the server actually parsed rather than re-parsing the raw header yourself. + ## Steps -**1. Create a server directory** — Add a directory under `src/Servers/YourServer/` with your server source code implementing the three endpoints above. +**1. Create a server directory** — Add a directory under `src/Servers/YourServer/` with your server source code implementing the endpoints above. -**2. Add a Dockerfile** — Build and run your server. It will run with `--network host`. +**2. Add a Dockerfile** — Build and run your server. The build context is the repository root, so `COPY src/Servers/YourServer/...` paths are correct. The container runs with `--network host`, so bind to `0.0.0.0:8080`. Use `ENTRYPOINT` rather than `CMD` for the server process. -**3. Add a `probe.json`** — One file, one field: +**3. Add a `probe.json`** — The display name and the implementation language: ```json -{"name": "Your Server"} +{"name": "Your Server", "language": "Python"} +``` + +The name appears on the leaderboard and in PR comments, and the language is used to group servers on the site. You can also add an optional `"repository"` field linking to the framework's own project. + +**4. Add a server documentation page** — Create `docs/content/servers/{server-name-lowercase}.md` so the framework gets a page on the site: + +````markdown +--- +title: "Your Server" +description: "Your Server (Language) tested against RFC 9110/9112 for HTTP/1.1 compliance, request smuggling resistance, and malformed input handling." +toc: true +breadcrumbs: false +--- + +**Language:** Language · [View source on GitHub](https://github.com/MDA2AV/Http11Probe/tree/main/src/Servers/YourServer) + +## Dockerfile + +```dockerfile +[Complete Dockerfile contents] ``` -Open a PR and the probe runs automatically. +## Source — `filename.ext` + +```language +[Complete source file contents] +``` +```` + +The Dockerfile section comes first, then one **Source** section per source file (excluding `probe.json`), each with the right syntax-highlight language for that file. + +## Verify + +Before opening the PR, build and exercise the server locally: + +```bash +docker build -f src/Servers/YourServer/Dockerfile -t yourserver . +docker run --network host yourserver +``` + +Then check each endpoint: + +```bash +curl http://localhost:8080/ # 200 OK +curl -X POST -d "hello" http://localhost:8080/ # hello +curl -X POST -d "test" http://localhost:8080/echo # one header per line +curl -H "Cookie: foo=bar; baz=qux" http://localhost:8080/cookie # foo=bar / baz=qux +``` + +Finally, run the probe against it: + +```bash +dotnet run --project src/Http11Probe.Cli -- --host localhost --port 8080 +``` + +Then open a PR and the probe runs automatically. ## How It Works @@ -66,7 +133,7 @@ Here's the Flask server as a reference: **`src/Servers/FlaskServer/probe.json`** ```json -{"name": "Flask"} +{"name": "Flask", "language": "Python"} ``` **`src/Servers/FlaskServer/Dockerfile`** @@ -86,7 +153,14 @@ from werkzeug.routing import Rule app = Flask(__name__) -@app.route('/echo', methods=['GET','POST','PUT','DELETE','PATCH','OPTIONS','HEAD']) +@app.route('/cookie', methods=['GET','POST','PUT','DELETE','PATCH']) +def cookie_endpoint(): + lines = [] + for name, value in request.cookies.items(): + lines.append(f"{name}={value}") + return '\n'.join(lines) + '\n', 200, {'Content-Type': 'text/plain'} + +@app.route('/echo', methods=['GET','POST','PUT','DELETE','PATCH']) def echo(): lines = [] for name, value in request.headers: @@ -109,6 +183,7 @@ if __name__ == "__main__": The key parts: - **`/echo`** — echoes all received headers back as plain text. +- **`/cookie`** — echoes the cookies Flask parsed, one `name=value` per line. - **`POST /`** — reads and returns the request body (needed for body and smuggling tests). - **`GET /`** (catch-all) — returns `"OK"` with `200`. - **`HEAD /`** and **`OPTIONS /`** — handled by the catch-all; return `200` so smuggling tests can evaluate body handling instead of getting `405`. diff --git a/docs/content/add-a-test.md b/docs/content/add-a-test.md index 71bba0d..86d2b37 100644 --- a/docs/content/add-a-test.md +++ b/docs/content/add-a-test.md @@ -3,7 +3,7 @@ title: Add a Test description: "How to add a new HTTP/1.1 compliance, smuggling, or malformed-input test case to Http11Probe, including the test case definition, documentation page, and category index entry." --- -A step-by-step guide to adding a new test to Http11Probe. Every test touches four places: the suite file, the docs URL map (sometimes), a documentation page, and the category index. +A step-by-step guide to adding a new test to Http11Probe. Every test touches five places: the suite file, the docs URL map (sometimes), a documentation page, the category index, and the RFC Requirement Dashboard. ## 1. Define the test case @@ -11,11 +11,15 @@ Pick the suite that matches your test's category and add a `yield return new Tes | Category | File | |----------|------| -| Compliance | `src/TestCases/Suites/ComplianceSuite.cs` | -| Smuggling | `src/TestCases/Suites/SmugglingSuite.cs` | -| Malformed Input | `src/TestCases/Suites/MalformedInputSuite.cs` | -| Normalization | `src/TestCases/Suites/NormalizationSuite.cs` | -| Cookies | `src/TestCases/Suites/CookieSuite.cs` | +| Compliance | `src/Http11Probe/TestCases/Suites/ComplianceSuite.cs` | +| Smuggling | `src/Http11Probe/TestCases/Suites/SmugglingSuite.cs` | +| Malformed Input | `src/Http11Probe/TestCases/Suites/MalformedInputSuite.cs` | +| Normalization | `src/Http11Probe/TestCases/Suites/NormalizationSuite.cs` | +| Cookies | `src/Http11Probe/TestCases/Suites/CookieSuite.cs` | +| WebSockets | `src/Http11Probe/TestCases/Suites/WebSocketsSuite.cs` | +| Capabilities | `src/Http11Probe/TestCases/Suites/CapabilitiesSuite.cs` | + +`CapabilitiesSuite` holds multi-step sequence tests and yields `SequenceTestCase` instead of `TestCase`; follow the existing entries in that file if your test needs several requests on one connection. ```csharp yield return new TestCase @@ -49,6 +53,8 @@ yield return new TestCase | `MAL-` | Malformed Input | | `NORM-` | Normalization | | `COOK-` | Cookies | +| `WS-` | WebSockets | +| `CAP-` | Capabilities | | `RFC9112-...` or `RFC9110-...` | Compliance (when the test maps directly to a specific RFC section) | ### Validation options @@ -90,7 +96,9 @@ Expected = new ExpectedBehavior ### Key conventions -- Set `RfcLevel` to match the RFC 2119 keyword for the requirement being tested. The default is `Must` — only set it explicitly for non-Must tests. Available values: `Must`, `Should`, `May`, `OughtTo`, `NotApplicable`. Check the [RFC Requirement Dashboard]({{< relref "docs/rfc-requirement-dashboard" >}}) for classification guidance. +- Set `RfcLevel` to match the RFC 2119 keyword for the requirement being tested. The default is `Must` — only set it explicitly for non-Must tests. Available values: `Must`, `Should`, `May`, `OughtTo`, `NotApplicable`. Check the [RFC Requirement Dashboard](/docs/rfc-requirement-dashboard.html) for classification guidance. +- Build payloads with the `MakeRequest` helper each suite defines, and always use `ctx.HostHeader` rather than a hardcoded host. +- Tests are auto-discovered — the `yield return` is the whole registration, there is no list to update. - Use `Exact(400)` with **no** `AllowConnectionClose` for strict MUST-400 requirements (SP-BEFORE-COLON, MISSING-HOST, DUPLICATE-HOST, OBS-FOLD, CR-ONLY). - Set `AllowConnectionClose = true` only when connection close is an acceptable alternative to a status code. - Set `Scored = false` for MAY-level or informational tests. @@ -101,9 +109,9 @@ Expected = new ExpectedBehavior **File:** `src/Http11Probe.Cli/Reporting/DocsUrlMap.cs` -Tests prefixed with `SMUG-`, `MAL-`, `NORM-`, or `COOK-` are auto-mapped to their doc URL based on the ID. For example, `SMUG-CL-TE-BOTH` maps to `smuggling/cl-te-both`. +Tests prefixed with `SMUG-`, `MAL-`, `NORM-`, `COOK-`, or `WS-` are auto-mapped to their doc URL based on the ID. For example, `SMUG-CL-TE-BOTH` maps to `smuggling/cl-te-both`. -For `COMP-*` or `RFC*` prefixed tests, add an entry to the `ComplianceSlugs` dictionary: +Every other prefix — `COMP-`, `RFC*`, `CAP-` — needs an explicit entry in the `ComplianceSlugs` dictionary, otherwise the test result won't link to its documentation: ```csharp ["COMP-MY-TEST"] = "headers/my-test", @@ -115,6 +123,25 @@ If the slug doesn't follow the standard pattern (e.g. the filename differs from **File:** `docs/content/docs/{category}/{test-slug}.md` +The `{category}` folder is the slug the doc URL map points at. Compliance tests are split across several folders by topic: + +| Category | Folder | +|----------|--------| +| Compliance (line endings) | `line-endings` | +| Compliance (request line) | `request-line` | +| Compliance (headers) | `headers` | +| Compliance (host header) | `host-header` | +| Compliance (content-length) | `content-length` | +| Compliance (body) | `body` | +| Smuggling | `smuggling` | +| Malformed Input | `malformed-input` | +| Normalization | `normalization` | +| Cookies | `cookies` | +| WebSockets | `websockets` | +| Capabilities | `caching` | + +Compliance folders are chosen by topic, so pick the one the requirement belongs to rather than deriving it from the test ID. + Use this template: ```markdown @@ -159,7 +186,31 @@ Add a card entry in the appropriate section (scored or unscored): {{}} ``` -## 5. Verify +The `link` value is the filename without `.md`. Place scored tests before unscored ones. + +## 5. Add a row to the RFC Requirement Dashboard + +**File:** `docs/content/docs/rfc-requirement-dashboard.md` + +This page classifies every test by its RFC 2119 requirement level, so a new test needs a row and the surrounding counts need updating. + +Pick the table by requirement level: + +| Level | Table | +|-------|-------| +| `MUST` / `MUST NOT` | "MUST-Level Requirements" — use the "Reject with 400" sub-table when the RFC explicitly mandates 400, otherwise "Reject (400 or Connection Close Acceptable)" | +| `SHOULD` / `SHOULD NOT` | "SHOULD-Level Requirements" | +| `MAY` | "MAY-Level Requirements" | +| Any level, `Scored = false` | "Unscored Tests" | + +The row carries the test ID, suite name, RFC link, and an exact RFC quote with the keyword bolded (`**MUST**`). Then update every count the new row affects: + +- The summary table at the top +- The total test count, in both the `description` frontmatter and the "Total: N tests" line +- The "Requirement Level by Suite" section +- The "RFC Section Cross-Reference" table — increment the existing section or add a row + +## 6. Verify Build and run the probe locally: @@ -171,6 +222,12 @@ dotnet run --project src/Http11Probe.Cli -- --host localhost --port 8080 Check that: - Your test appears in the JSON output with the correct ID - The verdict makes sense against a known server -- The documentation page renders correctly with `hugo server` in `docs/` +- The documentation page renders and is linked from its category index + +To preview the site, build it from `web/`: + +```bash +cd web && npm ci && node build.mjs +``` No changes are needed in the CI workflow -- new tests are discovered automatically. diff --git a/docs/content/add-with-ai-agent.md b/docs/content/add-with-ai-agent.md index 44be734..b86a59c 100644 --- a/docs/content/add-with-ai-agent.md +++ b/docs/content/add-with-ai-agent.md @@ -1,26 +1,26 @@ --- title: Add with AI Agent -description: "Use an AI coding agent with Http11Probe's machine-readable AGENTS.md guide to add a new compliance test or HTTP server framework." +description: "Use an AI coding agent to add a new compliance test or HTTP server framework to Http11Probe, following the Add a Test and Add a Framework guides." --- -Use an AI coding agent (Claude Code, Cursor, Copilot, etc.) to add a new test or framework to Http11Probe. The repository includes a machine-readable contribution guide at [`AGENTS.md`](https://github.com/MDA2AV/blob/main/AGENTS.md) designed specifically for LLM consumption. +Use an AI coding agent (Claude Code, Cursor, Copilot, etc.) to add a new test or framework to Http11Probe. Both contribution guides are written to be followed step by step, so an agent can work straight from them: -## How to use it +- [Add a Test](/add-a-test.html) — suite file, docs URL map, documentation page, category index card, dashboard row +- [Add a Framework](/add-a-framework.html) — server implementation, Dockerfile, `probe.json`, server page -Point your AI agent at the repository and reference the `AGENTS.md` file. It contains precise, unambiguous instructions for both tasks: +## How to use it -- **Task A** — Add a new test (4 steps: suite file, docs URL map, documentation page, category index card) -- **Task B** — Add a new framework (3 files: server implementation, Dockerfile, probe.json) +Point your agent at the repository and at the guide for the task. Give it the specific behaviour you want covered, or the framework and runtime version you want added, and let it work through the steps. ## Example prompts ### Adding a test -> Read AGENTS.md, then add a new compliance test that checks whether the server rejects requests with a space before the colon in a header field name. The RFC reference is RFC 9112 §5.1. +> Follow the Add a Test guide at https://www.http-probe.com/add-a-test.html, then add a new compliance test that checks whether the server rejects requests with a space before the colon in a header field name. The RFC reference is RFC 9112 §5.1. ### Adding a framework -> Read AGENTS.md, then add a new Express.js server to the platform. Use Node 22 and make sure all five endpoints are implemented. +> Follow the Add a Framework guide at https://www.http-probe.com/add-a-framework.html, then add a new Express.js server to the platform. Use Node 22 and make sure all three endpoints are implemented. ## What the agent will do @@ -35,13 +35,14 @@ For a new **test**, the agent will: For a new **framework**, the agent will: 1. Create a server directory under `src/Servers/` -2. Implement the server with all required endpoints (GET, HEAD, POST, OPTIONS on `/`, GET/POST on `/echo`, and GET/POST on `/cookie`) +2. Implement the three required endpoints (`/`, `/echo`, `/cookie`) 3. Write a Dockerfile that builds and runs the server on port 8080 -4. Add a `probe.json` with the display name +4. Add a `probe.json` with the display name and language +5. Add a server documentation page under `docs/content/servers/` ## Tips -- The `AGENTS.md` file includes verification checklists — make sure the agent runs them before submitting +- Both guides have a verification step — make sure the agent runs it before submitting - No changes to CI workflows are needed for either task; tests and servers are auto-discovered - For tests, the agent should check the RFC to determine the correct `RfcLevel` (MUST/SHOULD/MAY/"ought to"/N/A) and set it on the `TestCase`. The default is `Must` — only set explicitly for non-Must tests -- The agent should add a row to the [RFC Requirement Dashboard](docs/content/docs/rfc-requirement-dashboard.md) and update all counts +- The agent should add a row to the [RFC Requirement Dashboard](/docs/rfc-requirement-dashboard.html) and update all counts