Add programmable CUDA boosting and verified objective comparisons

.github/workflows/gpu-tests.yml

@@ -1,4 +1,4 @@
-name: v1 GPU Verification Pending
+name: v1 GPU Validation Requires an Approved Packet
 
 on:
   workflow_dispatch:
@@ -12,5 +12,7 @@ jobs:
     steps:
       - name: Explain current v1 boundary
         run: |
-          echo "No v1 CUDA implementation exists. Restore real-device verification at F3; reference tests are not GPU validation."
+          echo "Experimental CUDA components and recipes have committed real-device evidence."
+          echo "This hosted CPU job performs no GPU validation. Use a frozen packet and its separate hardware/upload allowance."
+          echo "See v1-sprints/README.md for the current evidence and execution boundary."
           exit 1

.github/workflows/unit-tests.yml

@@ -10,17 +10,21 @@ jobs:
   references:
     runs-on: ${{ matrix.os }}
     strategy:
+      fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest]
         python-version: ["3.10", "3.12"]
     steps:
       - uses: actions/checkout@v4
+        with:
+          # Historical execution SHAs are part of the committed evidence contract.
+          fetch-depth: 0
       - uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
       - uses: astral-sh/setup-uv@v4
       - run: uv sync --locked --extra test
       - run: uv run --no-sync ruff check src/openboost tests/v1 tests/conftest.py
-      - name: Run current v1 references, not retired production tests
-        run: uv run --no-sync pytest tests/ -n 0 -q
+      - name: Run current CPU correctness and offline evidence checks
+        run: uv run --no-sync pytest tests/ -m "not gpu and not benchmark" -n 0 -q
       - run: uv build

README.md

@@ -23,17 +23,37 @@ these operations and persist validated numeric/categorical trees. The first comp
 and [Normal recipe](docs/v1/normal.md) support weights, offsets and
 fixed/backtracking steps on CPU. Normal exposes ordinary/Fisher directions and
 joint mean/log-scale updates. [Formula and sequential runs](docs/v1/formula-runs.md)
-add structured full-metric updates and independent heterogeneous jobs. CUDA
-execution is not implemented yet. [Binary classification](docs/v1/binary.md) now
+add structured full-metric updates and independent heterogeneous jobs. Experimental
+resident scalar CUDA training passes all 212 bounded T4 checks, including
+weighted/missing parity, transactions and saved CPU inference. The shared scoring
+correction resolves the previous 14 failures without changing tolerances. See the
+[recorded result and scope](benchmarks/v1/evidence/cuda-score-symmetry-089/README.md).
+The first [Normal K=2 and installed D2 T4 run](benchmarks/v1/evidence/cuda-normal-090/README.md)
+passes 381/383 checks, including all earlier scalar cases and nineteen saved-model
+CPU replays. Two ordered acceptance decisions fail the frozen reference; full
+Normal conformance remains open. The [follow-up diagnostic run](benchmarks/v1/evidence/cuda-acceptance-091/README.md)
+preserves both failures and identifies rounding-induced false improvement at
+near-stationary loss. The subsequent objective-owned comparison correction and
+[bounded revalidation](benchmarks/v1/evidence/cuda-recipe-103/README.md) establish
+514 earlier passes plus fifteen new recipe passes with identical production.
+All 529 revised requirements have passing evidence across two executions;
+the original failed verdicts remain preserved. This is not full Normal conformance.
+[Binary classification](docs/v1/binary.md) now
 persists typed class order and exposes probability/label inference.
 [Multiclass and vector leaves](docs/v1/multiclass.md) add joint softmax updates
 and separate split/leaf statistics with arbitrary output mappings.
 
 Independent references and comparator/data checks remain evaluation preparation.
 F0.3 is still open; the user approved overlapping B03–B06 construction without
-removing any v1 scope or acceptance requirements. No real quality, GPU performance
+removing any v1 scope or acceptance requirements. No real quality, competitive GPU performance
 or agent/adoption advantage has been established for the new foundation.
 
+The [early same-host performance checkpoint](benchmarks/v1/evidence/early-performance-104/README.md)
+measures squared-error boosting at 10,000 rows in 7.04 s on CPU and 2.94 s on a
+warm T4, with comparable quality. The four other timing pairs and the separate
+profile are incomplete after deadlines. This synthetic internal result establishes
+neither external-library speed parity nor practical performance across all recipes.
+
 - [Execution and reflections](v1-sprints/README.md)
 - [Construction design](planning/foundation-construction-design.md)
 - [v1 plan](planning/agent-boosting-foundation-plan.md)
@@ -48,14 +68,16 @@ and formula models, and train-many each need their own implementation and eviden
 
 ```bash
 uv sync --extra test
-uv run pytest tests/ -n 0 -q
+uv run pytest tests/ -m "not gpu and not benchmark" -n 0 -q
 uv run ruff check src/openboost tests/v1 tests/conftest.py
 uv build
 ```
 
-Python 3.10+. Current tests are CPU-only reference checks. CUDA is a future
-required execution subset, not an implemented capability of this reset checkout.
-GPU and publishing workflows stay unavailable until their v1 gates are met.
+Python 3.10+. Current tests cover CPU implementation, independent references and
+evaluation infrastructure. Experimental CUDA storage, named fields and histograms
+and bounded resident squared training have real T4 evidence. The full required
+device recipe, quality and cost gates remain open. Run GPU-marked tests only on
+real hardware; publishing remains separate.
 
 ## Historical implementation and evidence
 
@@ -91,9 +113,9 @@ and persists two-model inference with explicit output units. Real A9 evaluation
 [Log-normal AFT](docs/v1/aft.md) adds event/right-censored CPU training and
 persisted scale-aware survival outputs. Real A10 evaluation remains open.
 
-[Current CPU coverage audit](v1-sprints/035-cpu-coverage-audit.md) identifies
-external author workflows and real-data integration as remaining CPU
-prerequisites; Sprint 036 supplies the audited A6 recipe/scaling gap.
+[Current execution and reflections](v1-sprints/README.md)
+separates implemented CPU coverage from remaining authoring, practical execution,
+real selection and GPU evidence.
 
 [Multi-output squared regression](docs/v1/multioutput.md) supports independent/shared trees,
 projected splits and persisted training-only target scaling. Real A6 evaluation remains open.
@@ -102,3 +124,21 @@ projected splits and persisted training-only target scaling. Real A6 evaluation
 across independent jobs, verified at M=1/8/32.
 [Independent stopping](docs/v1/stopping.md) adds validation patience to every CPU
 recipe while keeping model acceptance and best-model selection independent.
+
+
+[Experimental CUDA operations](docs/v1/execution.md) provide context-owned buffers,
+named fields, once-only weighting, routed histograms, candidate scores, composable
+feasibility masks, routing and scalar leaves, with
+[88 passing real T4 checks](benchmarks/v1/evidence/cuda-splits-078/README.md).
+Independent cohort constraints change split selection through the public device
+operations. Separate experimental resident squared geometry, scalar trees and
+accepted/proposal training now pass the separate
+[212-case T4 matrix](benchmarks/v1/evidence/cuda-score-symmetry-089/README.md).
+Shared mapped transactions, Normal geometry and joint/ordered recipes now have
+[bounded passing comparison and recipe evidence](benchmarks/v1/evidence/cuda-recipe-103/README.md),
+with the historical numerical failures preserved in the earlier archives.
+The installed D2 learner uses the same public field/feasibility/tree operations.
+Binary/Poisson objectives, numerical loss-change comparisons and the shared scalar
+recipe now pass [153 GLM T4 checks plus 418 regressions](benchmarks/v1/evidence/cuda-glm-108/README.md),
+with retained input bytes, class-aware inference and 32 audited final/best models.
+Other required CUDA recipes and full phase acceptance remain open.

benchmarks/v1/README.md

@@ -608,6 +608,26 @@ release. All trials and failures are retained. It scores no test labels and is
 not the frozen real-data quality grid, a speed benchmark or fused train-many.
 A6 final comparative quality reporting and the remaining real searches remain open.
 
+The same smoke accepts `--protected` on Linux with a root evaluator. It places
+protocol, validation/scale records, test features and the selection receipt under
+an evaluator-owned mode-0700 directory. Workers receive read-only train/validation
+packets and job files, run as UID/GID 65534 with an 8-GiB address limit and a
+1800-second deadline, and retain the current worker's one-thread contract. Completed
+trial directories are reclaimed by root before the next trial starts. Installed
+Python/package/source paths and output ancestors must be traversable by that UID.
+Unsupported hosts fail before creating output; there is no privilege fallback.
+
+The [Linux selection integration](evidence/protected-selection-070/README.md) passes
+all three focused tests and a retained 16-trial run. The saved Linux receipt now has a macOS replay regression. Score-only
+recomputation permits at most eight times the smaller binary64 spacing of each
+finite value. All other receipt fields and the selected winner remain exact;
+changes to ordering across a near tie still reject release. The original receipt
+byte pin and every artifact hash remain exact. This bounded rule does not promise
+portability across arbitrary numerical libraries or metric changes.
+It is a synthetic four-round grid, not the full 300/1000-round search. The earlier
+standalone permission probe does not validate this call path. Network/new-session
+restrictions and separate author containers remain outside this mode's guarantee.
+
 ### A6 paired quality reporting
 
 A6 quality cells require all `rmse_k` primary metrics followed by
@@ -648,3 +668,89 @@ remain pending. The smoke accepts explicit A2/A3 selections, with seven classes
 for Covertype. Defaults remain A1/A6/A11 to avoid silently broadening existing
 runs. These four-round probes establish no classification quality/calibration,
 search, CUDA or speed claim.
+
+## Evaluator-owned execution freeze (Sprint 070)
+
+For a pinned execution, provide a manifest independently frozen by the evaluator:
+
+```bash
+uv run --no-sync python -m benchmarks.v1.judge /path/to/producer-run \
+  --frozen-manifest /path/to/evaluator/frozen.json --frozen-sha256 EVALUATOR_PINNED_FILE_SHA256
+```
+
+Both options are required together. The file must be outside the producer run
+directory and match the evaluator-supplied raw-byte hash. Strict JSON parsing and
+manifest validation apply to both inputs. The producer manifest must match the
+entire evaluator manifest, including the expected cells, protocol and provenance.
+Rehashing a producer-shrunken matrix cannot satisfy the independent freeze.
+The Python API accepts `frozen_manifest=` from a trusted caller; its reported
+`frozen_manifest_sha256` hashes canonical JSON and can differ from the CLI raw-file pin.
+
+Without this argument, integrity remains relative to the producer's declared
+matrix and `frozen_manifest_match` is null. A true match does not validate the
+experimental design, authenticate provenance, prove all R/C/A/E obligations or
+establish quality. `gate_results` stays empty. The evaluator must control the pin,
+invocation and reference file. An outside-directory check is not an OS permission
+boundary: process/container isolation remains separate work in Sprint 070.
+
+### Current OpenBoost trial retention
+
+`openboost_worker` explicitly runs its A1–A12 recipe and independent A5 quantile
+paths with `retention="summary"`, reporting `diagnostic_retention` in training
+metadata. This is a worker policy, not a new search hyperparameter or a change to
+the public recipe default. Frozen model configurations, selected-model semantics
+and prediction artifacts are unchanged. It reduces stored round arrays without
+qualifying the full search's resource or quality gate. Other worker families must
+be audited separately before large jobs.
+
+### Explicit Linux worker identity and address limits
+
+`process_runner.execute(..., address_limit_bytes=8 * 1024**3, unprivileged=True)`
+requires a Linux root evaluator. It launches with hard/soft RLIMIT_AS limits,
+UID/GID 65534, no supplementary groups, no_new_privs and a minimal explicit
+environment. The fresh output directory belongs to that worker; its ancestors
+must permit traversal. Evaluator-private inputs need separate root ownership and
+permissions. Unsupported setup is rejected; there is no advisory fallback.
+
+The runner records actual launch commands, configured limits, identity/environment
+policy, logs and errors. It kills remaining same-process-group descendants before
+artifact inspection in this mode. This is not a general hostile-code sandbox:
+new sessions and network are not restricted. Use separate containers for independent
+attempts and never share same-UID outputs across them. RLIMIT_AS limits virtual
+address space, not measured resident memory; enclosing-container policy is separate.
+Default execution keeps the existing inherited-identity/environment behavior.
+
+`python -m benchmarks.v1.access_preflight /tmp/fresh-output` runs a bounded Modal
+CPU probe with synthetic protected fixtures; it uploads no real datasets or sealed
+tasks. Actual permission/resource errors remain distinct from the earlier injected
+judge statuses. Passing this probe does not qualify full-search or author-eval gates.
+
+### Full A6 OpenBoost resource planning
+
+`python -m benchmarks.v1.a6_preflight_plan OUTPUT.json` compiles the frozen
+300/1000-round configurations into 160 OpenBoost jobs (shared/independent topology,
+five folds, sixteen configurations each). It writes once and launches nothing.
+The plan records source-freeze hashes, fit-only upper bounds, first resource probes
+and remaining requirements. It does not represent the full comparator matrix or
+a completed full-search resource check. See `v1-sprints/070-a6-resource-plan.json`.
+
+### Explicit comparator bin budgets
+
+The numeric baseline worker accepts optional `config.bins`, an integer in [2, 256].
+XGBoost and LightGBM receive `max_bin=bins`; CatBoost receives
+`border_count=bins-1`, since that parameter counts split borders rather than
+intervals. This applies to the worker's finite encoded inputs; it does not align
+native quantization algorithms. NGBoost explicitly rejects this setting. Omitted
+bins retain native defaults. `bin_budget_smoke.py` checks installed effective
+parameters, stopping records and fresh-process A6 replay at 7 and 255 bins.
+
+`a6_resource_preflight --comparators` runs only the three frozen fold-zero
+configuration-00 comparator resource probes. It verifies the updated A6 plan's
+input pins, uses the protected worker policy and fresh A6 replay, and stops on
+failure without retries. Profile, paired and comparator modes are mutually
+exclusive. This mode does not execute or certify the 400-job search.
+
+Add `--comparator-config 5` to that mode for the frozen deeper-tree/1000-round
+configuration-05 probes. Only indices 0 and 5 are accepted; early stopping remains
+active, so a 1000-round budget need not produce 1000 completed rounds. These
+comparator probes do not qualify deeper OpenBoost execution or the full search.

benchmarks/v1/a6_preflight_plan.py

@@ -0,0 +1,95 @@
+"""Compile the OpenBoost portion of A6 resource preflight; never launch jobs."""
+
+import argparse
+import hashlib
+import json
+from pathlib import Path
+
+from benchmarks.v1.selection import digest
+
+
+def compile_plan(design):
+    if design["schema"] != "openboost-search-design-v1":
+        raise ValueError("unknown search design")
+    budgets = design["budgets"]
+    if any(
+        budgets[k] != v
+        for k, v in dict(trial_wall_s=1800, ram_mib=8192, cpu_threads=2, trial_retries=0).items()
+    ):
+        raise ValueError("resource policy differs from frozen preflight")
+    if design["selection"]["folds"] != 5 or design["selection"]["trials_per_method"] != 16:
+        raise ValueError("five folds and sixteen trials required")
+    configs = design["families"]["xgboost"]
+    fields = {"rounds", "learning_rate", "max_depth", "reg_lambda", "seed_from_fold"}
+    if (
+        len(configs) != 16
+        or any(set(c) != fields for c in configs)
+        or len({digest(c) for c in configs}) != 16
+        or {c["rounds"] for c in configs} != {300, 1000}
+        or any(c["seed_from_fold"] is not True for c in configs)
+    ):
+        raise ValueError("complete supported frozen numeric-tree configuration family required")
+    jobs = [
+        dict(
+            id=f"openboost-{mode}:{fold}:{index:02}",
+            fold=fold,
+            application="A6",
+            library="openboost",
+            device="cpu",
+            threads=1,
+            seed=fold,
+            early_stopping_rounds=design["shared"]["early_stopping_rounds"],
+            config={**config, "mode": mode, "bins": design["shared"]["bin_budget"]},
+        )
+        for mode in ("shared", "independent")
+        for fold in range(5)
+        for index, config in enumerate(configs)
+    ]
+    return dict(
+        schema="openboost-a6-resource-plan-v1",
+        search_design_sha256=digest(design),
+        scope="OpenBoost-only resource planning; not complete comparator coverage or launch authorization",
+        jobs=jobs,
+        job_count=len(jobs),
+        trials_per_fold=32,
+        maximum_sequential_worker_seconds=len(jobs) * budgets["trial_wall_s"],
+        maximum_reserved_cpu_seconds=len(jobs) * budgets["trial_wall_s"] * budgets["cpu_threads"],
+        policy=dict(
+            timeout_s=budgets["trial_wall_s"],
+            address_limit_bytes=8 * 1024**3,
+            requested_container_memory_mib=budgets["ram_mib"],
+            reserved_cpus=2,
+            worker_threads=1,
+            retries=0,
+            retention="summary",
+        ),
+        first_probe_ids=["openboost-shared:0:00", "openboost-independent:0:00"],
+        stop_on_failure=True,
+        open_requirements=[
+            "Bind verified real train/validation packets without test material",
+            "Run exact full-round workers under protected resource policy",
+            "Complete all required comparator methods and coverage ledger",
+            "Qualify selected-model release and per-target real quality",
+        ],
+    )
+
+
+def main(output):
+    design_path = Path(__file__).with_name("search-design.json")
+    plan = compile_plan(json.loads(design_path.read_text()))
+    plan["input_files"] = {
+        str(path.relative_to(design_path.parent)): hashlib.sha256(path.read_bytes()).hexdigest()
+        for path in (
+            design_path,
+            design_path.parent / "datasets/preprocessing.json",
+            design_path.parent / "datasets/parkinsons.json",
+        )
+    }
+    with Path(output).open("x") as stream:
+        stream.write(json.dumps(plan, indent=2, allow_nan=False) + "\n")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("output", type=Path)
+    main(parser.parse_args().output)