diff --git a/contracts/agents-api/harness-onboarding.md b/contracts/agents-api/harness-onboarding.md index 8a22dc571..5ff3c6d5c 100644 --- a/contracts/agents-api/harness-onboarding.md +++ b/contracts/agents-api/harness-onboarding.md @@ -224,7 +224,7 @@ python services/core/tests/qualify_public_native.py \ | Suite | Operations | Placement | | --- | --- | --- | -| `none` | Creation retry, foreign history rejection, two native text Turns, history recall, SSE ordering and SDK/raw Item and Turn parity | `none` | +| `none` | Initial-input creation/retry/conflict with one Turn, foreign history rejection, two native text Turns, history recall, SSE ordering, SDK/raw schema parity and nullable usage accounting | `none` | | `pending-actions` | Query/reconnect pending calls, success/error results, cancellation, exact target rejection, retries and durable Items | Any declared placement with function tools | | `functions` | SDK handlers, success/error, native file and Artifact bytes, continuation, pending cancellation and tenant isolation | Workspace | | `images` | Initial and active images, image results, retry/atomic rejection, native files/Artifacts, isolation and continuation | Workspace with declared image and function support | @@ -239,6 +239,8 @@ For `self_hosted`, choose a custom absolute `workspace_directory`. When the runn Warm continuation is the default and records cold recovery as `unverified`. To qualify cold continuation, additionally pass `--compose-directory` with the owned installation's absolute directory and `--compose-project` with its exact project name. The runner restarts only that project's `agent-host`, confirms its container start time changed, and then runs the unchanged history assertions. This does not qualify a Core restart or a sandbox checkpoint restore. The `pending-actions` suite reconnects the public client, not the agent-host process, and rejects those restart options. Select cold recovery only where the [declaration and coverage ledger](./index.md#known-gaps) support it; unsupported recovery remains a gap, never a successful skipped check. An API rejection fails the selected suite. +The `none` suite validates required and nullable Session, Turn, Item and event fields against the pinned schema. When native Turns supply measured usage, it checks the terminal event against the stored Turn and sums the measurements into Session totals. Unknown usage remains `null` and each affected Turn is listed in the evidence’s `proof.unverified`; a passing suite does not claim native measurement support for those Turns. + Run `python -m unittest discover -s services/core/tests -p qualify_public_native_test.py` with the pinned SDK to check credential handling and the owned restart boundary without a model. Existing deterministic Core integration tests remain the authority for schema validation, atomic admission, durable receipts and rejection semantics. Real-model results qualify only the selected suite, protocol, Harness and placement. Provider lifecycle, native identity, credential isolation, deferred discovery and unselected suites need separate evidence; view-only results do not qualify the public path. ## Native version pins diff --git a/contracts/agents-api/zh/harness-onboarding.md b/contracts/agents-api/zh/harness-onboarding.md index e4e271fa3..a29da6a56 100644 --- a/contracts/agents-api/zh/harness-onboarding.md +++ b/contracts/agents-api/zh/harness-onboarding.md @@ -1,7 +1,7 @@ --- title: "添加 Harness" source: contracts/agents-api/harness-onboarding.md -source_hash: 32b85617efe41a23bd25768a42fd7a57019c75ae723d07d8cc5106124709f402 +source_hash: 84ebd7cebed1995c89a59f8c4c71542ca41ef6ccc890b4eba96340a970ae161c --- **Harness** 是一种运行模型和工具循环的原生代理引擎（Codex、Claude Code、MiniMax Code）。**Harness 适配器**将 Runtime 的 Executor 和 Turn 契约转换到该引擎的 SDK 或协议。本文档定义 Runtime–Harness 协议：适配器接口及其生命周期义务、注册、支持声明和验收。 @@ -226,7 +226,7 @@ python services/core/tests/qualify_public_native.py \ | 套件 | 操作 | 放置方式 | | --- | --- | --- | -| `none` | 创建重试、外部历史拒绝、两个原生文本 Turn、历史回忆、SSE 顺序，以及 SDK/原始 Item 和 Turn 一致性 | `none` | +| `none` | 带初始输入的创建、重试和冲突保持单个 Turn，跨项目历史拒绝，两个原生文本 Turn，历史回忆，SSE 顺序，SDK／原始响应 schema 一致性与可空 usage 核算 | `none` | | `pending-actions` | 查询和重连待处理调用、成功/错误结果、取消、精确目标拒绝、重试和持久化 Item | 声明支持函数工具的任意放置方式 | | `functions` | SDK handler、成功/错误、原生文件和 Artifact 字节、继续执行、待处理调用取消和租户隔离 | 工作区 | | `images` | 初始和活动图像、图像结果、重试/原子拒绝、原生文件/Artifact、隔离和继续执行 | 声明支持图像和函数的工作区 | @@ -241,6 +241,8 @@ python services/core/tests/qualify_public_native.py \ 默认验证热继续执行，并将冷恢复记录为 `unverified`。验证冷继续执行时，额外传入指向所拥有安装的绝对目录的 `--compose-directory`，以及指定其精确项目名称的 `--compose-project`。运行器仅重启该项目的 `agent-host`，确认容器启动时间已改变，然后执行相同的历史断言。这不能证明 Core 重启或 sandbox 检查点恢复。`pending-actions` 套件重连的是公共客户端，而非 agent-host 进程，因此拒绝这些重启选项。仅在[声明和覆盖台账](./index.md#known-gaps) 支持时选择冷恢复；不支持的恢复仍是缺口，不能把跳过的检查记为成功。API 拒绝会使所选套件失败。 +`none` 套件依据固定版本 schema 校验 Session、Turn、Item 和事件的必需字段及可空字段。原生 Turn 提供用量测量时，它会对比终止事件与已存储 Turn，并将测量值求和核对 Session 总量。未知用量保留为 `null`，每个受影响 Turn 都列入证据的 `proof.unverified`；套件通过不代表这些 Turn 已验证原生用量测量支持。 + 使用锁定的 SDK 运行 `python -m unittest discover -s services/core/tests -p qualify_public_native_test.py`，可在不调用模型的情况下检查凭据处理和所拥有的重启边界。现有确定性 Core 集成测试仍负责 schema 验证、原子准入、持久化回执和拒绝语义。真实模型结果仅证明所选套件、协议、Harness 和放置方式。Provider 生命周期、原生身份、凭据隔离、延迟发现和未选择的套件需要独立证据；仅通过 view 测试不能证明公共调用路径。 ## 原生版本固定 {#native-version-pins} diff --git a/services/core/tests/official_session_initial_input.py b/services/core/tests/official_session_initial_input.py index c5cdc1b84..4783ffd9b 100644 --- a/services/core/tests/official_session_initial_input.py +++ b/services/core/tests/official_session_initial_input.py @@ -10,6 +10,18 @@ from official_session_creation_stream import verify_creation_streams +def verify_initial_input_retry(sessions, raw, endpoint, headers, request, session_id): + """Retry and conflict preserve the one Turn admitted by Session creation.""" + turns = [turn.id for turn in sessions.turns.list(session_id)] + assert len(turns) == 1 + reply = raw.post(endpoint, headers=headers, json=request) + assert reply.status_code == 201 and reply.json()["id"] == session_id + changed = raw.post(endpoint, headers=headers, json={**request, "input": "Changed " + uuid.uuid4().hex}) + assert changed.status_code == 409 + assert [turn.id for turn in sessions.turns.list(session_id)] == turns + return reply, changed + + def main(): base, token, foreign, unsupported = sys.argv[1:] headers = {"Authorization": "Bearer " + token, "OpenAI-Beta": "agents=v1"} @@ -56,16 +68,11 @@ def main(): assert len(turns) == 1 items = list(sessions.items.list(session.id, order="asc")) assert [item.content[0].text for item in items] == (["First"] if i == 0 else ["First", "Second"]) - reply = raw.post(base + "/v1/agents/sessions", headers={**headers, **key}, - json={**configuration, "input": initial}) - assert reply.status_code == 201 and reply.json()["id"] == session.id - assert [turn.id for turn in sessions.turns.list(session.id)] == [turns[0].id] + verify_initial_input_retry(sessions, raw, base + "/v1/agents/sessions", {**headers, **key}, + {**configuration, "input": initial}, session.id) denied = raw.get(base + "/v1/agents/sessions/" + session.id, headers={**headers, "Authorization": "Bearer " + foreign}) assert denied.status_code == 404 - changed = raw.post(base + "/v1/agents/sessions", headers={**headers, **key}, - json={**configuration, "input": "Changed"}) - assert changed.status_code == 409 sessions.events.create(session.id, events=[{"type": "agent.session.input.cancel"}]) assert sessions.create(**configuration, input=initial, extra_headers=key).status == "idle" assert len(list(sessions.turns.list(session.id))) == 1 diff --git a/services/core/tests/qualify_public_native.py b/services/core/tests/qualify_public_native.py index c5dd67afc..6d8a3cfa4 100644 --- a/services/core/tests/qualify_public_native.py +++ b/services/core/tests/qualify_public_native.py @@ -14,11 +14,14 @@ import httpx2 from openai import OpenAI +from jsonschema import Draft202012Validator from official_hosted_functions_native import verify_hosted_functions from official_hosted_structured_native import verify_hosted_structured from official_pending_actions_native import verify_pending_actions from official_workspace_images_native import verify_workspace_images +from official_schema import ResponseValidator +from official_session_initial_input import verify_initial_input_retry from official_environment_composition import verify_composition from session_cleanup import delete_session @@ -33,46 +36,116 @@ def verify_none(client, foreign, http, agent_options, session_options, ready, re sessions = client.beta.agents.sessions marker = "memory-" + uuid.uuid4().hex key = "create-" + uuid.uuid4().hex - session = sessions.create(agent=agent_options, **session_options, idempotency_key=key) - proof = {"checks": [], "session": session.id, "runs": []} - root = str(client.base_url).rstrip("/") + "/agents/sessions/" + session.id + initial = "Remember " + marker + ". Reply with exactly that string. Do not use tools." + request = {"agent": agent_options, **session_options, "input": initial} + request.update(request.pop("extra_body", {})) + endpoint = str(client.base_url).rstrip("/") + "/agents/sessions" headers = {"Authorization": "Bearer " + client.api_key, "OpenAI-Beta": "agents=v1"} + validate = ResponseValidator(Path(__file__).resolve().parents[3] / "contracts/agents-api/openapi.yaml") + events_schema = Draft202012Validator({"components": validate.contract["components"], + "$ref": "#/components/schemas/SessionEvent"}) + session = None + proof = {"checks": [], "runs": [], "snapshots": [], "unverified": []} try: - assert sessions.create(agent=agent_options, **session_options, idempotency_key=key).id == session.id - ready(session) - for suffix in ("", "/items", "/turns"): - assert http.get(root + suffix, headers={**headers, "Authorization": "Bearer " + foreign.api_key}).status_code == 404 - proof["checks"].append("create_retry_and_foreign_history_rejection") - for index, prompt in enumerate(("Remember " + marker + ". Reply with exactly that string. Do not use tools.", - "Recall the string from our previous turn. Reply with exactly that string. Do not use tools.")): - if index and restart is not None: - restart() - with sessions.stream(session.id, input=prompt, idempotency_key=key + str(index), timeout=240) as stream: - events = [event.to_dict() for event in stream] + for index in range(2): + if index == 0: + stream = sessions.create(agent=agent_options, **session_options, input=initial, + stream=True, extra_headers={"Idempotency-Key": key}, timeout=240) + else: + if restart is not None: + restart() + stream = sessions.stream(session.id, + input="Recall the string from our previous turn. Reply with exactly that string. Do not use tools.", + idempotency_key=key + "-continue", timeout=240) + events = [] proof["runs"].append(events) + with stream: + for event in stream: + value = event.to_dict() + events.append(value) + if index == 0 and len(events) == 1: + assert event.type == "agent.session.created" + session = event.session + proof["session"] = session.id + events_schema.validate(value) + ready(session) + root = endpoint + "/" + session.id + # Admission has committed before the first creation event. + for response in verify_initial_input_retry(sessions, http, endpoint, + {**headers, "Idempotency-Key": key}, request, session.id): + validate(response) + assert sessions.create(agent=agent_options, **session_options, input=initial, + extra_headers={"Idempotency-Key": key}).id == session.id + for suffix in ("", "/items", "/turns"): + response = http.get(root + suffix, headers={**headers, "Authorization": "Bearer " + foreign.api_key}) + assert response.status_code == 404 + validate(response) + else: + events_schema.validate(value) record(proof) types = [event["type"] for event in events] terminals = [event for event in events if event["type"] in { "agent.session.turn.completed", "agent.session.turn.failed", "agent.session.turn.cancelled"}] assert len(terminals) == 1 and terminals[0]["type"] == "agent.session.turn.completed" + terminal = terminals[0] assert types[-1] == "agent.session.idle" + assert types.count("agent.session.turn.created") == 1 assert types.index("agent.session.turn.created") < types.index("agent.session.turn.completed") < len(types) - 1 assert len({event["event_id"] for event in events}) == len(events) + assert all("usage" not in event for event in events if event is not terminal) + snapshots = {} for name in ("items", "turns"): response = http.get(root + "/" + name, headers=headers, params={"limit": 100, "order": "asc"}) assert response.status_code == 200 and not response.json()["has_more"] + validate(response) expected = getattr(sessions, name).list(session.id, limit=100, order="asc").to_dict() assert response.json() == expected - items = list(sessions.items.list(session.id, limit=100, order="asc")) - answers = [item.to_dict() for item in items if item.type == "message" and item.role == "assistant"] + snapshots[name] = response.json()["data"] + response = http.get(root, headers=headers) + assert response.status_code == 200 + validate(response) + current = response.json() + assert current == sessions.retrieve(session.id).to_dict() + assert current["status"] == "idle" and current["required_actions"] == [] and current["error"] is None + turns = snapshots["turns"] + assert len(turns) == index + 1 + assert all(turn["status"] == "completed" and turn["error"] is None and turn["subagent_id"] is None for turn in turns) + assert terminal["turn_id"] == turns[-1]["id"] + assert terminal["turn"] == turns[-1] and terminal["usage"] == turns[-1]["usage"] + answers = [item for item in snapshots["items"] if item["type"] == "message" and item["role"] == "assistant"] assert marker in "".join(part.get("text", "") for part in answers[-1]["content"]) - assert len(sessions.turns.list(session.id).data) == index + 1 - assert sessions.retrieve(session.id).required_actions == [] - proof["checks"].append(("cold_agent_host" if restart is not None else "warm") + "_text_history_sse_and_sdk_raw_parity") + measured = [turn["usage"] for turn in turns if turn["usage"] is not None] + for usage in measured: + assert usage["input_tokens"] > 0 and usage["output_tokens"] > 0 + assert usage["total_tokens"] == usage["input_tokens"] + usage["output_tokens"] + assert 0 <= usage["input_tokens_details"]["cached_tokens"] <= usage["input_tokens"] + assert 0 <= usage["output_tokens_details"]["reasoning_tokens"] <= usage["output_tokens"] + if len(measured) == len(turns): + totals = {field: sum(usage[field] for usage in measured) + for field in ("input_tokens", "output_tokens", "total_tokens")} + for field, detail in (("input_tokens_details", "cached_tokens"), ("output_tokens_details", "reasoning_tokens")): + totals[field] = {detail: sum(usage[field][detail] for usage in measured)} + assert current["usage"] == totals + else: + assert current["usage"] is None + for turn in turns: + if turn["usage"] is None: + gap = "Native measured usage unavailable for Turn " + turn["id"] + if gap not in proof["unverified"]: + proof["unverified"].append(gap) + proof["snapshots"].append({**snapshots, "session": current}) + if index == 0: + for response in verify_initial_input_retry(sessions, http, endpoint, + {**headers, "Idempotency-Key": key}, request, session.id): + validate(response) + proof["checks"].append("initial_input_create_retry_conflict_one_native_turn_and_foreign_history_rejection") + proof["checks"].append("native_usage_measurements_or_explicit_unknown_and_session_totals") + proof["checks"].append(("cold_agent_host" if restart is not None else "warm") + "_text_history_sse_and_sdk_raw_schema_parity") return proof["checks"] finally: record(proof) - delete_session(sessions, session.id) + if session is not None: + delete_session(sessions, session.id) def main(): diff --git a/services/core/tests/qualify_public_native_test.py b/services/core/tests/qualify_public_native_test.py index 5612da39c..1214e26b0 100644 --- a/services/core/tests/qualify_public_native_test.py +++ b/services/core/tests/qualify_public_native_test.py @@ -1,6 +1,7 @@ """Check qualification's secret handling and owned restart boundary without a model.""" import contextlib +import copy import base64 import io import json @@ -14,6 +15,7 @@ from unittest.mock import MagicMock, patch import httpx2 +from jsonschema import ValidationError from openai import BadRequestError, NotFoundError, OpenAI from official_environment_files_native import generate_files @@ -97,7 +99,7 @@ def reject(request): client = OpenAI(base_url="https://core.example/v1", api_key="project-secret", http_client=http, max_retries=0) options = {"environment": {"type": "self_hosted", "workspace_directory": "/custom/work"}, "extra_body": {"x_agents_core": {"model_provider": self.settings["model_provider"]}}} - for verify in (qualification.verify_hosted_functions, qualification.verify_workspace_images, qualification.verify_pending_actions): + for verify in (qualification.verify_none, qualification.verify_hosted_functions, qualification.verify_workspace_images, qualification.verify_pending_actions): kwargs = {"ready": lambda session: None, "record": lambda proof: None} if verify is not qualification.verify_pending_actions: kwargs["restart"] = None @@ -265,5 +267,142 @@ def test_mcp_hold_waits_for_directory_and_turn_without_hiding_errors(self): self.assertEqual(sessions.events.create.call_count, 3) +class NoneSessionTests(unittest.TestCase): + def run_none(self, measured=True, fault=None): + client, foreign, http = MagicMock(), SimpleNamespace(api_key="foreign"), MagicMock() + client.base_url, client.api_key = "https://core.example/v1", "caller" + sessions = client.beta.agents.sessions + usage = {"input_tokens": 7, "input_tokens_details": {"cached_tokens": 2}, "output_tokens": 3, + "output_tokens_details": {"reasoning_tokens": 1}, "total_tokens": 10} if measured else None + current = {"id": "session", "object": "agent.session", "metadata": {}, "created_at": 1, "last_active_at": 1, + "status": "idle", "required_actions": [], "error": None, "environment": {"type": "none"}, + "vault_ids": [], "usage": None, "agent": {"id": "agent", "name": None, "model": "fixture", + "reasoning": {"effort": None, "summary": None}, "text": {"format": {"type": "text"}, "verbosity": "medium"}, + "service_tier": "auto", "instructions": None, "tools": [], + "multi_agent": {"enabled": False, "max_concurrent_subagents": None}}} + turns, items, prompts, proofs = [], [], [], [] + + def sdk(value): + return SimpleNamespace(**value, to_dict=lambda: copy.deepcopy(value)) + + def page(values): + return {"object": "list", "data": copy.deepcopy(values), "has_more": False, + "first_id": values[0]["id"] if values else None, "last_id": values[-1]["id"] if values else None} + + def listing(values): + result = MagicMock() + result.to_dict.side_effect = lambda: page(values) + result.__iter__.side_effect = lambda: iter([sdk(value) for value in values]) + return result + + sessions.items.list.side_effect = lambda *a, **k: listing(items) + sessions.turns.list.side_effect = lambda *a, **k: listing(turns) + sessions.retrieve.side_effect = lambda *a, **k: sdk(current) + + def events(initial): + number = len(turns) + 1 + turn = {"id": "turn-" + str(number), "object": "agent.session.turn", "session_id": "session", "agent_id": "agent", + "subagent_id": None, "status": "in_progress", "created_at": number, "started_at": number, + "completed_at": None, "error": None, "usage": None} + turns.append(turn) + current.update(status="in_progress", usage=None) + if initial: + created = {"type": "agent.session.created", "event_id": "created", "session": copy.deepcopy(current)} + yield SimpleNamespace(type=created["type"], session=sdk(current), to_dict=lambda: created) + yield sdk({"type": "agent.session.turn.created", "event_id": str(number) + "-created", "session_id": "session", + "turn_id": turn["id"], "turn": copy.deepcopy(turn)}) + turn.update(status="completed", completed_at=number, usage=copy.deepcopy(usage)) + marker = prompts[0].split("Remember ", 1)[1].split(".", 1)[0] + items.append({"id": "answer-" + str(number), "turn_id": turn["id"], "phase": None, "type": "message", "role": "assistant", "status": "completed", + "content": [{"type": "output_text", "text": marker}]}) + if measured: + current["usage"] = {"input_tokens": 7 * number, "input_tokens_details": {"cached_tokens": 2 * number}, + "output_tokens": 3 * number, "output_tokens_details": {"reasoning_tokens": number}, "total_tokens": 10 * number} + if fault == "totals": + current["usage"]["total_tokens"] += 1 + current["status"] = "idle" + terminal_usage = copy.deepcopy(usage) + if fault == "terminal_usage": + terminal_usage["total_tokens"] += 1 + yield sdk({"type": "agent.session.turn.completed", "event_id": str(number) + "-completed", "session_id": "session", + "turn_id": turn["id"], "turn": copy.deepcopy(turn), "usage": terminal_usage}) + yield sdk({"type": "agent.session.idle", "event_id": str(number) + "-idle", "session": copy.deepcopy(current)}) + + def create(**options): + if not options.get("stream"): + return sdk(current) + prompts.append(options["input"]) + stream = MagicMock() + stream.__iter__.side_effect = lambda: events(True) + return stream + + def continuation(*args, **options): + stream = MagicMock() + stream.__iter__.side_effect = lambda: events(False) + return stream + + sessions.create.side_effect, sessions.stream.side_effect = create, continuation + + def response(method, url, status, body): + return httpx2.Response(status, json=body, request=httpx2.Request(method, url)) + + def post(url, headers, json): + if json["input"].startswith("Changed "): + return response("POST", url, 409, {"error": {"message": "Conflict", "type": "conflict_error", "code": "conflict_error", "param": None}}) + if fault == "duplicate": + turns.append({**turns[0], "id": "duplicate"}) + return response("POST", url, 201, current) + + def get(url, headers, **kwargs): + if headers["Authorization"] == "Bearer foreign": + return response("GET", url, 404, {"error": {"message": "Missing", "type": "not_found_error", "code": "not_found", "param": None}}) + if url.endswith("/items"): + body = page(items) + elif url.endswith("/turns"): + body = page(turns) + else: + body = copy.deepcopy(current) + if fault == "missing_usage": + del body["usage"] + return response("GET", url, 200, body) + + http.get.side_effect, http.post.side_effect = get, post + restart = MagicMock() + with patch.object(qualification, "delete_session") as delete: + checks = qualification.verify_none(client, foreign, http, {"model": "fixture"}, {"environment": {"type": "none"}}, + ready=lambda session: None, restart=restart, record=lambda proof: proofs.append(copy.deepcopy(proof))) + delete.assert_called_once_with(sessions, "session") + restart.assert_called_once_with() + self.assertEqual(len(turns), 2) + return checks, proofs[-1] + + def test_measured_turns_and_session_totals_survive_continuation(self): + checks, proof = self.run_none() + self.assertEqual(proof["unverified"], []) + self.assertEqual(proof["snapshots"][-1]["session"]["usage"]["total_tokens"], 20) + self.assertIn("initial_input_create_retry_conflict_one_native_turn_and_foreign_history_rejection", checks) + + def test_unknown_usage_is_explicit_not_a_measurement_claim(self): + _, proof = self.run_none(measured=False) + self.assertEqual(len(proof["unverified"]), 2) + self.assertIsNone(proof["snapshots"][-1]["session"]["usage"]) + + def test_missing_required_usage_is_not_nullable_usage(self): + with self.assertRaises(ValidationError): + self.run_none(fault="missing_usage") + + def test_rejects_incorrect_session_sum(self): + with self.assertRaises(AssertionError): + self.run_none(fault="totals") + + def test_terminal_usage_must_match_the_stored_turn(self): + with self.assertRaises(AssertionError): + self.run_none(fault="terminal_usage") + + def test_creation_retry_cannot_append_a_turn(self): + with self.assertRaises(AssertionError): + self.run_none(fault="duplicate") + + if __name__ == "__main__": unittest.main()