From 910776beaee2ec022495e74ed483b792d30d391d Mon Sep 17 00:00:00 2001 From: Ming Wen Date: Fri, 21 Aug 2026 15:48:33 +0800 Subject: [PATCH 1/2] docs: catch README and ROADMAP up to shipped semantic caching v0.9.0 shipped semantic response caching (#918, #921), but the README still listed it only under Roadmap and described the cache as exact-match only. Move it into the Caching feature entry (wording aligned with the v0.9.0 release notes), fold it into the shipped-since trailer, and drop the ROADMAP Now row. Also narrow the 'Conditional and wildcard routing' Next row to the undelivered remainder: wildcard model rows (provider/*) and tag-conditional targets via x-aisix-routing-tags are already on main. --- README.md | 19 +++++++++++-------- ROADMAP.md | 3 +-- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 0fa044b5..3b6a51d8 100644 --- a/README.md +++ b/README.md @@ -174,10 +174,13 @@ Covered by 183 end-to-end scenario files (496 cases) that run against real gatew Moderation, AWS Bedrock Guardrails, Azure AI Content Safety (Prompt Shield + text moderation), and two Alibaba Cloud services. A block returns `422 content_filter`; monitor mode records what would have happened without blocking. -- **Caching** — exact-match response cache with per-policy TTL and model/key scope matchers; - memory and Redis backends; cost-saved telemetry on every hit. Separately, **automatic - prompt caching** can be enabled per direct Anthropic model to inject cache breakpoints, so - callers get provider-side prompt discounts without changing their requests. +- **Caching** — exact-match and **semantic** response caching with per-policy TTL and + model/key scope matchers; memory and Redis backends; cost-saved telemetry on every hit. + A cache policy carrying a `semantic` block serves a cached answer to a differently-worded + question at or above its cosine similarity threshold — in-process, or shared across + gateway replicas on Redis vector search. Separately, **automatic prompt caching** can be + enabled per direct Anthropic model to inject cache breakpoints, so callers get + provider-side prompt discounts without changing their requests. - **MCP gateway** — front registered upstream MCP servers at `/mcp` with gateway-held credentials, per-server tool namespaces, and per-caller access. It serves every Streamable HTTP revision from `2025-03-26` through stateless `2026-07-28` without @@ -305,14 +308,14 @@ crates/ Highlights on the [roadmap](ROADMAP.md); tracked live in [issues](https://github.com/api7/aisix/issues): -- Semantic (embedding-similarity) response caching - More observability sinks — Langsmith, Helicone, Slack alerts - Prompt templates managed as gateway resources - Llama-Guard as a guardrail provider -Shipped since this list was last written: the MCP gateway, the A2A agent gateway, -OIDC/JWT inbound auth, Redis-backed distributed rate limiting, and the Lakera, Presidio, -PII, and OpenAI Moderation guardrails — see **Features** above. +Shipped since this list was last written: semantic (embedding-similarity) response caching, +the MCP gateway, the A2A agent gateway, OIDC/JWT inbound auth, Redis-backed distributed +rate limiting, and the Lakera, Presidio, PII, and OpenAI Moderation guardrails — see +**Features** above. ## 🛠️ Development diff --git a/ROADMAP.md b/ROADMAP.md index 8b5ca480..bf82edcf 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -20,14 +20,13 @@ The **Surface** column shows where a capability lands: **Gateway** is the AISIX | Enterprise SSO | Single sign-on through SAML and generic OIDC, beyond today's social logins. | Cloud | | Directory sync (SCIM) | Provision and deprovision users and groups from your identity provider. | Cloud | | Service accounts | Login-less, first-class principals for automated callers. | Cloud | -| Semantic caching | Serve responses for prompts close in meaning, on top of today's exact-match cache. | Gateway | ## Next | Capability | What's planned | Surface | | --- | --- | --- | | Fine-grained authorization | Custom roles with per-resource and per-action permissions, beyond today's fixed roles and read/write scopes. | Cloud | -| Conditional and wildcard routing | Route on request metadata, headers, and tags, and match upstreams by wildcard names such as `provider/*`. | Gateway | +| Conditional routing | Route on request metadata and arbitrary headers, beyond today's tag-conditional targets and wildcard model names such as `provider/*`. | Gateway | | Prompt management | Store, version, and reuse prompt templates with variables, resolved at the gateway. | Gateway · Cloud | | Caller key rotation experience | Self-service key rotation in the dashboard, plus scheduled auto-rotation with a grace overlap. | Cloud | | Production-path playground | Run the Cloud playground through a connected AISIX gateway so it reflects real routing, caching, guardrails, and rate limiting. | Cloud | From fbc386b4f1a49a5f11c13aa4ee233c42ff09c200 Mon Sep 17 00:00:00 2001 From: Ming Wen Date: Fri, 21 Aug 2026 15:58:31 +0800 Subject: [PATCH 2/2] docs: resource collections are thirteen; note semantic-cache limits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit follow-up: the declarative-configuration bullet still said ten collections (filesource KINDS is 13 on main — claim_mappings shipped in v0.9.0, plus passthrough_routes and mcp_auth_settings); 'scope matchers' collided with the v0.9.0 scope field, renamed to match rules; and the semantic sentence now carries the textual-only and per-key isolation caveats from the release notes. --- README.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 3b6a51d8..4efa89ce 100644 --- a/README.md +++ b/README.md @@ -175,12 +175,14 @@ Covered by 183 end-to-end scenario files (496 cases) that run against real gatew moderation), and two Alibaba Cloud services. A block returns `422 content_filter`; monitor mode records what would have happened without blocking. - **Caching** — exact-match and **semantic** response caching with per-policy TTL and - model/key scope matchers; memory and Redis backends; cost-saved telemetry on every hit. + model/key match rules; memory and Redis backends; cost-saved telemetry on every hit. A cache policy carrying a `semantic` block serves a cached answer to a differently-worded question at or above its cosine similarity threshold — in-process, or shared across - gateway replicas on Redis vector search. Separately, **automatic prompt caching** can be - enabled per direct Anthropic model to inject cache breakpoints, so callers get - provider-side prompt discounts without changing their requests. + gateway replicas on Redis vector search. Only fully textual requests match semantically, + and entries stay private to the caller's API key unless the policy sets `scope: env`. + Separately, **automatic prompt caching** can be enabled per direct Anthropic model to + inject cache breakpoints, so callers get provider-side prompt discounts without changing + their requests. - **MCP gateway** — front registered upstream MCP servers at `/mcp` with gateway-held credentials, per-server tool namespaces, and per-caller access. It serves every Streamable HTTP revision from `2025-03-26` through stateless `2026-07-28` without @@ -196,9 +198,10 @@ Covered by 183 end-to-end scenario files (496 cases) that run against real gatew events, OTLP/GenAI span export (Langfuse, Honeycomb, Grafana Cloud, or any OTLP receiver), plus dedicated Datadog and Aliyun SLS log exporters and object-storage (S3/GCS/Azure Blob) telemetry. -- **Declarative configuration** — one `resources.yaml` carries all ten resource collections - (provider keys, models, caller keys, guardrails, MCP servers, A2A agents, cache policies, - observability exporters, rate-limit policies, OIDC providers), validated against the same +- **Declarative configuration** — one `resources.yaml` carries all thirteen resource + collections (provider keys, models, caller keys, guardrails, MCP servers, MCP auth + settings, A2A agents, cache policies, observability exporters, rate-limit policies, + OIDC providers, JWT claim mappings, passthrough routes), validated against the same JSON Schemas the gateway uses at runtime. `aisix validate` checks a file offline; `SIGHUP` reloads it atomically. - **Operational endpoints** — `/livez` and `/readyz` on the proxy listener; `/status/config`,