Skip to content

feat(upstream): verify the upstream certificate against configurable CAs - #13863

Open
nic-6443 wants to merge 3 commits into
apache:masterfrom
nic-6443:feat/upstream-tls-verify-ca
Open

feat(upstream): verify the upstream certificate against configurable CAs#13863
nic-6443 wants to merge 3 commits into
apache:masterfrom
nic-6443:feat/upstream-tls-verify-ca

Conversation

@nic-6443

@nic-6443 nic-6443 commented Aug 21, 2026

Copy link
Copy Markdown
Member

Description

upstream.tls.verify has been in the schema for a while, but only the kafka scheme ever read it — for an https/grpcs upstream APISIX connected without ever checking the certificate it was handed. This makes the field mean what it says for those schemes, and adds upstream.tls.ca_certs so the trust anchors can be picked per upstream instead of only globally through ssl_trusted_certificate.

{
  "scheme": "https",
  "type": "roundrobin",
  "nodes": { "127.0.0.1:8443": 1 },
  "tls": {
    "verify": true,
    "ca_certs": ["<content of ca.crt>"]
  }
}

Both settings are applied with the apisix-nginx-module upstream API (set_ssl_verify / set_ssl_trusted_store), so this needs APISIX-Runtime — on a plain OpenResty build a route carrying either field gets the usual "need to build APISIX-Runtime" error instead of a nil call.

A few details worth calling out:

  • verify loses its false default. Unset now means "fall back to the nginx configuration", which is what happens today; the kafka scheme reads a nil exactly the way it read a false, so nothing changes there.
  • grpcs cannot apply these in set_by_route. Both settings live in the apisix-nginx-module request context, and ngx.exec("@grpc_pass") goes through ngx_http_named_location(), which zeroes r->ctx — the same reason the client certificate is already re-applied from grpc_access_phase. So grpcs applies them there.
  • @grpc_pass also gains grpc_ssl_name $upstream_host. nginx reads it only once verification is on, and without it the upstream certificate would be checked against the name of the upstream block, apisix_backend, so grpcs verification could never pass. It mirrors proxy_ssl_name $upstream_host on the proxy_pass side and is a no-op while verify is unset.
  • The parsed X509_STORE is cached under the ca_certs array itself rather than under the config version. In standalone mode every resource loaded in one reload shares a modifiedIndex, so a version-keyed entry would hand one upstream's CA certificates to another.

APISIX-Runtime 1.3.17

ca_certs without a client certificate needs api7/apisix-nginx-module#126: ngx_http_apisix_set_upstream_ssl() used to apply the trusted store only inside its ctx->upstream_cert != NULL branch, and nginx loads proxy_ssl_trusted_certificate into the SSL_CTX only when proxy_ssl_verify is on at configuration time, which APISIX never sets — so an upstream that asked for verification and supplied its own CA ended up with an empty trust store.

That is the field's main use, so the runtime bump travels with this PR rather than separately; otherwise this could merge onto 1.3.16 and quietly do nothing. 1.3.17 also picks up ngx_http_ffi_client v0.1.2/v0.1.3.

Which issue(s) this PR fixes:

N/A

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

Tests

t/node/upstream-tls2.t covers verify on its own over https, then ca_certs with no client certificate anywhere — an upstream on :8768 whose certificate proxy_ssl_trusted_certificate does not cover, rejected with verify alone and accepted once ca_certs names its CA. That pair fails on 1.3.16 and passes on 1.3.17. The file ends with grpcs blocks for both fields; they fail without the grpc_access_phase half of the change.

t/node/upstream-mtls2.t covers ca_certs alongside a client certificate: a non-certificate entry rejected by the Admin API, a CA that does not match, one that does, and several CAs where only the last one anchors the chain.

t/node/upstream-mtls.t TEST 21 and TEST 23 set tls.verify on an https upstream to check that it does not disturb the client certificate path. That field is no longer inert there, and the test upstream serves a self-signed certificate issued for another name, so they now ask for false — what they were written to assert is unchanged.

Copilot AI lite review requested due to automatic review settings August 21, 2026 08:02
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. enhancement New feature or request labels Aug 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

`upstream.tls.verify` existed but only the `kafka` scheme read it, so an
HTTP/gRPC upstream was always reached without checking the certificate it
presented. Give it meaning for `https` and `grpcs` through the
apisix-nginx-module upstream API, and add `upstream.tls.ca_certs` so the
trust anchors can be set per upstream instead of only through
`ssl_trusted_certificate` in `config.yaml`.

`verify` loses its `false` default: leaving it unset now falls back to the
nginx configuration, which keeps today's behaviour, while the `kafka`
scheme reads a nil the same way it read a false.

The settings live in the apisix-nginx-module request context, which is
wiped by the internal redirect to `@grpc_pass`, so `grpcs` applies them
from `grpc_access_phase` the way the client certificate already does.
`@grpc_pass` also gains `grpc_ssl_name $upstream_host`: nginx only reads
it once verification is on, and without it the certificate would be
checked against the name of the upstream block, `apisix_backend`.

`t/node/upstream-mtls.t` TEST 21 and TEST 23 set `tls.verify` on an https
upstream to check that it does not disturb the client certificate path.
That field is no longer inert there, and the test upstream serves a
self-signed certificate for another name, so they now ask for `false`.
@nic-6443
nic-6443 force-pushed the feat/upstream-tls-verify-ca branch from b138e4c to 5794995 Compare August 21, 2026 09:40
1.3.17 builds against apisix-nginx-module 1.19.10, which makes
`upstream.set_ssl_trusted_store()` take effect without a client
certificate (api7/apisix-nginx-module#126) - `upstream.tls.ca_certs`
in apache#13863 depends on it - and picks up ngx_http_ffi_client v0.1.2/v0.1.3.
Every `ca_certs` block so far pairs the field with a client certificate,
because apisix-nginx-module before 1.19.10 applied the trusted store only
inside its `ctx->upstream_cert != NULL` branch. That left the field's main
use - verifying an upstream without doing mTLS - untested.

Add an upstream on :8768 serving a certificate that
`proxy_ssl_trusted_certificate` does not cover: it is rejected with
`verify` alone and accepted once `ca_certs` names its CA, with no client
certificate anywhere. The pair fails on APISIX-Runtime 1.3.16 and passes
on 1.3.17.

The new blocks go before the grpcs ones: those run under the yaml config
provider, and a block that follows them still looks for `conf/apisix.yaml`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants