feat(upstream): verify the upstream certificate against configurable CAs - #13863
Open
nic-6443 wants to merge 3 commits into
Open
feat(upstream): verify the upstream certificate against configurable CAs#13863nic-6443 wants to merge 3 commits into
nic-6443 wants to merge 3 commits into
Conversation
`upstream.tls.verify` existed but only the `kafka` scheme read it, so an HTTP/gRPC upstream was always reached without checking the certificate it presented. Give it meaning for `https` and `grpcs` through the apisix-nginx-module upstream API, and add `upstream.tls.ca_certs` so the trust anchors can be set per upstream instead of only through `ssl_trusted_certificate` in `config.yaml`. `verify` loses its `false` default: leaving it unset now falls back to the nginx configuration, which keeps today's behaviour, while the `kafka` scheme reads a nil the same way it read a false. The settings live in the apisix-nginx-module request context, which is wiped by the internal redirect to `@grpc_pass`, so `grpcs` applies them from `grpc_access_phase` the way the client certificate already does. `@grpc_pass` also gains `grpc_ssl_name $upstream_host`: nginx only reads it once verification is on, and without it the certificate would be checked against the name of the upstream block, `apisix_backend`. `t/node/upstream-mtls.t` TEST 21 and TEST 23 set `tls.verify` on an https upstream to check that it does not disturb the client certificate path. That field is no longer inert there, and the test upstream serves a self-signed certificate for another name, so they now ask for `false`.
nic-6443
force-pushed
the
feat/upstream-tls-verify-ca
branch
from
August 21, 2026 09:40
b138e4c to
5794995
Compare
5 tasks
1.3.17 builds against apisix-nginx-module 1.19.10, which makes `upstream.set_ssl_trusted_store()` take effect without a client certificate (api7/apisix-nginx-module#126) - `upstream.tls.ca_certs` in apache#13863 depends on it - and picks up ngx_http_ffi_client v0.1.2/v0.1.3.
Every `ca_certs` block so far pairs the field with a client certificate, because apisix-nginx-module before 1.19.10 applied the trusted store only inside its `ctx->upstream_cert != NULL` branch. That left the field's main use - verifying an upstream without doing mTLS - untested. Add an upstream on :8768 serving a certificate that `proxy_ssl_trusted_certificate` does not cover: it is rejected with `verify` alone and accepted once `ca_certs` names its CA, with no client certificate anywhere. The pair fails on APISIX-Runtime 1.3.16 and passes on 1.3.17. The new blocks go before the grpcs ones: those run under the yaml config provider, and a block that follows them still looks for `conf/apisix.yaml`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
upstream.tls.verifyhas been in the schema for a while, but only thekafkascheme ever read it — for anhttps/grpcsupstream APISIX connected without ever checking the certificate it was handed. This makes the field mean what it says for those schemes, and addsupstream.tls.ca_certsso the trust anchors can be picked per upstream instead of only globally throughssl_trusted_certificate.{ "scheme": "https", "type": "roundrobin", "nodes": { "127.0.0.1:8443": 1 }, "tls": { "verify": true, "ca_certs": ["<content of ca.crt>"] } }Both settings are applied with the apisix-nginx-module upstream API (
set_ssl_verify/set_ssl_trusted_store), so this needs APISIX-Runtime — on a plain OpenResty build a route carrying either field gets the usual "need to build APISIX-Runtime" error instead of a nil call.A few details worth calling out:
verifyloses itsfalsedefault. Unset now means "fall back to the nginx configuration", which is what happens today; thekafkascheme reads a nil exactly the way it read afalse, so nothing changes there.grpcscannot apply these inset_by_route. Both settings live in the apisix-nginx-module request context, andngx.exec("@grpc_pass")goes throughngx_http_named_location(), which zeroesr->ctx— the same reason the client certificate is already re-applied fromgrpc_access_phase. Sogrpcsapplies them there.@grpc_passalso gainsgrpc_ssl_name $upstream_host. nginx reads it only once verification is on, and without it the upstream certificate would be checked against the name of the upstream block,apisix_backend, so grpcs verification could never pass. It mirrorsproxy_ssl_name $upstream_hoston the proxy_pass side and is a no-op whileverifyis unset.X509_STOREis cached under theca_certsarray itself rather than under the config version. In standalone mode every resource loaded in one reload shares amodifiedIndex, so a version-keyed entry would hand one upstream's CA certificates to another.APISIX-Runtime 1.3.17
ca_certswithout a client certificate needs api7/apisix-nginx-module#126:ngx_http_apisix_set_upstream_ssl()used to apply the trusted store only inside itsctx->upstream_cert != NULLbranch, and nginx loadsproxy_ssl_trusted_certificateinto the SSL_CTX only whenproxy_ssl_verifyis on at configuration time, which APISIX never sets — so an upstream that asked for verification and supplied its own CA ended up with an empty trust store.That is the field's main use, so the runtime bump travels with this PR rather than separately; otherwise this could merge onto 1.3.16 and quietly do nothing. 1.3.17 also picks up ngx_http_ffi_client v0.1.2/v0.1.3.
Which issue(s) this PR fixes:
N/A
Checklist
Tests
t/node/upstream-tls2.tcoversverifyon its own overhttps, thenca_certswith no client certificate anywhere — an upstream on :8768 whose certificateproxy_ssl_trusted_certificatedoes not cover, rejected withverifyalone and accepted onceca_certsnames its CA. That pair fails on 1.3.16 and passes on 1.3.17. The file ends withgrpcsblocks for both fields; they fail without thegrpc_access_phasehalf of the change.t/node/upstream-mtls2.tcoversca_certsalongside a client certificate: a non-certificate entry rejected by the Admin API, a CA that does not match, one that does, and several CAs where only the last one anchors the chain.t/node/upstream-mtls.tTEST 21 and TEST 23 settls.verifyon an https upstream to check that it does not disturb the client certificate path. That field is no longer inert there, and the test upstream serves a self-signed certificate issued for another name, so they now ask forfalse— what they were written to assert is unchanged.