Skip to content

fix(auth): verify public/private key match with retries in mTLS keystore loading - #14829

Open
macastelaz wants to merge 3 commits into
googleapis:mainfrom
macastelaz:fix/mtls-keystore-key-match-retries
Open

macastelaz wants to merge 3 commits into
googleapis:mainfrom
macastelaz:fix/mtls-keystore-key-match-retries

Conversation

@macastelaz

Copy link
Copy Markdown
Contributor

Summary

Verifies that the parsed PrivateKey cryptographically matches the leaf X509Certificate's PublicKey in MtlsKeyStoreUtils.createMtlsKeyStore(), and adds retry handling (3 attempts with a 100ms backoff, matching AgentIdentityUtils.loadAndVerifyCredentials()) in X509Provider.getKeyStore() when loading separate cert_path and key_path files from disk.

Problem

  1. Unverified KeyStore entries: KeyStore.getInstance("JKS").setKeyEntry("alias", key, new char[] {}, chain) does not verify that key corresponds to chain[0].getPublicKey() as long as the key algorithm matches (e.g., both are RSA or both are EC).
  2. Non-atomic rotation across separate cert_path and key_path files: When an external certificate rotator updates cert_path (cert_A -> cert_B) slightly before key_path (key_A -> key_B), transport channel refresh (CertificateRotationTracker, which tracks cert_path's SHA-256 fingerprint) can trigger X509Provider.getKeyStore() during the intermediate (cert_B, key_A) window. Without key-pair verification and retries, getKeyStore() succeeds with a mismatched KeyStore and the channel marks FP(cert_B) as refreshed, leaving the mTLS channel with a mismatched certificate and private key until the next rotation.

Solution

  • MtlsKeyStoreUtils.createMtlsKeyStore(): Perform a test signature and verification (SHA256withRSA for RSA keys, SHA256withECDSA for EC/ECDSA keys with SunEC "ECDSA" -> "EC" public key normalization) to ensure the parsed private key matches the leaf certificate's public key, throwing GeneralSecurityException("Certificate and private key do not match") on mismatch.
  • X509Provider.getKeyStore(): Retry up to 3 times (100ms interval) when reading and verifying separate cert_path and key_path files from WorkloadCertificateConfiguration, handling transient mid-rotation key mismatches or partial file writes while preserving interrupt restoration on InterruptedException.

Testing

  • Added unit tests in MtlsKeyStoreUtilsTest covering mismatched RSA and EC key pairs.
  • Added unit tests in X509ProviderTest covering:
    • Retrying and succeeding when key_path finishes rotating to match cert_path on a subsequent attempt.
    • Failing with IOException caused by GeneralSecurityException("Certificate and private key do not match") after exhausting all 3 attempts.
    • Retrying and succeeding when cert_path is transiently empty/truncated on the first attempt.
    • Restoring the thread interrupt flag and failing fast with IOException if interrupted during retry backoff.

…ore loading

Verify that the parsed private key matches the leaf certificate's public key in MtlsKeyStoreUtils.createMtlsKeyStore() via a test signature and verification, and retry up to 3 times (with a 100ms interval) in X509Provider.getKeyStore() when loading separate cert_path and key_path files from disk to handle transient mid-rotation mismatches.
@macastelaz
macastelaz requested review from a team as code owners October 29, 2026 02:23

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces key pair verification when creating an mTLS KeyStore to ensure the private key matches the leaf certificate's public key. It also adds retry logic in X509Provider to handle transient mismatches during atomic key rotations. The review feedback recommends separating cryptographic setup/execution failures from actual key verification mismatches to prevent misleading error messages, and advises against adding out-of-scope algorithm normalization (such as converting 'ECDSA' to 'EC') to maintain behavioral alignment.

@macastelaz
macastelaz requested a review from nbayati October 30, 2026 03:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant