Add the option to install a large-cluster manifest and cordon control plane to dedicated machines - #1632
Open
Max Smythe (maxsmythe) wants to merge 7 commits into
Open
Conversation
The bundled postgres ships sized for small clusters (--cluster-size=size0, the default): max_connections=100 and modest CPU and memory requests. Larger clusters saturate that and cap ate-api-server well below the load they generate. --cluster-size=size10 merge-patches a tuned postgresql.conf into the postgres configmap, resizes the statefulset to fill a dedicated node, and pins the pgxpool size so the clients open the connections the server is provisioned for. The benchmark orchestrator selects it for the 10k tests.
… overlay Bound the install-time wait for the podcertificate-controller's ClusterTrustBundles at 300s and print where to look when it expires, instead of looping forever on a controller that is Ready but not producing bundles. On --cluster-size=size10 clusters install the podcert-size10 kustomize overlay, which raises the controller's client-go rate limits to --kube-api-qps=100 / --kube-api-burst=200 so signing keeps up with the request volume the size10 postgres profile enables.
Teardown often runs against a cluster where an earlier upgrade already removed a CRD that manifests/ate-install still references (SandboxConfig, for instance). kubectl delete then fails with "no matches for kind", which --ignore-not-found does not cover, and errexit aborted the rest of the sweep. kubectl_delete_tolerant swallows only that error class.
…o dedicated nodes Under the flag, every control plane workload (ate-api-server, ate-controller, atenet-router, atenet-egress, podcertificate-controller, postgres) gets a nodeSelector and toleration for ate.dev/workloadType=ate-control-plane and a hostname anti-affinity against the other control plane pods, so each runs alone on a node of a dedicated pool. Without the flag the manifests are applied unchanged. The pinning lives in one kustomize component with a name-regex target. The install applies these workloads through several different streams (the base bundle, a lone ate-api-server redeploy, the podcert overlay, the postgres file, and the egress variants), so render_manifests wraps whichever path is in use in a throwaway kustomization that includes the component; kustomize leaves a stream alone when nothing in it matches.
Max Smythe (maxsmythe)
requested a review
from Benjamin Elder (BenTheElder)
September 13, 2026 07:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change allows users to install Substrate on larger clusters. It adds a
--cluster-sizeflag to enable more t-shirt-style sizing in the future to accommodate clusters of different sizes.It also adds a --cordon-control-plane flag that allows taints/tolerances/antiaffinity/node labels to have each control plane element run on its own dedicated machine.
Fixes #<issue_number_goes_here>