Conversation
… tree + scrub) Tracked-files copy of eks-mr-demo main @ cc13708 (131 files). Excluded at the fork: GitLab CI machinery, .kiro/ agent skills, AGENTS.md/CLAUDE.md, and all untracked working trees. Mechanical scrub applied: repo URL, deploy-variable defaults (account/role/Bindle -> REPLACE-ME placeholders), internal host and policy citations in comments, runbook prerequisites 2/5/8, threat-model account. Not yet done (phase 2): replace the CloudFront/CFS front door with an observer bastion + SSM tunnel, remove the unused Cognito constructs, fix the write-pool defect in src/app/common.py, port the deploy rail to Makefile + GitHub Actions, decide the ECR mirror path. The three CFS front-door files still carry internal identifiers and are deleted in phase 2 -- this commit is NOT publishable as-is.
…ling write pool Replace the internal-only front door and remove the demo's planted defect so the sample shows what good looks like for EKS + ARC Region Switch + FIS. Front door -> observer bastion + SSM tunnel - New ObserverStack (us-east-1): private VPC, requester peerings to both workload VPCs, three SSM interface endpoints, t4g.nano AL2023-arm64 bastion with no public IP and a zero-ingress security group. AMI resolved via the SSM public parameter at deploy time. - FrontDoorStack -> OperatorAccessStack: keeps the internal ALB, the Argo CD IP target group and the /cockpit* rule; drops CloudFront, the VPC origin, the S3 403 bucket and the signed-cookie gate. ALB ingress is exactly the observer CIDR (10.2.0.0/16) on tcp/80, listener open:false. The unrouted IGW that existed only for CloudFront VPC origins is removed. - Accepter side: per-workload-region post-deploy step accepts the peering and adds the observer-CIDR return route (idempotent CLI), threaded off the dotenv rail like PeeringStack -- never Fn::ImportValue across regions. - build/tunnel.sh: SSM port-forward from the bastion to a region's internal ALB; prints the Argo and /cockpit URLs. - Removed: front-door-stack.ts, render-403.py, 403.html, front-door.test.ts, the GitLab CI workflow generator call (CI is owned by the monorepo root). - test/observer.test.ts (10 tests) pins the new contract: no AWS::CloudFront::* anywhere, single observer-CIDR ingress, no public IP / zero ingress on the bastion, three SSM endpoints, non-overlapping CIDRs, tunnel.sh bash -n. Proven to FAIL against the pre-change stack before landing. Write pool -> validate-after-idle, discard-on-error, max-lifetime - src/app/common.py: a connection whose write raised is closed and dropped, never returned; a pooled connection idle >5s is pinged (SELECT 1) at checkout; anything >5min old is retired. A writer failover now costs one failed write per pooled connection and recovers within a few requests -- no restart, no operator. /health and /ready stay database-free by design (gray-failure fidelity; existing test pins it). - test/fixtures/pool_probe.py simulates the failover against a fake driver. Against the OLD pool: 10/10 writes fail, 0 new connections, never recovers. Against the new one: 1 fails, 9 succeed, 1 replacement. Two Jest tests assert it. - Docs updated: README, runbook, az-segment-script, restore-steady-state.sh header, fluent-bit header no longer describe a defect that is not there. Contract-test fixes surfaced while gating: SSM-resolved parameters recognised by type (not name) in the template-vs-deploy test; accepter env vars renamed so they cannot parse as <STACK>_<OUTPUT>; GitLab-CI YAML tests removed (their contracts move to the root GitHub workflow); no-bitwise CIDR helper. Gate: npx projen build exit 0 -- 369 tests, 12 templates cfn-lint clean, zero CloudFront resources, zero internal identifiers or concept prose in the tree.
…nternal-reference scrub CI (monorepo root .projenrc.ts) - New Pattern fields: `selfManaged` (pattern owns its own .projenrc.ts; root emits only workflows + dependabot) and `runsOn` (arm64 runners: the images are Graviton-only and kaniko cannot cross-build). e2e gets a concurrency group with cancel-in-progress:false because the rail's stack names are fixed. - eks-multi-region entry: build job installs cfn-lint==1.45.0 and runs `yarn ci:build:cdk` with CFN_LINT_REQUIRED=1; e2e assumes github-actions-eks-multi-region via OIDC (8h session, matching the other multi-region samples), runs make buckets/mirror/deploy/verify, captures failure diagnostics (failed stacks in all three regions + installer CodeBuild logs) and cleans up on success. - Two-tier IAM, the shape `cdk deploy` gives siblings via the cdk-* bootstrap roles: a narrow GitHub role (only what the workflow's shell steps call + PassRole on one role) and a CloudFormation execution role passed as ROLE_ARN on every change-set. Trust and permission policies in eks-multi-region/docs/iam/ with create commands. - eks-multi-region/Makefile (thin front on the projen rail), cleanup.sh (reverse- dependency teardown across three regions + mirror ECR repos + assets buckets), build/verify-stacks.sh. Four tests derive the stack/region set from tasks.json and assert both scripts match it exactly; proven to fail when a stack is dropped. Public docs - README rewritten for a public deployer: SSM-tunnelled observer bastion, make-driven deploy, correct stack table, prerequisites, MIT-0. - runbook §0-§2 and §7 rewritten around make + the bastion; stale numbers fixed (3->5 replicas not 2->4, targetPercent 150 not 200, EKS step timeout 8 measured not 20 headroom, the round-trip compounding rule); the old "open AllowedCidr" advice replaced (it gates the EKS endpoint, not the UI). - docs/lessons.md: the 26 failure modes this stack has hit, scrubbed; ~40 code comments now cite `docs/lessons.md #N`. - Root README samples table gains the eks-multi-region row. Removed: the unused Cognito auth constructs (operator access needs no IdP), the scaffold-template tsconfig-fragments.md, a stray Apache LICENSE (root MIT-0 applies). Comment scrub across 46 files: internal policy ids, CI-provider name, predecessor project citations, internal file pointers. Repo-wide grep for every internal identifier and concept: zero hits. Gate: npx projen build exit 0 -- 373 tests, 12 templates cfn-lint clean.
…a bucket prefix e2e iteration 1 (run 34911294219) failed at `make mirror`: "crane not found on PATH". build/mirror-images.sh, build-docker.sh and deploy-docker.sh all shell out to crane for daemonless registry copies; the internal CI installed it in a before_script that was not carried into this workflow. Install the pinned v0.20.2 arm64 binary before the mirror step. Everything upstream of the mirror (OIDC assume-role, pre-flight cleanup, bucket creation) succeeded, so no stacks were created. The cleanup workflow now takes a workflow_dispatch input `assets_bucket_prefix` (self-managed patterns only): e2e cleanup runs only on success, so a failed run's sha-suffixed buckets outlive it and the next run's pre-flight cleanup uses a new prefix. Siblings' cleanup workflows are unchanged.
…cate to ECR Public e2e iteration 2 (run 34912475843): argocd mirrored to both regions with digest verified -- proving the GitHub role's ECR create/push permissions -- then the redis pull from public.ecr.aws failed with TOOMANYREQUESTS. Anonymous pulls are rate-limited per source IP and a CI runner shares its egress with many jobs; five of the eight mirrored images come from public.ecr.aws. - copy_with_retry: bounded exponential backoff (5 attempts, 10s..160s) on throttling signatures ONLY; any other error fails on the first attempt so a wrong digest or denied push is never retried into a pass. Proven with a fake crane: 429,429,ok -> success on attempt 3; DENIED -> immediate failure. - login_ecr_public: one authenticated session to public.ecr.aws (much higher limit). Best-effort: needs ecr-public:GetAuthorizationToken + sts:GetServiceBearerToken, added to docs/iam/github-actions-role-policy.json; without them the script warns and pulls anonymously. - The digest-pin test now sees through the wrapper and asserts no bare crane copy remains.
…least-privilege deployer e2e iteration 3 (run 34912978025): mirror passed for all 8 images in both regions and the CI build passed; the rail then died at deploy phase 0 with AccessDenied on iam:CreateServiceLinkedRole -- while AWSServiceRoleForFIS has existed in the account since April. The step tolerated only 'has been taken', and a deployer without the create grant is denied before the service can say so. - Check first with iam:GetRole; create only when absent. - AccessDenied on either call is a WARNING, not a failure: nothing in the deploy rail depends on this account-scoped role, only the first cockpit-fired FIS experiment does, and that path reports its own error. Any other error aborts. - docs/iam/github-actions-role-policy.json grants iam:GetRole + iam:CreateServiceLinkedRole scoped to the FIS SLR ARN so the step is quiet. - New test executes the GENERATED exec from .projen/tasks.json against a fake aws for all five paths (exists/fresh/taken/denied -> 0, other -> 1) and asserts the no-inner-single-quote invariant (bug class 1).
…ainers + assets) e2e iteration 4 (run 34913825457): the FIS SLR step now warns and continues as designed; the rail then failed at deploy:s3 -- 'cannot find dist/content.zip'. The Makefile's build target ran only ci:build:cdk. The original pipeline had four build jobs (build:cdk -> build:container-plan -> build:docker -> build:package) threaded via artifacts; the e2e path skipped the last three, so the app images were never built and the zip yarn deploy unpacks never existed. - make build now runs all four in order: DOCKER_IMAGE_REGIONS derived from REGIONS, REQUIRE_CONTAINER_BUILD defaults to true (a silently skipped image build would otherwise surface much later as an ECR push with nothing to push). build:docker uses docker-wrapped kaniko on the arm64 GitHub runner. - Test pins the chain, its order, the fail-loud default, and that ci:build:package -> package:content writes dist/content.zip. - Proven locally: the chain produces a 31-file content.zip including containers/manifest.json.
…hain test (eslint no-shadow)
e2e iteration 5 (run 34914682920): the rail deployed both region stacks, peering, globaldata and secondarydb, then the observer stack died at create-change-set in us-east-1 with 'S3 error: The specified bucket does not exist'. CloudFormation reads a stack's template from the bucket in the STACK's region; the observer is a singleton in a third region, and both 'make buckets' and deploy:s3 covered only the two workload regions. This is the 7th template gap (fixed 2026-08-26 for regions[1]) recurring for a region that regions[] never named. - deploy:s3 now derives its sync set from regions[] UNION every singleton / postDeployStack region, so a future fourth-region stack cannot repeat this. - make buckets creates BUCKET_REGIONS = REGIONS + OBSERVER_REGION; cleanup.sh deletes the observer bucket as well; runbook prerequisite says three. - The sync test now derives the expected set from the deploy rail's own AWS_REGION= assignments and pins Makefile, cleanup.sh and tasks.json to it. Proven failing without the fix (us-east-1 synced: false).
…g, and never leaves an EKS cluster behind
e2e iteration 6 (run 34918680587, pre-flight cleanup): CloudFormation's first
delete of the EKS cluster got a transient 409 ('cluster currently has an update
in progress'). cleanup.sh went straight to --retain-resources, which left an
ACTIVE cluster outside any stack: its ENIs pinned the subnets (stack stuck in
DELETE_IN_PROGRESS) and the next deploy would collide on the cluster name.
- wait_deleted: on DELETE_FAILED, sleep 90s and re-issue a PLAIN delete first;
only a second failure falls back to --retain-resources.
- sweep_eks: after the region-stack wave, delete any cluster the rail would
name (nodegroups first), whether or not a stack still owns it; warns loudly
if the deployer lacks the grant.
- docs/iam/github-actions-role-policy.json gains eks:DeleteCluster and
eks:DeleteNodegroup (the cleanup runs under the GitHub role).
…on-stack wave Recovery from iteration 6: the stack eks-mr-demo-region-us-east-2 sits in DELETE_FAILED on two subnets and a security group pinned by the ENIs of the cluster CloudFormation retained (DELETE_SKIPPED). Deleting the stack again without first removing the cluster fails the same way. cleanup.sh now checks whether the region stack still OWNS an EKS::Cluster resource; if not, it deletes the cluster before issuing the stack delete, and keeps the post-wave sweep for the retain fallback. The ownership check counts client-side with wc -w: --query is applied per page (bug class 25) and the live stack paginated into two pages, so length(@) returned '0\n0' and an exact compare silently skipped the sweep. Proven against the live account for the orphaned, absent and owned cases. Test pins: no length( inside any --query in cleanup.sh, wc -w on the EKS::Cluster query, and sweep_eks placed both before and after the region wave.
…rive the set in a test
Run 7 deployed six stacks in 50 minutes, then the post-globaldata step died on
AccessDenied rds:DescribeGlobalClusters -- the third e2e failure of the same class
(runs 3 and 6: iam:CreateServiceLinkedRole, eks:DeleteCluster). Nothing tied the aws
calls the GENERATED deploy task and its scripts make to the policy document operators
apply, so each gap cost a full deploy.
Policy: add rds:DescribeGlobalClusters (scoped to global-cluster ARNs),
arc-region-switch:UpdatePlan (scoped to the account's plans; the verifier's verbatim
no-op update that forces a fresh evaluation), logs:GetLogEvents (CodeBuild log groups;
installer-failure diagnostics), ec2:DescribeNetworkInterfaces/DescribeAvailabilityZones,
elasticloadbalancing:DescribeLoadBalancers, arc-region-switch:ListRoute53HealthChecks/
GetPlanEvaluationStatus, arc-zonal-shift:GetManagedResource,
cloudformation:ListStackResources.
Test: scan .projen/tasks.json, Makefile, cleanup.sh, the build/ scripts and both
workflows for 'aws <svc> <op>' (shell) and aws("svc","op") (python), map each to its
IAM action (waiters, login helpers and s3 high-level verbs via an explicit table) and
assert the policy grants it -- one named test per call. Proven against the old policy:
12 failures, including all four gaps that had reached a live run.
Run 8 deployed all 12 stacks and both installers, then failed in the phase-7.0 verifier: ARC validates iam:PassRole on the plan's execution role for EVERY UpdatePlan, including the verbatim no-op that forces a fresh evaluation. Scoped to the one role and conditioned on iam:PassedToService = arc-region-switch.amazonaws.com, the same shape as the cfn-exec PassRole. The derived IAM-contract test scans CLI calls, which cannot see an action the API implies rather than the command names; a new test pins the implication, reading the role name and its trust principal from the synthesized failover template. Proven failing without the grant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem / Motivation
The repository has samples for Aurora Global Database and S3 MRAP behind ARC Region Switch, but nothing for the most common compute target customers ask about: EKS. There is also no sample that shows single-AZ gray failures (a brownout, not an outage) from the client's side, or what an application connection pool must do to survive a writer failover without a restart.
Why it matters
Customers running EKS multi-region need a reference for how ARC's EKS scaling block coexists with a GitOps controller, how to inject and measure a gray failure, and how to keep an application writable through an Aurora Global writer move. Without one, each team rediscovers the same failure modes (
eks-multi-region/docs/lessons.mdrecords twenty-six of them).What changed
A new pattern,
eks-multi-region/: us-east-2 primary, us-west-2 standby, Aurora Serverless v2 in both as an Aurora Global Database, EKS with Karpenter in isolated subnets (no NAT), Argo CD managing the app, oneactivePassiveARC Region Switch plan (EKS scale-up → Aurora writer switch → Route 53 health-check flip), a resilience cockpit that arms and fires FIS network faults per AZ, and a Locust load generator emitting per-op / per-AZ / per-region client availability.Operator access has no public ingress at all. A third-region observer VPC holds a bastion with no public IP and no inbound rules, peered to both workload VPCs;
build/tunnel.shport-forwards through it over SSM Session Manager to each region's internal ALB (Argo CD UI, and the cockpit in the standby). IAM is the only identity in the path.The application's write pool is built to survive a writer failover: a connection whose write raises is closed and dropped, an idle pooled connection is pinged before use, nothing outlives a max lifetime. A failover costs one failed write per pooled connection and heals within a few requests.
flowchart LR subgraph Operator access L[operator laptop]:::ctx -->|SSM port-forward| B[observer bastion<br/>us-east-1, no public IP]:::added B -->|VPC peering| A1[internal ALB<br/>us-east-2]:::added B -->|VPC peering| A2[internal ALB<br/>us-west-2]:::added A1 --> G1[Argo CD]:::ctx A2 --> G2[Argo CD + cockpit]:::ctx end classDef added fill:#DCFCE7,stroke:#16A34A,color:#14532D,stroke-width:2px classDef ctx fill:#E0F2FE,stroke:#0284C7,color:#0C4A6E🟩 added · 🟦 context
The laptop reaches both regions' internal ALBs only through the bastion; nothing is reachable from the internet.
The pattern is self-managed in the root
.projenrc.ts(selfManaged: true): it keeps its own projen config (yarn berry, aws-cdk-lib 2.248 for the ARC L1s, a multi-phase deploy rail with an in-VPC CodeBuild installer), and the root emits only its three workflows and dependabot entry. Jobs run onubuntu-24.04-armbecause the images are Graviton-only and kaniko cannot cross-build. CI uses a two-tier IAM split (a narrow GitHub OIDC role plus a CloudFormation execution role passed on every change-set); both are documented with create commands ineks-multi-region/docs/iam/.Tests
373 Jest tests (
eks-multi-region/test/). Beyond template assertions, many pin cross-file contracts that synth cannot check: every template parameter is threaded by a deploy step (and vice versa);cleanup.shandbuild/verify-stacks.shlist exactly the stacks the rail deploys; noAWS::CloudFront::*resource exists anywhere; the access ALB has a single observer-CIDR ingress; the bastion has no public IP and zero ingress; an ARC round trip fits under the pod ceiling derived from the real HPA / NodePool / node-group values;test/fixtures/pool_probe.pysimulates a writer failover against a fake driver and asserts the pool recovers (against the original pool it never does). Several were proven to fail before their fix landed.Manual verification
The
eks-multi-region: buildworkflow is green on this branch (373 tests, 12 templates cfn-lint clean, run 34911074343). The e2e (deploy the full rail, verify, tear down) has not yet run -- this draft PR triggers it for the first time; it is a ~90-minute deploy-and-teardown proof, not a failover exercise, because ARC's EKS block sizes the standby from a 24-hour replica sample and a same-run failover would prove nothing.no linked issue: new pattern, no tracking issue exists.