From 0323be29c9a09ec9c5832f51293dc64cce6bab74 Mon Sep 17 00:00:00 2001 From: Justin Gordon Date: Thu, 21 May 2026 10:10:08 -1000 Subject: [PATCH 1/2] Document cpflow workflow testing --- .../docs/testing-cpflow-github-actions.md | 132 ++++++++++++++++++ .controlplane/readme.md | 10 +- .controlplane/shakacode-team.md | 11 +- .github/cpflow-help.md | 4 +- 4 files changed, 149 insertions(+), 8 deletions(-) create mode 100644 .controlplane/docs/testing-cpflow-github-actions.md diff --git a/.controlplane/docs/testing-cpflow-github-actions.md b/.controlplane/docs/testing-cpflow-github-actions.md new file mode 100644 index 00000000..cab69ec3 --- /dev/null +++ b/.controlplane/docs/testing-cpflow-github-actions.md @@ -0,0 +1,132 @@ +# Testing cpflow GitHub Actions Changes + +Use this guide when changing generated `cpflow-*` GitHub Actions, updating the +`cpflow` generator version, or debugging review-app automation. + +## What To Test + +Test the flow in three layers: + +1. Local generated-file checks catch YAML, metadata, and lint problems before a PR. +2. GitHub workflow checks prove GitHub can load the workflow and run CI. +3. A real review-app deploy proves the default-branch trusted actions, GitHub + secrets, Docker build, and Control Plane deploy all work together. + +The third layer matters because the review-app workflow intentionally checks out +trusted workflow sources from the repository default branch before passing +Control Plane secrets to local composite actions. A PR branch can contain fixed +`.github/actions/*` files, but the deploy job still loads those local actions +from `master` until the fix is merged there. + +## Local Checks + +After regenerating the flow, run these checks from the repository root. When +testing an unreleased upstream `control-plane-flow` checkout, invoke that +checkout's `bin/cpflow` directly: + +```sh +bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master +bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness +bin/conductor-exec ruby -e 'require "yaml"; Dir[".github/actions/**/action.yml", ".github/workflows/*.yml"].sort.each { |path| YAML.load_file(path, aliases: true); puts "parsed #{path}" }' +bin/conductor-exec ruby -e 'require "yaml"; bad=[]; Dir[".github/actions/**/action.yml"].sort.each { |path| doc=YAML.load_file(path, aliases: true); doc.fetch("inputs", {}).each { |name, spec| bad << "#{path}:#{name}" if spec["description"].to_s.include?("${{") } }; }; abort bad.join("\n") unless bad.empty?; puts "no action metadata descriptions contain GitHub expressions"' +actionlint -ignore 'SC2129' .github/workflows/cpflow-*.yml +``` + +Why the explicit description check exists: GitHub parses expression-like snippets +inside composite action metadata, including `description:` fields. Literal +examples such as `${{ vars.SOME_VALUE }}` can fail action loading before any +shell step starts. + +## PR Checks + +Open a normal PR for the generated-file diff and wait for CI. The workflow PR +itself is useful for syntax and CI validation, but it does not fully prove +review-app deployment changes that live under `.github/actions/`. + +For top-level workflow edits, you can manually dispatch the PR branch workflow: + +```sh +gh workflow run cpflow-deploy-review-app.yml --ref -f pr_number= +``` + +This loads the workflow file from ``, but the deploy workflow's +`Checkout trusted workflow sources` step still checks out `master` before using +local composite actions with secrets. Treat this as a partial smoke test, not as +proof that PR-branch composite action changes work. + +## Post-Merge Review-App Test + +After the workflow PR merges to `master`, test a real review-app deployment: + +1. Pick a same-repository PR to use as the canary. +2. If the review app does not exist yet, comment exactly `+review-app-deploy` on + that PR. +3. If a previous deploy run failed, rerun the failed deploy run after the + workflow PR is merged. +4. Confirm the deploy run checks out `master` at the merge commit in + `Checkout trusted workflow sources`. +5. Confirm `Setup environment` succeeds and prints the expected `cpflow` version. +6. Confirm `Check if review app exists`, `Build Docker image`, and + `Deploy to Control Plane` all run as expected. +7. Open the review-app URL from the PR comment or deployment status and verify + it returns HTTP 200. + +Use the generated app name from the workflow log: + +```text +APP_NAME: ${REVIEW_APP_PREFIX}-${PR_NUMBER} +``` + +For this repo, verify the actual `REVIEW_APP_PREFIX` repository variable before +assuming the final app name. + +## Troubleshooting Signals + +### Composite action metadata fails before setup + +Error shape: + +```text +Unrecognized named-value: 'vars' +Failed to load ./.github/actions/cpflow-setup-environment/action.yml +``` + +Cause: GitHub parsed a literal expression inside composite action metadata, +usually an input description. Because trusted local actions come from `master`, +fix and merge the generated action metadata on `master`, then rerun the deploy. + +### Setup succeeds, then `cpflow exists` reports token format + +Error shape: + +```text +ERROR: Unknown API token format. Please re-run 'cpln profile login' or set the correct CPLN_TOKEN env variable. +``` + +Cause: the workflow can read `CPLN_TOKEN_STAGING`, but the value is not a valid +Control Plane service-account token for the installed Control Plane CLI. Rotate +the GitHub secret, then rerun the failed deploy job. + +### PR pushes do not create a new review app + +This is expected. Pushes redeploy only after the review app already exists. +Create the first review app by commenting exactly: + +```text ++review-app-deploy +``` + +## Ways To Make This Easier + +- Add a no-secret GitHub Actions smoke workflow that loads generated local + composite actions from the PR branch and fails fast on action metadata parsing. +- Add a small repo script, for example `bin/test-cpflow-github-flow`, that wraps + the local YAML, metadata-description, readiness, and `actionlint` checks. +- Add an early token sanity step after `Setup environment` so invalid + `CPLN_TOKEN_STAGING` and `CPLN_TOKEN_PRODUCTION` values fail with a named + "validate Control Plane token" step instead of surfacing later during + `cpflow exists`. +- Keep a tiny canary PR open for review-app workflow testing so post-merge + deploy verification does not depend on whichever feature PR happens to exist. +- Upstream the metadata-description check to `cpflow github-flow-readiness` so + downstream repos get the guard automatically. diff --git a/.controlplane/readme.md b/.controlplane/readme.md index 8c8c8db4..729b44e6 100644 --- a/.controlplane/readme.md +++ b/.controlplane/readme.md @@ -424,10 +424,10 @@ repository variables, secrets, and docs aligned with `.controlplane/controlplane For this app, validate a regenerated flow with: ```bash -bundle exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master -bundle exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness +bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master +bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness actionlint .github/workflows/cpflow-*.yml -bundle exec rubocop +bin/conductor-exec bundle exec rubocop ``` Then open a normal PR and let GitHub Actions prove the generated review-app, @@ -445,3 +445,7 @@ After the workflow reports a review-app URL, verify the URL returns HTTP 200. If a project needs to track generator changes automatically, use a scheduled maintenance PR or Renovate-style workflow that bumps the `cpflow` version, regenerates these files, and runs the same validation commands. + +For a fuller checklist, including the gotcha that review-app deploys load local +composite actions from `master` before using Control Plane secrets, see +[Testing cpflow GitHub Actions Changes](docs/testing-cpflow-github-actions.md). diff --git a/.controlplane/shakacode-team.md b/.controlplane/shakacode-team.md index 45ac0269..ca03e4a2 100644 --- a/.controlplane/shakacode-team.md +++ b/.controlplane/shakacode-team.md @@ -66,9 +66,14 @@ flow, regenerate the `cpflow-*` actions/workflows in this repo from the target `cpflow` version or branch using `--staging-branch master`, review the diff, and keep the repository variables above aligned with `.controlplane/controlplane.yml`. Validate with `cpflow github-flow-readiness`, `actionlint .github/workflows/cpflow-*.yml`, and -the normal CI checks before merging. - -See [readme.md](readme.md) for more details. +the normal CI checks before merging. For review-app workflow changes, remember +that the deploy workflow checks out trusted local actions from `master` before +passing Control Plane secrets; PR-branch composite action changes are not fully +tested until they land on `master` and a real review-app deploy is rerun. + +See [readme.md](readme.md) and +[Testing cpflow GitHub Actions Changes](docs/testing-cpflow-github-actions.md) +for more details. ## Links diff --git a/.github/cpflow-help.md b/.github/cpflow-help.md index dd82ba08..ba51f5ec 100644 --- a/.github/cpflow-help.md +++ b/.github/cpflow-help.md @@ -60,7 +60,7 @@ You asked for review app help. These commands are generated by [cpflow](https://
Advanced: testing changes to generated workflows -When iterating on the generated workflow YAML on a PR branch, comment-triggered runs (`+review-app-deploy`, `+review-app-delete`, `+review-app-help`) execute the workflow code from the repository's default branch — not your PR branch. To exercise the PR-branch workflow code before merging, dispatch the workflow manually with `gh`: +When iterating on the generated workflow YAML on a PR branch, comment-triggered runs (`+review-app-deploy`, `+review-app-delete`, `+review-app-help`) execute the workflow code from the repository's default branch — not your PR branch. To exercise the top-level PR-branch workflow file before merging, dispatch the workflow manually with `gh`: ```sh gh workflow run cpflow-deploy-review-app.yml --ref -f pr_number= @@ -68,6 +68,6 @@ gh workflow run cpflow-delete-review-app.yml --ref -f pr_number gh workflow run cpflow-help-command.yml --ref -f pr_number= ``` -`workflow_dispatch` runs use the workflow file from the `--ref` you pass, so this is the supported way to test PR-branch workflow edits before merge. After merge, comment triggers go back to running the default-branch workflow code as usual. +`workflow_dispatch` runs use the workflow file from the `--ref` you pass, but workflows that intentionally check out trusted local actions from the default branch will still load those local composite actions from the default branch before using secrets. Treat this as a partial smoke test for top-level workflow edits. For changes under `.github/actions/`, merge the generated fix to the default branch and rerun a real review-app deploy.
From e4302c5164ced1e9c01fb918f30224b357c59030 Mon Sep 17 00:00:00 2001 From: Justin Gordon Date: Thu, 21 May 2026 11:35:49 -1000 Subject: [PATCH 2/2] Address cpflow testing doc reviews --- .../docs/testing-cpflow-github-actions.md | 32 ++++++++------ .controlplane/readme.md | 9 +++- bin/test-cpflow-github-flow | 42 +++++++++++++++++++ 3 files changed, 69 insertions(+), 14 deletions(-) create mode 100755 bin/test-cpflow-github-flow diff --git a/.controlplane/docs/testing-cpflow-github-actions.md b/.controlplane/docs/testing-cpflow-github-actions.md index cab69ec3..934ed176 100644 --- a/.controlplane/docs/testing-cpflow-github-actions.md +++ b/.controlplane/docs/testing-cpflow-github-actions.md @@ -20,22 +20,28 @@ from `master` until the fix is merged there. ## Local Checks -After regenerating the flow, run these checks from the repository root. When -testing an unreleased upstream `control-plane-flow` checkout, invoke that -checkout's `bin/cpflow` directly: +After regenerating the flow, run these checks from the repository root. If +`cpflow` is installed as a gem, use `cpflow` directly: + +```sh +bin/conductor-exec cpflow generate-github-actions --staging-branch master +bin/test-cpflow-github-flow +``` + +When testing an unreleased upstream `control-plane-flow` checkout, replace +`cpflow` with that checkout's `bin/cpflow`: ```sh bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master -bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness -bin/conductor-exec ruby -e 'require "yaml"; Dir[".github/actions/**/action.yml", ".github/workflows/*.yml"].sort.each { |path| YAML.load_file(path, aliases: true); puts "parsed #{path}" }' -bin/conductor-exec ruby -e 'require "yaml"; bad=[]; Dir[".github/actions/**/action.yml"].sort.each { |path| doc=YAML.load_file(path, aliases: true); doc.fetch("inputs", {}).each { |name, spec| bad << "#{path}:#{name}" if spec["description"].to_s.include?("${{") } }; }; abort bad.join("\n") unless bad.empty?; puts "no action metadata descriptions contain GitHub expressions"' -actionlint -ignore 'SC2129' .github/workflows/cpflow-*.yml +bin/test-cpflow-github-flow ruby /path/to/control-plane-flow/bin/cpflow ``` Why the explicit description check exists: GitHub parses expression-like snippets inside composite action metadata, including `description:` fields. Literal examples such as `${{ vars.SOME_VALUE }}` can fail action loading before any -shell step starts. +shell step starts. The wrapper runs `cpflow github-flow-readiness`, parses the +generated YAML, checks action input descriptions for literal GitHub expressions, +and runs `actionlint -ignore 'SC2129' .github/workflows/cpflow-*.yml`. ## PR Checks @@ -77,8 +83,10 @@ Use the generated app name from the workflow log: APP_NAME: ${REVIEW_APP_PREFIX}-${PR_NUMBER} ``` -For this repo, verify the actual `REVIEW_APP_PREFIX` repository variable before -assuming the final app name. +This is a template from the workflow output, not a literal command to evaluate +unless those environment variables are already set. For this repo, verify the +actual `REVIEW_APP_PREFIX` repository variable before assuming the final app +name. ## Troubleshooting Signals @@ -120,8 +128,8 @@ Create the first review app by commenting exactly: - Add a no-secret GitHub Actions smoke workflow that loads generated local composite actions from the PR branch and fails fast on action metadata parsing. -- Add a small repo script, for example `bin/test-cpflow-github-flow`, that wraps - the local YAML, metadata-description, readiness, and `actionlint` checks. +- Extend `bin/test-cpflow-github-flow` as more local cpflow GitHub Actions + checks become worth standardizing. - Add an early token sanity step after `Setup environment` so invalid `CPLN_TOKEN_STAGING` and `CPLN_TOKEN_PRODUCTION` values fail with a named "validate Control Plane token" step instead of surfacing later during diff --git a/.controlplane/readme.md b/.controlplane/readme.md index 729b44e6..9e946224 100644 --- a/.controlplane/readme.md +++ b/.controlplane/readme.md @@ -426,7 +426,7 @@ For this app, validate a regenerated flow with: ```bash bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness -actionlint .github/workflows/cpflow-*.yml +actionlint -ignore 'SC2129' .github/workflows/cpflow-*.yml bin/conductor-exec bundle exec rubocop ``` @@ -435,12 +435,17 @@ staging, lint, JS, and RSpec workflows before merging. For review-app workflow changes, test both the local workflow syntax and a real deployment. GitHub runs `issue_comment` workflows from the default branch, so a `+review-app-deploy` comment on the PR does not fully exercise command changes that are only on the -PR branch. Before merge, run the PR branch workflow explicitly: +PR branch. For top-level workflow edits, run the PR branch workflow explicitly: ```bash gh workflow run cpflow-deploy-review-app.yml --ref -f pr_number= ``` +This loads the workflow file from ``, but trusted local composite +actions still come from the default branch before secrets are used. Treat it as +a partial smoke test, then verify a real deploy after the workflow changes land +on `master`. + After the workflow reports a review-app URL, verify the URL returns HTTP 200. If a project needs to track generator changes automatically, use a scheduled maintenance PR or Renovate-style workflow that bumps the `cpflow` version, diff --git a/bin/test-cpflow-github-flow b/bin/test-cpflow-github-flow new file mode 100755 index 00000000..593211bf --- /dev/null +++ b/bin/test-cpflow-github-flow @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$ROOT" + +cpflow_cmd=(cpflow) +if [[ $# -gt 0 ]]; then + cpflow_cmd=("$@") +fi + +echo "==> cpflow github-flow-readiness" +bin/conductor-exec "${cpflow_cmd[@]}" github-flow-readiness + +echo "==> parse generated GitHub Actions YAML" +bin/conductor-exec ruby <<'RUBY' +require "yaml" + +Dir[".github/actions/**/action.yml", ".github/workflows/*.yml"].sort.each do |path| + YAML.load_file(path, aliases: true) + puts "parsed #{path}" +end +RUBY + +echo "==> check composite action input descriptions" +bin/conductor-exec ruby <<'RUBY' +require "yaml" + +bad = [] +Dir[".github/actions/**/action.yml"].sort.each do |path| + doc = YAML.load_file(path, aliases: true) + doc.fetch("inputs", {}).each do |name, spec| + bad << "#{path}:#{name}" if spec["description"].to_s.include?("${{") + end +end + +abort bad.join("\n") unless bad.empty? +puts "no action metadata descriptions contain GitHub expressions" +RUBY + +echo "==> actionlint" +actionlint -ignore "SC2129" .github/workflows/cpflow-*.yml