Add a workflows-scripts action that returns the scripts of the running workflow commit

.github/actions/workflows-scripts/action.yml

@@ -0,0 +1,75 @@
+name: Locate the workflows scripts
+description: >-
+  Returns the paths of the scripts and configuration files that belong to the
+  running commit of swiftlang/github-workflows, and can copy the scripts into
+  the workspace. Reference it with `uses: $/.github/actions/workflows-scripts`
+  from a workflow in this repository, so the scripts come from the same commit
+  as the workflow even when a caller pins the workflow to a release.
+
+inputs:
+  copy_into_workspace:
+    description: >-
+      Whether to copy the scripts into the job's workspace. Set this when the
+      scripts must be visible somewhere other than on the runner's own file
+      system, such as inside a container or a virtual machine that is given only
+      the workspace. Run it after the checkout of the repository under test,
+      which would otherwise remove the copy.
+    required: false
+    default: "false"
+
+outputs:
+  scripts_directory:
+    description: "Absolute path of the scripts directory."
+    value: ${{ steps.locate.outputs.scripts_directory }}
+  scripts_directory_windows:
+    description: "The scripts_directory with backslash separators, for PowerShell."
+    value: ${{ steps.locate.outputs.scripts_directory_windows }}
+  scripts_directory_relative:
+    description: "The scripts directory relative to the workspace. Empty unless copy_into_workspace is true."
+    value: ${{ steps.locate.outputs.scripts_directory_relative }}
+  configs_directory:
+    description: "Absolute path of the directory of shared configuration files. It is never copied."
+    value: ${{ steps.locate.outputs.configs_directory }}
+
+runs:
+  using: composite
+  steps:
+    - name: Locate the scripts
+      id: locate
+      shell: bash
+      env:
+        ACTION_PATH: ${{ github.action_path }}
+        COPY_INTO_WORKSPACE: ${{ inputs.copy_into_workspace }}
+      run: |
+        set -euo pipefail
+
+        # The runner downloads the whole repository at the running commit and
+        # places this action inside it, so the workflows directory is two levels up.
+        workflows_directory="${ACTION_PATH//\\//}/../../workflows"
+
+        # Windows keeps the parent references, which its tools resolve themselves;
+        # elsewhere they are resolved so the paths are clean.
+        if [ "$RUNNER_OS" != "Windows" ]; then
+          workflows_directory="$(cd "$workflows_directory" && pwd -P)"
+        fi
+
+        if [ "$COPY_INTO_WORKSPACE" = "true" ]; then
+          relative=".github-workflows-scripts"
+          scripts="$GITHUB_WORKSPACE/$relative"
+          rm -rf "$scripts"
+          mkdir -p "$scripts"
+          cp -R "$workflows_directory/scripts/." "$scripts/"
+        else
+          relative=""
+          scripts="$workflows_directory/scripts"
+        fi
+        configs="$workflows_directory/configs"
+
+        {
+          echo "scripts_directory=$scripts"
+          # The Windows dispatch invokes the runner through PowerShell, which
+          # needs backslashes.
+          echo "scripts_directory_windows=${scripts//\//\\}"
+          echo "scripts_directory_relative=$relative"
+          echo "configs_directory=$configs"
+        } >> "$GITHUB_OUTPUT"

.github/workflows/architecture.md

@@ -14,9 +14,9 @@ Workflows and actions must be versioned using GitHub releases to guarantee stabi
 
 No reusable workflow offered should show skipped jobs. Skipped jobs add visual noise, look confusing to contributors and imply that something is configured incorrectly.
 
-### 3. Scripts must be cloned not curled
+### 3. Scripts must come from the workflow's own commit
 
-Some jobs rely on scripts to execute their functionality. This is often preferred over inline scripts in the workflow definition `yml` files since it gives a better developer experience to the maintainers of those scripts. However, one downside of this approach is that these scripts are not present when a reusable workflow is executed from another repository. It is important that a unified solution checks the scripts out explicitly rather than curling them, since curling frequently runs into rate limiting with GitHub's API.
+Some jobs rely on scripts to execute their functionality. This is often preferred over inline scripts in the workflow definition `yml` files since it gives a better developer experience to the maintainers of those scripts. However, one downside of this approach is that these scripts are not present when a reusable workflow is executed from another repository, and a script taken from the default branch does not match a workflow pinned to a release. It is important that a unified solution takes the scripts from the same commit as the workflow, and does so without curling them, since curling frequently runs into rate limiting with GitHub's API. "How workflows get their scripts" below describes how this is done.
 
 ### 4. Custom matrix builds
 
@@ -43,8 +43,27 @@ The design is intended to be layered, so that adopters may use the whole stack o
 * `toolchain_matrix.yml`: produces a matrix of toolchains with no command attached, for callers supplying their own
 * `execute_matrix.yml`: expands a work definition into one job per entry and dispatches each to its platform. FreeBSD runs in a VM step here rather than through a job runner
 * `generate-matrix.swift`: produces the canonical work definition YAML or JSON. It reads the workflow's inputs from the environment and fails the run rather than emitting a matrix that would silently drop a job the caller asked for
+* `.github/actions/workflows-scripts`: the composite action that makes the scripts and configuration files of the running commit available and returns their paths
 * `job-runner-linux.sh`, `job-runner-macos.sh`, `job-runner-windows.ps1`: helper scripts which execute the defined work. They handle the complexity of running inside/outside Docker and installing Swift if required.
 
+## How workflows get their scripts
+
+`$/` in a `uses:` reference means the repository that holds the running workflow file, at the commit that file was loaded from. In a workflow called from another repository it still means this repository, at the tag or branch the caller named. Every workflow here reaches its scripts with `uses: $/.github/actions/workflows-scripts`, so the scripts always match the workflow that runs them. See "Workflow syntax for GitHub Actions" in GitHub's documentation for the `$/` syntax.
+
+The action takes one input and returns four outputs:
+
+| Name | Meaning |
+| --- | --- |
+| `copy_into_workspace` (input) | Copy the scripts into `.github-workflows-scripts` in the workspace. Defaults to `false`. |
+| `scripts_directory` | Absolute path of the scripts. |
+| `scripts_directory_windows` | `scripts_directory` with backslashes, for PowerShell. |
+| `scripts_directory_relative` | The scripts directory relative to the workspace. Empty unless `copy_into_workspace` is `true`. |
+| `configs_directory` | Absolute path of the shared configuration files. It is never copied. |
+
+`execute_matrix.yml` sets `copy_into_workspace` because the Linux and Windows containers and the FreeBSD VM are given only the workspace; the job runners read the copy there. The other workflows use the scripts where the runner placed them. A job that sets `copy_into_workspace` must run the action after checking out the repository under test; otherwise the checkout removes the copy.
+
+To test a change to the workflows or scripts, call the workflow at the branch or fork that holds it, for example `rnro/github-workflows/.github/workflows/package_test.yml@my-branch`. The scripts come from that same ref, so no separate input names the repository or ref for them.
+
 ## Swift version labels
 
 A version list names toolchains with labels: a version such as `6.3`, `nightly-release`, `nightly-main`, or a position counted back from the current release, `current`, `current-1` and `current-2`. A job's name carries the label the caller wrote. Job names are required status checks, so the default lists use the relative labels: a new Swift release changes which versions they name and leaves the checks alone. A caller who writes versions out gets those names, and the two spellings can be mixed in one list.

.github/workflows/benchmarks.yml

@@ -74,15 +74,6 @@ on:
         type: boolean
         description: "Whether PRs can be tested together with linked PRs mentioned in the PR description."
         default: false
-
-      workflows_repository:
-        type: string
-        description: "Repository to take the matrix scripts from. Point this at a fork to test a change to the workflows before it lands; it carries no version, so Dependabot has only the `uses:` line to bump."
-        default: "swiftlang/github-workflows"
-      workflows_ref:
-        type: string
-        description: "Ref to take the scripts from. Empty uses workflows_repository's default branch, which is correct for a released version; set it when workflows_repository is a fork whose default branch does not carry the change."
-        default: ""
 jobs:
   generate-matrix:
     name: Generate benchmark matrix
@@ -94,37 +85,13 @@ jobs:
         uses: actions/checkout@v7
         with:
           persist-credentials: false
-      - name: Resolve the workflows source
-        id: workflows_source
-        shell: bash
-        env:
-          WORKFLOWS_REPOSITORY: ${{ inputs.workflows_repository }}
-          WORKFLOWS_REF: ${{ inputs.workflows_ref }}
-        run: |
-          set -euo pipefail
-
-          # Empty uses the default branch.
-          echo "ref=$WORKFLOWS_REF" >> $GITHUB_OUTPUT
-
-          if [ "$GITHUB_REPOSITORY" = "$WORKFLOWS_REPOSITORY" ]; then
-            echo "needs_checkout=false" >> $GITHUB_OUTPUT
-            echo "root_directory=$GITHUB_WORKSPACE" >> $GITHUB_OUTPUT
-          else
-            echo "needs_checkout=true" >> $GITHUB_OUTPUT
-            echo "root_directory=$GITHUB_WORKSPACE/github-workflows" >> $GITHUB_OUTPUT
-          fi
-      - name: Checkout the workflows repository
-        if: ${{ steps.workflows_source.outputs.needs_checkout == 'true' }}
-        uses: actions/checkout@v7
-        with:
-          repository: ${{ inputs.workflows_repository }}
-          ref: ${{ steps.workflows_source.outputs.ref }}
-          path: github-workflows
-          persist-credentials: false
+      - name: Locate the workflows scripts
+        id: workflows_scripts
+        uses: $/.github/actions/workflows-scripts
       - name: Generate matrix
         id: generate
         env:
-          WORKFLOWS_CHECKOUT: ${{ steps.workflows_source.outputs.root_directory }}
+          SCRIPTS_DIRECTORY: ${{ steps.workflows_scripts.outputs.scripts_directory }}
           LINUX_USER_ENV_VARS: ${{ inputs.linux_env_vars }}
           MACOS_USER_ENV_VARS: ${{ inputs.macos_env_vars }}
           BENCHMARK_PACKAGE_PATHS: ${{ inputs.benchmark_package_paths }}
@@ -185,20 +152,18 @@ jobs:
           MACOS_COMMAND="$threshold_command"
           export LINUX_COMMAND MACOS_COMMAND
 
-          matrix_yaml=$(swift "${WORKFLOWS_CHECKOUT}/.github/workflows/scripts/matrix/generate-matrix.swift")
+          matrix_yaml=$(swift "${SCRIPTS_DIRECTORY}/matrix/generate-matrix.swift")
           echo "matrix_yaml<<EOF" >> $GITHUB_OUTPUT
           echo "$matrix_yaml" >> $GITHUB_OUTPUT
           echo "EOF" >> $GITHUB_OUTPUT
 
   benchmarks:
     name: Benchmarks
     needs: generate-matrix
-    uses: ./.github/workflows/execute_matrix.yml
+    uses: $/.github/workflows/execute_matrix.yml
     with:
       name: ${{ inputs.name }}
       matrix_yaml_string: ${{ needs.generate-matrix.outputs.matrix_yaml }}
-      workflows_repository: ${{ inputs.workflows_repository }}
-      workflows_ref: ${{ inputs.workflows_ref }}
       job_timeout: ${{ inputs.job_timeout }}
       needs_token: ${{ inputs.needs_token }}
       enable_cross_pr_testing: ${{ inputs.enable_cross_pr_testing }}

.github/workflows/execute_matrix.yml

@@ -42,15 +42,6 @@ on:
         type: boolean
         description: "Whether PRs can be tested together with linked PRs mentioned in the PR description"
         default: false
-
-      workflows_repository:
-        type: string
-        description: "Repository to take the matrix scripts from. Point this at a fork to test a change to the workflows before it lands; it carries no version, so Dependabot has only the `uses:` line to bump."
-        default: "swiftlang/github-workflows"
-      workflows_ref:
-        type: string
-        description: "Ref to take the scripts from. Empty uses workflows_repository's default branch, which is correct for a released version; set it when workflows_repository is a fork whose default branch does not carry the change."
-        default: ""
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.name }}
   cancel-in-progress: true
@@ -165,44 +156,11 @@ jobs:
           persist-credentials: false
           submodules: true
 
-      - name: Resolve the workflows source
-        id: workflows_source
-        shell: bash
-        env:
-          WORKFLOWS_REPOSITORY: ${{ inputs.workflows_repository }}
-          WORKFLOWS_REF: ${{ inputs.workflows_ref }}
-        run: |
-          set -euo pipefail
-
-          # Empty uses the default branch.
-          echo "ref=$WORKFLOWS_REF" >> $GITHUB_OUTPUT
-
-          if [ "$GITHUB_REPOSITORY" = "$WORKFLOWS_REPOSITORY" ]; then
-            echo "needs_checkout=false" >> $GITHUB_OUTPUT
-            root="$GITHUB_WORKSPACE"
-            scripts_relative=".github/workflows/scripts"
-          else
-            echo "needs_checkout=true" >> $GITHUB_OUTPUT
-            root="$GITHUB_WORKSPACE/github-workflows"
-            scripts_relative="github-workflows/.github/workflows/scripts"
-          fi
-
-          echo "root_directory=$root" >> $GITHUB_OUTPUT
-          # The Windows dispatch invokes the runner through PowerShell, which
-          # needs backslashes.
-          echo "root_directory_windows=$(echo "$root" | sed 's|/|\\|g')" >> $GITHUB_OUTPUT
-          # The FreeBSD VM works in its own copy of the workspace, at a path the host
-          # does not know, so it rebuilds ${SCRIPTS_ROOT} from this relative form.
-          echo "scripts_root_relative=$scripts_relative" >> $GITHUB_OUTPUT
-
-      - name: Checkout the workflows repository
-        if: ${{ steps.workflows_source.outputs.needs_checkout == 'true' }}
-        uses: actions/checkout@v7
+      - name: Locate the workflows scripts
+        id: workflows_scripts
+        uses: $/.github/actions/workflows-scripts
         with:
-          repository: ${{ inputs.workflows_repository }}
-          ref: ${{ steps.workflows_source.outputs.ref }}
-          path: github-workflows
-          persist-credentials: false
+          copy_into_workspace: true
 
       # Linux and Windows install their toolchain inside the job runner, so they
       # check out linked PRs there afterwards, to compile the checkout script with
@@ -213,7 +171,7 @@ jobs:
         if: ${{ inputs.enable_cross_pr_testing && github.event_name == 'pull_request' && matrix.config.platform == 'macOS' }}
         shell: bash
         env:
-          SCRIPTS_ROOT: ${{ steps.workflows_source.outputs.root_directory }}/.github/workflows/scripts
+          SCRIPTS_ROOT: ${{ steps.workflows_scripts.outputs.scripts_directory }}
           PR_REPO: ${{ github.repository }}
           PR_NUMBER: ${{ github.event.number }}
         run: |
@@ -226,7 +184,7 @@ jobs:
         shell: bash
         env:
           GITHUB_TOKEN: ${{ inputs.needs_token && secrets.GITHUB_TOKEN || '' }}
-          SCRIPTS_ROOT: ${{ steps.workflows_source.outputs.root_directory }}/.github/workflows/scripts
+          SCRIPTS_ROOT: ${{ steps.workflows_scripts.outputs.scripts_directory }}
           CONTAINER_JSON: ${{ toJson(matrix.config.swift_build.container) }}
           MATRIX_SWIFT_VERSION: ${{ matrix.config.swift_build.swift_version }}
           MATRIX_TOOLCHAIN: ${{ matrix.config.swift_build.toolchain }}
@@ -265,7 +223,7 @@ jobs:
         timeout-minutes: ${{ matrix.config.timeout_minutes || inputs.job_timeout }}
         shell: bash
         env:
-          SCRIPTS_ROOT: ${{ steps.workflows_source.outputs.root_directory }}/.github/workflows/scripts
+          SCRIPTS_ROOT: ${{ steps.workflows_scripts.outputs.scripts_directory }}
           SDK_TRIPLES: ${{ join(matrix.config.swift_build.sdk.triples, ' --android-sdk-triple=') }}
         run: |
           # The SDK build exported the matched toolchain and SDK bundle to GITHUB_ENV for
@@ -280,7 +238,7 @@ jobs:
         timeout-minutes: ${{ matrix.config.timeout_minutes || inputs.job_timeout }}
         env:
           GITHUB_TOKEN: ${{ inputs.needs_token && secrets.GITHUB_TOKEN || '' }}
-          SCRIPTS_ROOT: ${{ steps.workflows_source.outputs.root_directory_windows }}\.github\workflows\scripts
+          SCRIPTS_ROOT: ${{ steps.workflows_scripts.outputs.scripts_directory_windows }}
           CONTAINER_IMAGE: ${{ matrix.config.swift_build.container.image }}
           CROSS_PR_TESTING: ${{ inputs.enable_cross_pr_testing && github.event_name == 'pull_request' }}
           CROSS_PR_REPO: ${{ github.repository }}
@@ -308,7 +266,7 @@ jobs:
         shell: bash
         env:
           GITHUB_TOKEN: ${{ inputs.needs_token && secrets.GITHUB_TOKEN || '' }}
-          SCRIPTS_ROOT: ${{ steps.workflows_source.outputs.root_directory }}/.github/workflows/scripts
+          SCRIPTS_ROOT: ${{ steps.workflows_scripts.outputs.scripts_directory }}
           XCODE_TARGETS_JSON: ${{ toJson(matrix.config.xcode_build.targets) }}
           XCODE_DEBUG_OUTPUT: ${{ matrix.config.xcode_build.debug_output }}
           SWIFTLY_TOOLCHAIN: ${{ matrix.config.xcode_build.swiftly_toolchain }}
@@ -399,7 +357,7 @@ jobs:
           MATRIX_COMMAND: ${{ matrix.config.command }}
           # Shell-quoted on the host, where jq is available.
           MATRIX_COMMAND_ARGUMENTS: ${{ steps.freebsd_arguments.outputs.command_arguments }}
-          SCRIPTS_ROOT_RELATIVE: ${{ steps.workflows_source.outputs.scripts_root_relative }}
+          SCRIPTS_ROOT_RELATIVE: ${{ steps.workflows_scripts.outputs.scripts_directory_relative }}
           GITHUB_TOKEN: ${{ inputs.needs_token && secrets.GITHUB_TOKEN || '' }}
           CROSS_PR_TESTING: ${{ inputs.enable_cross_pr_testing && github.event_name == 'pull_request' }}
           CROSS_PR_REPO: ${{ github.repository }}