[build-tools] Add eas/posthog_wait_for_metric workflow function

[gwdp]

Why

Gate an EAS workflow on a PostHog metric. Hold a rollout until errors stay low (or any metric clears) before it continues, instead of watching dashboards by hand. imo this is the coolest 🔥

How

Adds eas/posthog_wait_for_metric. It polls a HogQL query until operator/threshold is met, every interval_seconds (default 30) for up to timeout_seconds (default 600). Auth is the same personal key as the flag step, and the key needs the query read scope. (good docs coming)

It also sets a value output with the query result that met the threshold, so a later step can use ${{ steps.<step id>.outputs.value }}.

There's no ignore_error here on purpose. It's a gate, so timing out fails the step. Network errors, 429 and 5xx retry until the deadline. Auth and query errors fail right away.

Test Plan

CI passes. Test workflow below. The second and third jobs fail on purpose (timeout and bad query shape). Not live-run yet, since the key from connect lacks the query scope. Will run it before undrafting.

Test workflow

name: PostHog wait-for-metric test

jobs:
  wait_cleared:
    steps:
      - uses: eas/posthog_wait_for_metric
        with:
          query: select count() from events
          operator: gte
          threshold: 0

  wait_timeout:
    steps:
      - uses: eas/posthog_wait_for_metric
        with:
          query: select count() from events
          operator: lt
          threshold: 0
          interval_seconds: 5
          timeout_seconds: 20

  wait_bad_query:
    steps:
      - uses: eas/posthog_wait_for_metric
        with:
          query: select event from events limit 1
          operator: gte
          threshold: 0
          interval_seconds: 5
          timeout_seconds: 15

Examples

These are simple workflows just to exercise the step. More fun and useful ones, mixed with EAS Update and deploys, are coming in the docs.

Send an event and wait for it

Two jobs with no needs, so they run at the same time. One sends the event, the other waits for it to land. This needs real concurrency. It won't clear on a local runner set to concurrency 1, since the jobs run one after another there.

name: Send an event and await it in parallel

jobs:
  send_ping:
    steps:
      - uses: eas/posthog_capture_event
        with:
          event: workflow_ping
          properties:
            source: parallel-demo

  await_ping:
    steps:
      - uses: eas/posthog_wait_for_metric
        with:
          query: select count() from events where event = 'workflow_ping' and timestamp > now() - interval 5 minute
          operator: gte
          threshold: 1
          interval_seconds: 10
          timeout_seconds: 300

Use the metric value in a later step

The gate exposes the reading it settled on as an output. A later step reads it with ${{ steps.gate.outputs.value }}.

name: Gate on a metric, then use its value

jobs:
  gated_announce:
    steps:
      - id: gate
        uses: eas/posthog_wait_for_metric
        with:
          query: select count() from events where event = '$exception' and timestamp > now() - interval 1 hour
          operator: lte
          threshold: 5
      - uses: eas/posthog_capture_event
        with:
          event: rollout_gate_cleared
          properties:
            error_count: ${{ steps.gate.outputs.value }}

[linear-code]

ENG-21551

ENG-23097

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.69%. Comparing base (f37a747) to head (bc4056b).

Additional details and impacted files

@@                           Coverage Diff                           @@
##           gwdp/eng-21551-posthog-flag-rollout    #3945      +/-   ##
=======================================================================
+ Coverage                                59.63%   59.69%   +0.06%     
=======================================================================
  Files                                      941      942       +1     
  Lines                                    41498    41555      +57     
  Branches                                  8746     8757      +11     
=======================================================================
+ Hits                                     24745    24802      +57     
  Misses                                   16659    16659              
  Partials                                    94       94              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gwdp]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [build-tools] Add eas/posthog_wait_for_query workflow function #3949
• [build-tools] Add eas/posthog_annotation workflow function #3948
• [build-tools] Add eas/posthog_upload_sourcemaps workflow function #3947
• [build-tools] Add eas/posthog_wait_for_metric workflow function #3945 👈 (View in Graphite)
• [build-tools] Add eas/posthog_flag_rollout workflow function #3944
• [build-tools] Add eas/posthog_capture_event workflow function #3934
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Subscribed to pull request

File Patterns	Mentions
**/*	@douglowder

Generated by CodeMention

Warning: The preamble and epilogue options in commentConfiguration are deprecated. Use template instead.

[sjchmiela]

i guess it would be even cooler if we could do it not-from-a-VM (type: posthog-wait-for-metric) and poll from a background job

[gwdp]

i guess it would be even cooler if we could do it not-from-a-VM (type: posthog-wait-for-metric) and poll from a background job

Tysm for the idea, it would be even cooler, tracking it under ENG-23097 as a follow-up

[github-actions]

✅ Thank you for adding the changelog entry!