Skip to content

test: cover cloud billing against Stripe test mode - #7457

Open
otavio wants to merge 4 commits into
masterfrom
test/e2e-cloud-billing
Open

otavio wants to merge 4 commits into
masterfrom
test/e2e-cloud-billing

Conversation

@otavio

@otavio otavio commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Summary

testcontainers cases for the Cloud billing items of shellhub-io/team#243, on the cloud stack against Stripe test mode, a CI leg that runs them, and the billing contract fixes the cases turned up.

tests/
├── cloud_billing_test.go          TestCloudBilling: device acceptance, device connection, webhook, namespace deletion
├── environment/stripe_api.go      StripeAPI, StripeSignature, PostStripeWebhook, BillNamespaceThrough
├── environment/stack.go           Config.StripeEvents: stripe-cli runs only on a stack that asks for it
├── cmd/stack/main.go              the console's Playwright stack still asks for it
└── main_test.go                   bundles the cloud OpenAPI spec the cloud stack validates against
openapi/spec/paths/api@billing@*   201 for customer and subscription, 400 for an unsigned webhook
.github/workflows/qa.yml           validate (tests - enterprise|cloud) matrix

Each case owns a namespace and a real test-mode subscription, deleted with its customer when the case ends. Stripe's own events no longer reach a TestCloudBilling stack: real events for a case's own subscription would race the events the case sends itself, and every other run on the account feeds events in too. A case signs the event it needs with the webhook secret and posts it to /api/webhook-billing.

 TestCloudBilling stack
-  stripe-cli  listen --forward-to gateway   (every event on the account, async)
+  test        PostStripeWebhook(payload, StripeSignature(payload))

The re-subscribe case puts its customer on a Stripe test clock with a card that declines, and advances the clock until Stripe cancels the subscription with reason payment_failed. It then pays the open invoice and sends two concurrent invoice.paid events. The namespace must re-subscribe exactly once.

fix(openapi)  document the statuses the billing routes answer   201 on create customer/subscription; 400 unsigned
test          run cloud cases without Stripe's own events       harness
test          cover cloud billing against Stripe test mode      TestCloudBilling
ci            run the cloud tests against Stripe test mode      qa.yml

Paired with shellhub-io/cloud#2652, which makes the webhook answer 400 instead of 500 to a request without a valid Stripe-Signature. The cloud CI leg checks out that branch by name.

Running them

Locally, put the three Stripe test-mode values (secret key, price ID, publishable key) in shellhub/.env.override, under the names the cloud compose file reads, and run from shellhub/:

./bin/docker-compose run --rm -T -w "$(pwd -P)/tests" test go test -run '^TestCloudBilling$' -v ./...

The cases assume what the test account has today:

  • the price is metered, graduated with the first three devices free, and aggregates usage with max
  • the server's stripe-go v79 speaks API version 2024-06-20, whose legacy usage records the server reports through
  • failed-payment retries end by canceling the subscription

CI

 validate (tests - postgres)     -skip ^TestEnterprise
+                                -skip '^(TestEnterprise|TestCloud)'
 validate (tests - enterprise)   -run ^TestEnterprise
+validate (tests - cloud)        -run ^TestCloud    with the Stripe test secrets ui-e2e.yml reads

The job keeps the validate (tests - enterprise) check name, and forks skip the cloud leg as they skip the enterprise one. The job was small enough to add here instead of leaving it as a follow-up.

team#243 items

Lines for the issue, once merged:

- [x] `[testcontainers]` Cloud billing blocks acceptance (shellhub-io/shellhub#7457, `TestCloudBilling/device_acceptance/a_namespace_without_a_subscription_is_refused_a_device_past_the_free_tier_until_it_subscribes`)
- [x] `[testcontainers]` Cloud billing reports device acceptance (shellhub-io/shellhub#7457, `TestCloudBilling/device_acceptance/accepting_a_device_reports_it_to_Stripe`)
- [x] `[testcontainers]` Active subscription does not block SSH (shellhub-io/shellhub#7457, `TestCloudBilling/device_connection/an_active_subscription_lets_SSH_through_past_the_free_tier`)
- [x] `[testcontainers]` `invoice.paid` activates subscription (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/invoice.paid_keeps_an_active_subscription_active_and_sets_the_period's_usage_to_the_accepted_devices`)
- [x] `[testcontainers]` `invoice.paid` re-subscribes canceled namespace (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/concurrent_invoice.paid_events_re-subscribe_a_namespace_canceled_for_a_failed_payment_once`)
- [x] `[testcontainers]` `invoice.paid` recovers unpaid/incomplete to active (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/invoice.paid_recovers_an_unpaid_subscription_to_active`, `TestCloudBilling/webhook/invoice.paid_recovers_an_incomplete_subscription_to_active`)
- [x] `[testcontainers]` `invoice.payment_failed` logged (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/invoice.payment_failed_is_logged_and_changes_nothing`)
- [x] `[testcontainers]` `customer.deleted` clears billing and reverts to free tier (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/customer.deleted_clears_billing_and_reverts_to_the_free_tier`)
- [x] `[testcontainers]` `subscription.updated` syncs status (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/customer.subscription.updated_syncs_the_status_and_the_period_end`)
- [x] `[testcontainers]` `subscription.updated` with `cancel_at_period_end` (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/customer.subscription.updated_with_cancel_at_period_end_keeps_the_subscription_until_the_period_ends`)
- [x] `[testcontainers]` `subscription.deleted` sets status to canceled (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/customer.subscription.deleted_cancels_the_subscription_and_takes_back_the_namespace_it_allowed`)
- [x] `[testcontainers]` Webhook rejects mismatched customer ID (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/an_event_naming_another_customer_changes_nothing`)
- [x] `[testcontainers]` Webhook rejects mismatched subscription ID (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/an_event_naming_another_subscription_changes_nothing`)
- [x] `[testcontainers]` Webhook rejects missing `Stripe-Signature` (shellhub-io/shellhub#7457, shellhub-io/cloud#2652, `TestCloudBilling/webhook/an_event_that_Stripe_did_not_sign_is_refused`)
- [x] `[testcontainers]` Webhook ignores unhandled event types (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/an_event_of_a_type_the_server_does_not_handle_is_answered_and_ignored`)
- [x] `[testcontainers]` Device accept reports usage to Stripe (shellhub-io/shellhub#7457, `TestCloudBilling/device_acceptance/accepting_a_device_reports_it_to_Stripe`)
- [x] `[testcontainers]` Device accept report blocked on `past_due` (shellhub-io/shellhub#7457, `TestCloudBilling/device_acceptance/a_past-due_subscription_refuses_accepting_a_device_and_reports_nothing`)
- [x] `[testcontainers]` Namespace delete blocked with open invoices (shellhub-io/shellhub#7457, `TestCloudBilling/namespace_deletion/a_namespace_whose_canceled_subscription_has_an_open_invoice_is_deleted_only_once_the_invoice_is_voided`)
- [x] `[testcontainers]` Subscription deletion decreases user `MaxNamespaces` (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/customer.subscription.deleted_cancels_the_subscription_and_takes_back_the_namespace_it_allowed`)
- [x] `[testcontainers]` Webhook tenant locking — concurrent events serialized (shellhub-io/shellhub#7457, `TestCloudBilling/webhook/concurrent_invoice.paid_events_re-subscribe_a_namespace_canceled_for_a_failed_payment_once`)

"invoice.paid activates subscription" covers the handler's path for a subscription that is already active: it stays active and the period's usage is set to the namespace's accepted device count. Recovering an unpaid or incomplete subscription is its own item. "Unpaid/incomplete/paused blocks both accept AND connect" stays out, as the issue marks it unreachable. Nothing else is left uncovered. The SSH case also checks that SSH past the free tier is refused once the subscription is deleted, which shows the subscription is what lets it through.

Not duplicated from #7437 or #7429: no case here asserts the community device limit, slot freeing or name conflicts.

Refs: shellhub-io/team#243

Evidence

Every run is in the test compose runner against Stripe test mode.

  • First run, before the fixes: 15 of 19 cases passed. The other four:
    an_event_that_Stripe_did_not_sign_is_refused   expected: 400  actual: 500 (both requests)
    invoice.paid_keeps_...                         expected usage 1, actual 5: the price aggregates with max
    concurrent_invoice.paid_..._once               "active" after 5 steps: usage 1 is in the free tier, invoices were $0
    a_namespace_whose_canceled_subscription_...    Stripe rejects pending_invoice_items_behavior with subscription
    
    The first is the cloud fix in shellhub-io/cloud#2652. The other three were fixture errors, fixed before commit.
  • Mutant A broke each behavior the cases cover at once and ran every case but the concurrent one. The changes: the free-tier guard off; past-due and the Stripe report skipped on accept; evaluate refusing SSH when active; the open-invoice check off; invoice.paid neither recovering nor setting usage; the payment_failed log reworded; customer.deleted keeping the subscription; subscription.updated dropping the period end and the ID checks; cancel_at_period_end ignored; subscription.deleted keeping MaxNamespaces; customer.subscription.created handled as an update. Every item case failed for its own reason:
    free tier / past-due accept     expected: 402  actual: 200
    reports to Stripe               usage never reached 1
    invoice.paid usage              usage never reached 3
    active SSH                      never connected
    recovers unpaid / incomplete    expected: "active"  actual: "unpaid" / "incomplete"
    payment_failed                  log line never appeared
    customer.deleted                subscription still set
    subscription.updated            period end expected: 1796933780  actual: 0
    cancel_at_period_end            expected: "to_cancel_at_end_of_period"  actual: "active"
    subscription.deleted            MaxNamespaces not taken back
    mismatched customer / sub       event applied, status past_due
    unhandled type                  event applied, status past_due
    open invoice                    expected: 402  actual: 200
    
  • Mutant B, the tenant lock removed: the concurrent case failed in both of two runs, with MaxNamespaces raised twice and two active Stripe subscriptions where one is expected.
  • Mutant C, re-subscribing disabled: the concurrent case failed with the subscription ID unchanged and the status canceled.
  • After: TestCloudBilling passes, 19/19 subtests in 290 s, the re-subscribe case taking about 90 s. golangci-lint v2.14.0 on tests/ with --build-tags docker and uncapped issues: 0 issues. go vet, the environment unit tests and go mod tidy are clean. The OpenAPI prettier and redocly checks pass, with the same two redocly warnings master has. actionlint passes on qa.yml. The community suite did not run locally; this branch changes it only by bundling one more spec in TestMain, and CI runs it.

Review

Three Codex (gpt-6.1-sol, high effort) rounds on the Standards and Spec axes:

  1. Changes requested.
    • Blocker: the draft invoice. The open-invoice fixture made a draft invoice and deleted it, which Stripe refuses for a subscription invoice. Accepted: it now finalizes the invoice to open, and the case voids it.
    • Blocker: the 201s. Create customer and create subscription answer 201 where the spec says 200. I first rebutted it, since every subscribe passed strict validation; round 2 showed that proves nothing.
    • The unsigned 500. Accepted: the cloud fix.
    • Customer cleanup. It was registered after the asserts. Accepted: it is registered before the request now.
    • Usage under max aggregation. Accepted: both cases were rebuilt around it.
    • Serialization. The concurrent case didn't prove the requests overlap. Accepted, answered with mutants B and C.
    • The ctx doc nit. Accepted.
    • Rejected: host Go in CI. Every qa.yml job runs Go through setup-go, the enterprise leg this extends included.
    • Rejected: pkg/errors. The tests module's helpers use fmt.Errorf throughout.
  2. Changes requested.
    • Blocker: the skip pattern. The unquoted -skip ^(TestEnterprise|TestCloud) broke bash in the community job. Fixed: the regex is quoted.
    • kin-openapi. The validator passes a status the spec doesn't document, so the 201s needed the spec fixed. Fixed in fix(openapi).
    • The StripeWebhook doc. The doc comment I touched didn't state its whole contract. Fixed.
    • Round 2 accepted both rejections.
  3. Clean on both axes.

Merge Danger

Door: two-way

Blast Radius: CI

tests/ and CI change, plus two status codes and one sentence in the OpenAPI spec that now match what the server answers. The cloud leg adds a validate (tests - cloud) check that creates and deletes customers, subscriptions and test clocks on the shared Stripe test account. It depends on that account keeping the price and failed-payment settings listed above. TestCloudBilling runs its stack without stripe-cli. The stack the console's Playwright suite runs on still has it, because billing.spec.ts waits on Stripe's own events and cleans up through it. The validate (tests - postgres) job now also skips ^TestCloud.

@otavio
otavio requested a review from a team as a code owner October 10, 2026 21:05
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude finished @otavio's task in 6m 26s —— View job


Code Review Complete

Reviewed 11 files. 2 inline issues posted. Standards: 0 violations, 2 judgement calls. Spec: not run. Correctness: 0.

Standards

No rule is broken. The more useful of the two judgement calls is the StripeSignature doc comment (tests/environment/stripe_api.go:35-37). It leaves out the empty-secret case that its sibling StripeAPI states.

Spec

The Spec axis did not run. The only issue this PR references is shellhub-io/team#243, which is in another repository this run cannot read.

Additional notes

  • A correctness candidate was discarded. It claimed openStripeInvoice would produce a $0 invoice because pending_invoice_items_behavior is unset. That claim conflicts with how Stripe handles invoices tied to a subscription, and with the PR's evidence: Stripe rejected that parameter, and the case passed 19/19.
  • Two weaker judgement calls were dropped at the bar, not by the cap:
    • folding the three "changes nothing" webhook cases into a table;
    • splitting the 217-line testBillingWebhook.
  • Checked and clean:
    • CI fork gating and secret handling on the new cloud leg.
    • The validate (tests - enterprise) check name is preserved.
    • The quoted -skip regex.
    • The Stripe HMAC construction.
    • The webhook secret is still read through stripe listen --print-secret.
    • Cleanup ordering for Stripe customers and test clocks.
    • Nothing in cloud/ references the deleted docker-compose.cloud.test.yml.

To request another review round, comment /review.

Comment thread tests/environment/stripe_api.go Outdated
Comment thread tests/cloud_billing_test.go Outdated
POST /api/billing/customer and POST /api/billing/subscription answer 201, as the cloud handlers
always have, but the spec declared 200. The strict response validator the e2e stacks run does not
catch it: kin-openapi passes a status an operation leaves undocumented unless the validator is
built with IncludeResponseStatus, and ours is not.

The webhook now says a request without a valid Stripe-Signature is answered with 400, which the
matching shellhub-io/cloud fix makes true; it answered 500 before.
@otavio
otavio force-pushed the test/e2e-cloud-billing branch from bf8ba2c to c5b0f43 Compare October 10, 2026 23:28
@otavio

otavio commented Oct 10, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude finished @otavio's task in 4m 46s —— View job


Code Review Complete

Reviewed 11 files for Standards (repo rules, Go conventions, tests, code smells) and Correctness (security and runtime defects). No new issues found. The code looks good as-is.

The Spec axis did not run. The only issue this PR references is shellhub-io/team#243, and this run cannot read another repository.

The cloud test stack ran stripe-cli, which forwards every event on the Stripe test account to the
stack's webhook, asynchronously and in no fixed order. A case that moves a subscription to past_due
by sending an event would race the customer.subscription.updated Stripe itself emits moments after
the subscription is created, and every other run or dev stack on the account feeds events in too.
A stack now runs stripe-cli only when its Config sets StripeEvents. A case leaves it unset, signs
the event it needs with the secret `stripe listen --print-secret` returns, the one the server
verifies against, and posts it itself.

The stack command behind the console's Playwright suite sets StripeEvents for a cloud stack:
billing.spec.ts subscribes through the UI and waits on the events Stripe sends back, and it
cleans up through `docker compose exec stripe-cli`.

The cloud stack serves openapi-cloud.json under strict validation, but TestMain bundled only the
community and enterprise specs, so a cloud stack would mount a file the run never wrote.

StripeAPI pins Stripe-Version to 2024-06-20, the version stripe-go v79 in cloud speaks. Versions
from 2025-03-31 on dropped the legacy usage records the server reports devices through and the
tests read back, and an account's default version can be one of them.

BillNamespaceThrough writes the namespace's billing row directly: ShellHub creates the customer it
bills a namespace through itself, and only the test can create one on a Stripe test clock.
@otavio
otavio force-pushed the test/e2e-cloud-billing branch from c5b0f43 to fe59642 Compare October 11, 2026 23:00
@otavio

otavio commented Oct 11, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Claude finished @otavio's task in 7m 15s —— View job


Code Review Complete

Reviewed 13 files. 2 inline issues posted. Standards: 0 violations, 2 judgement calls. Spec: not run. Correctness: 0.

Standards

No rule is broken. The more useful of the two judgement calls: the same customer.subscription.deleted event is built by hand in four cases (tests/cloud_billing_test.go:441). It would fit as a billedSubscription builder next to object, invoice and customerObject.

Spec

The Spec axis did not run. The only issue this PR references is shellhub-io/team#243, which is in another repository this run cannot read.

Additional notes

  • Both earlier findings are fixed on this head:
    • the StripeSignature doc comment now states the other-edition case;
    • no raw "canceled" literal is left.
  • One judgement call fell short of the bar and was dropped: a helper for the repeated StripeAPI → NoError → StatusOK sequence.
  • The Correctness pass checked these against the cloud handlers and found nothing:
    • the 201 and 400 statuses in the OpenAPI spec;
    • every expected webhook outcome;
    • how the tenant lock orders the concurrent invoice.paid events;
    • StripeEvents turning stripe-cli on and off;
    • the direct billing-row write in BillNamespaceThrough.
  • One flakiness risk, not counted as a finding: requireStripeUsageHolds treats any Stripe API error as the usage changing. A single transient network error during its 10-second window would fail the case.

To request another review round, comment /review.

Comment thread tests/cloud_billing_test.go Outdated
Comment thread tests/cloud_billing_test.go Outdated
@otavio
otavio force-pushed the test/e2e-cloud-billing branch from fe59642 to 8f87efa Compare October 11, 2026 23:40
@otavio

otavio commented Oct 11, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Claude finished @otavio's task in 4m 24s —— View job


Code Review Complete

Reviewed 13 files. 2 inline issues posted. Standards: 0 violations, 2 judgement calls. Spec: not run. Correctness: 0.

Standards

No rule is broken. The more useful of the two judgement calls is about the webhook event types (tests/cloud_billing_test.go:603). They are raw strings repeated 21 times. A typo in one of the "changes nothing" cases would still pass, because the server answers an unknown type with 200 and ignores it. The second call is the missing Billing nil check in startSubscription (:164). It is the only first read of Billing that skips the guard, and a nil there would panic and abort the whole run.

Spec

The Spec axis did not run. The only issue this PR references is shellhub-io/team#243, which is in another repository this run cannot read.

Additional notes

  • All four findings from earlier rounds are fixed on this head.
  • Four judgement calls fell short of the bar and were dropped:
    • the commit message doesn't say why payOpenInvoices also pays uncollectible invoices;
    • the commit message doesn't say that deleting the test clock also deletes its customer;
    • the unsigned-signature cases loop over a map without t.Run;
    • enrollIn overlaps with enrollPendingDevice.
  • The Correctness pass checked these against the cloud handlers and found nothing:
    • every webhook outcome;
    • the [200, 200] result of the concurrent re-subscribe case under the tenant lock;
    • past_due refusing accept before it reports usage;
    • deliverConcurrently leaking no goroutines;
    • the stripe compose profile and StripeEvents;
    • the cloud spec bundling;
    • the qa.yml matrix inputs.
  • No pkg/ or server/api/services symbol changed, so cloud is not affected.

To request another review round, comment /review.

Comment thread tests/cloud_billing_test.go Outdated
Comment thread tests/cloud_billing_test.go
The cases read Stripe the way the test account bills: the price is graduated with the first three
devices free and aggregates usage with max. So the invoice.paid case accepts the free tier's devices
before subscribing, leaving Stripe no usage record, and expects the paid invoice to set usage to
three; and the re-subscribe case sets usage to four so the period's invoice is not $0 and the
declining card fails it.

The re-subscribe case runs its customer on a Stripe test clock and advances it in 30-day steps until
Stripe cancels the subscription with reason payment_failed. That relies on the account's failed
payment setting canceling the subscription after the last retry, as it does today.

The past-due case holds Stripe's usage for ten seconds after the refused accept. The server reports
inside the accept request, before it answers, so any record already exists by then; the hold only
covers the usage summary catching up.

Refs: shellhub-io/team#243
The enterprise job becomes a matrix over the editions that build from the cloud source. Its name
template still yields "validate (tests - enterprise)", so the check keeps the name it has today. The
cloud leg reads the Stripe test secrets ui-e2e.yml already uses, and a fork skips it as it skips the
enterprise leg, for want of the cloud source.

The community job's skip pattern is quoted because run: pastes it into a bash line, where an
unquoted ( is a syntax error.
@otavio
otavio force-pushed the test/e2e-cloud-billing branch from 8f87efa to 1c4c9e9 Compare October 12, 2026 00:38
@otavio

otavio commented Oct 12, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Claude finished @otavio's task in 2m 44s —— View job


Code Review Complete

I reviewed 13 files for Standards (repo rules, Go conventions, tests, code smells) and Correctness (security and runtime defects). I found no new issues, so the code can stay as it is.

The Spec axis did not run. The only issue this PR references is shellhub-io/team#243, and this run cannot read another repository.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant