SRVOCF-1038: Show build status and pipeline failures in the functions list - #177
SRVOCF-1038: Show build status and pipeline failures in the functions list#177matejvasek wants to merge 40 commits into
Conversation
Design for surfacing GitHub Actions build status and pipeline failures in the functions list, via an SSE stream from the backend (polling GH Actions) read with consoleFetch. Covers the status merge with the existing cluster watch, new Building/BuildFailed statuses, parameterless user-scoped endpoints, fakegithub Actions API with /_admin control, and the test strategy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add admin endpoints so tests and dev can script a repo's GitHub Actions workflow run status, conclusion, and jobs, making build status deterministic to exercise end to end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fetch the latest workflow run for a repo's default branch through the GitHub Actions REST API and, on failure, derive a "<job> / <step>" reason from the failed job. Includes fakegithub coverage and failureReason fallbacks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a snapshot endpoint and an SSE watch endpoint that streams per-repo build status, polling GitHub on an interval with heartbeats and periodic repo rediscovery. Per-repo errors are surfaced in the snapshot, and each snapshot is marshalled once with the bytes reused as the change key so unchanged polls are skipped. Wire the routes into main. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stream build status over SSE via consoleFetch, sending the PAT in the X-SCM-Token header, with reconnect/backoff and stop-on-auth-error. Includes a consoleFetch stream test stub. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Merge streamed build status into the functions list: render Building and BuildFailed (with a link to the run and a failure-reason tooltip), while letting a Running cluster status win over a stale Failed build. Also repairs the setup-guide test orphaned by a master helper rename. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pass the auth connectionId into useBuildStatus so the stream tears down and reconnects with the current PAT on in-place login and account switch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…limit usage The build-status poll loop hit GitHub every 3s per repo, exhausting the 5,000/hr core rate limit. Wire a per-client in-memory httpcache transport so unchanged responses come back as 304 Not Modified, which do not count against the primary rate limit. GitHub sends Cache-Control: max-age=60 on these responses, which would let the cache serve a stale build status for up to ~60s. A forceRevalidate transport sets Cache-Control: max-age=0 on every request so the cache always revalidates with a conditional request: unchanged status stays a free 304, but a real change is seen on the next poll. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A 2xx response with no body previously fell through the `if (!res.body) return;` guard and permanently stopped the SSE stream, so the build-status badges would silently freeze until the next connectionId change. Treat a body-less response like any other stream end: fall through to the backoff-and-reconnect path instead of giving up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LatestWorkflowRun took the newest run across *all* workflows in a repo
(ListRepositoryWorkflowRuns), so an unrelated workflow (lint, CodeQL, a cron)
could mask or misrepresent the func build: a passing lint run could hide a
failed build, or a failing unrelated workflow could paint a function red.
Filter to the func build workflow by file name via ListWorkflowRunsByFileName.
The identifier is func's own DefaultGitHubWorkflowFilename ("func-deploy.yaml"),
re-exported from the scaffold package as scaffold.WorkflowFilename so it stays in
sync with what we actually scaffold. The scm layer stays func-agnostic: the
workflow file name is passed in as a parameter, supplied by the func-aware
handler. A repo without that workflow file returns 404, which we map to a nil
run (no build signal) so non-func repos and not-yet-pushed workflows fall back
to the cluster-derived status instead of erroring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tream consoleFetch applies a default ~60s request timeout that aborts the request when it fires. On the long-lived build-status SSE stream that tore the connection down every minute regardless of the backend's 15s heartbeats, forcing a reconnect and a full initial snapshot re-fetch from GitHub each time. Pass timeout 0 to disable it so the stream is ended only by the hook's own AbortController (on unmount or connectionId change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e functions A function that is deployed and available (serving `Running` or idle `ScaledToZero`) now keeps its cluster status while a rebuild runs or fails, instead of being overwritten by `Building`/`BuildFailed`. The build activity is surfaced only as a small secondary indicator next to the status: a spinner (tooltip "Build in progress") while building, or a red danger-colored warning icon (tooltip "Latest build failed: <reason>", link to the run) when the latest build failed. This stops an available function from flip-flopping to a build-centric status on every redeploy and keeps availability accurate. Deferred: giving a cluster `Error` (broken deployed revision) the same non-destructive treatment. `Error` is overloaded (it also covers a repo/list error with no cluster resource), so doing it right means gating on cluster presence rather than the status string. Noted in the design doc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… workflow
The fake's by-file-name runs endpoint
(/repos/{owner}/{repo}/actions/workflows/{workflow}/runs) shared a handler with
the repo-wide endpoint and ignored the {workflow} path segment, so both returned
every scripted run. That gave the fakegithub and e2e suites no fidelity for
workflow-file scoping: a regression where build status stopped querying only
func-deploy.yaml would go uncaught.
Give each scripted run a workflow-file identity (defaulting to
functions.WorkflowFilename so it stays in sync with what the client requests, and
overridable via the admin /_admin/actions/runs "workflow" field) and filter by
the {workflow} path segment on the by-file-name route. The repo-wide route still
returns all runs. Add a test asserting a run under a different workflow is not
returned when querying the func workflow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…id-stream Once the SSE stream is established, a per-repo LatestWorkflowRun error (including ErrUnauthorized) is logged and the last-known status is carried forward, and the 30s rediscover ListRepos error was only logged. So if the caller's PAT was revoked after connecting, every poll failed, the change-detection key never moved, no new frame was sent, and the client showed stale build status indefinitely without ever seeing an auth error to trigger re-auth. ListRepos is a single global call, so its ErrUnauthorized unambiguously means the token is no longer valid. End the stream in that case; the client's reconnect then hits the initial ListRepos, gets a 401 before the SSE upgrade, and its existing isAuthError path stops the loop / prompts re-auth. Detection latency is bounded by the rediscover interval. Non-auth rediscover errors still just log. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-repo build-status item carried an "Err" field populated with the raw error string from a failed workflow-run fetch. That string was never consumed by the frontend but was serialized onto the wire, exposing internal error detail to the browser. Drop the field: a failed fetch with no prior state now reports a plain "None" item and the cause is logged server-side instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
Skipping CI for Draft Pull Request. |
|
/test all |
|
/test e2e-aws |
|
@matejvasek: This pull request references SRVOCF-1038 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.1.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
… fakegithub
GET /repos/{owner}/{repo}/actions/runs was added when the client still listed runs
repo-wide. Scoping build status to the func workflow left it without a caller: the
only Actions requests the backend makes are ListWorkflowRunsByFileName, which hits
the by-file-name route, and ListWorkflowJobs. Nothing else reaches the repo-wide
listing either, neither the fakegithub tests, the watch tests, nor the e2e helper.
Removing it lets handleListWorkflowRuns filter on the workflow unconditionally,
since the remaining route always supplies it. The two comments that go with it only
existed to explain the difference between the two routes.
The jobs endpoint and the admin run endpoint stay. failureReason depends on the
first, and it is the only way e2e and the unit tests script run state.
Signed-off-by: Matej Vašek <matejvasek@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@twoGiants Production code is about |
|
/test all |
| InitRepo(ctx context.Context, owner, name, branch string, topics []string) error | ||
| StoreSecret(ctx context.Context, owner, repo, name, value string) error | ||
| DeleteRepo(ctx context.Context, owner, repo string) error | ||
| WatchWorkflowRuns(ctx context.Context, workflowFile string) (<-chan []RepoRun, error) |
There was a problem hiding this comment.
Self-review note on this signature, error handling specifically. Weak suggestion, not necessary for this PR, filing it so the reasoning is recorded rather than lost.
WatchWorkflowRuns(ctx context.Context, workflowFile string) (<-chan []RepoRun, error)The setup/stream split is right: repo discovery is synchronous, so HandleBuildWatch can still return a real 401 before the response turns into SSE. The weaker end is termination. Closing the channel is the only signal, and it carries one bit.
Three different things close it:
- ctx cancelled, i.e. the browser disconnected (normal)
- the token was revoked mid-stream (
scm/github/watch.go:87) - any terminal producer error we add later
The consumer cannot tell them apart, which is why handler/build.go:85 has to hedge: "Watch ended (cancelled, or the token was revoked mid-stream)." That is fine today because both paths end the SSE stream, but the handler cannot log the difference, and a future terminal error would be indistinguishable from a clean shutdown.
If we ever want to distinguish them, abort would stay ctx-only (deliberately no Stop(), so there is no "who wins" question and no drain-to-avoid-deadlock contract):
// RunWatch carries a stream of workflow-run snapshots. C is closed when the
// watch ends; Err then reports why, or nil if it ended cleanly because ctx
// was cancelled.
type RunWatch struct {
C <-chan []RepoRun
err error
}
// Err is valid only after C is closed: the close is the happens-before edge
// that publishes err. Same contract as sql.Rows.Err.
func (w *RunWatch) Err() error { return w.err }WatchWorkflowRuns(ctx context.Context, workflowFile string) (*RunWatch, error)Producer sets err before close(out); consumer reads it only in the !ok branch:
case snapshot, ok := <-runs.C:
if !ok {
if err := runs.Err(); err != nil {
slog.Info("build watch: stream ended", "err", err)
}
return
}Two notes on the shape:
Erris a method rather than a field mainly so it could gain anatomic.Pointer[error]later without touching call sites, if the "only after close" contract turned out to be too sharp. As written it is no more synchronized than a field would be; the safety comes from the close, not from the accessor.- Not generic. One instantiation, and
*Watch[[]RepoRun]reads badly. Easy to promote later if a second watch appears.
The other common shape, an Event{Value, Err} channel, seems worse here: every consumer branches per element on a field set at most once, and you can still receive after the error unless you also close.
Separate and much smaller, same file: ClientStub.WatchWorkflowRuns returns (nil, nil) by default (client.go:170). A nil channel never fires in select, so a test that forgets to set OnWatchWorkflowRuns gets an SSE stream that heartbeats forever instead of failing. Returning a closed channel would surface the mistake immediately.
Neither is blocking. Happy to leave both for a follow-up.
…ntation The spec was written before the backend settled and drifted in eight places. The snapshot endpoint it documents no longer exists, the SCM method it names is LatestWorkflowRun rather than WatchWorkflowRuns, the payload is a keyed map and not an array, the fakegithub route is the by-file-name one, the hook takes a connectionId, and the failed tooltip dropped the reason. Two decisions taken during implementation were never captured: scoping build status to the func workflow file, and revalidating every poll against the ETag cache so unchanged runs answer 304 and stay off the rate limit. Both are now in the GitHub implementation section, along with the change-only channel, the per-repo error carry-forward, and the rediscover path that ends the stream on a revoked token. The problem statement, the status merge table, the deferred cluster Error note, and the transport decision all still match the code and are untouched. Signed-off-by: Matej Vašek <matejvasek@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
/test all |
| { | ||
| path: 'func.yaml', | ||
| mode: '100644', | ||
| content: `name: ${FUNC_NAME}\nruntime: node\nnamespace: default\n`, |
There was a problem hiding this comment.
default namespace should not be used. Consider switching to PRESEEDED_FUNC_NAMESPACE.
It does not change much right now as we are faking everything, but in the future, after integrating act into the fakegithub, IMO this test should be adapted to full flow: create function, let the workflow run in act, verify Building, update to broken code in function, deploy it, verify...
@matejvasek yes we commit the plans and designs. I want the AI contributions to this project be transparent and visible. Especially I want it to be visible how much review is needed for AI powered PRs. Communicating that the plan is the biggest chunk of the PR already reduces PR review burden. |
@Cragsmann lets connect and you show me exactly what you mean. I want to understand where you're seeing the mess and the scalability issues. Keeping child components which are only used by one parent in the same module scales equally to keeping private methods of a class in that same class. Please, lets align and clarify. |
Clicking the "build in progress" spinner left a rounded focus outline that then rotated with the spinner animation (PR review r3932309368). The cause is not the spinner: PatternFly's Tooltip defaults to trigger="mouseenter focus click" and so attaches a focus listener to whatever it wraps, and Blink makes an <svg> focusable as soon as it has a focus or blur listener. Wrapping the spinner in a span moves the listener onto an HTML element, which Blink does not treat that way, so nothing in the cell is focusable any more and the ring is gone. Verified in Chromium: activeElement stays on BODY after clicking the spinner, and the tooltip still opens on hover and on click. Signed-off-by: Matej Vašek <matejvasek@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The composed "<job> / <step>" reason was rendered in exactly one place, the BuildFailed tooltip, after the secondary indicator moved to a fixed "Latest build failed". It was not worth what it cost: - an extra ListWorkflowJobs call per failed repo on every 3s poll, for as long as the function stayed failed - a GitHub-Actions-shaped string on the wire, with no meaning for a Tekton or Konflux build - little information: "deploy / Run tests" says where, not what, so you open the run either way The BuildFailed tooltip goes with it rather than being reworded, because the badge already reads "BuildFailed". The badge still links to the failing run, which is the part that was actually useful. Removes the field from scm.WorkflowRun, the ghClient.failureReason lookup, the wire DTO, the fakegithub jobs endpoint and its job/step types, and the frontend plumbing down to FunctionTableItem. Signed-off-by: Matej Vašek <matejvasek@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… stub A nil channel blocks a receive forever, so a test that used ClientStub without setting OnWatchWorkflowRuns left HandleBuildWatch parked in its select until the request context was cancelled. Hanging is a worse failure than failing. A closed channel takes the end-of-stream path the handler already has, matching the other stub methods, which all return promptly. Suggested by @pmeida in review of #177. Signed-off-by: Matej Vašek <matejvasek@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
as discussed in F2F we also need to change 3 icons, so that it is consistent with the rest of the platform and it doesn't repeat the same icon twice in the status column: My suggestion: secondary status:
primary status:
|
GitHub can push workflow_run events to a webhook, which would be lower latency than a 3s poll, but it needs a publicly reachable route into the cluster, a webhook plus signing secret registered and kept in sync on every function repo, signature verification, and server-side state to route each event to the right browser session. Polling needs none of that, runs inside the existing user-scoped request, and unchanged polls are 304s. The spec recorded a transport decision only for the backend-to-browser hop, which is the smaller of the two choices. Signed-off-by: Matej Vašek <matejvasek@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rebuilding a deployed function flickered its row through Building for a second or two: Running -> Building -> Running. The merge treated only Running and ScaledToZero as available, but a live function reports Deploying while the build applies its new revision (ksvc Ready=Unknown, or no Deployment for the latest ready revision yet), so the in-progress build overwrote the cluster status during that window. Gate the non-destructive branch on cluster presence rather than on the status value. mergeBuild now takes inCluster, which the caller already has from its ClusterFunction lookup. A cluster Error now also keeps its status with the build as a secondary indicator, which the spec had deferred for exactly this reason, since presence is what separates a broken ksvc from a repo-level read error that still has to fall through to BuildFailed. StatusCell routes Deploying and Error through withBuildActivity so the secondary indicator survives those statuses. Building and BuildFailed stay bare, as they only occur when the cluster knows nothing about the function. Reported by @pmeida in review of #177. Signed-off-by: Matej Vašek <matejvasek@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
twoGiants
left a comment
There was a problem hiding this comment.
First bigger batch of the FE review.
Great job overall as always. That was a lot of work, I appreciate it! Thank you for using the new testing patterns in the List page and in the hook tests. And thank you for creating the test doubles using the new style. It's cool that we're moving in this direction.
Now lets improve a few things:
- The hook can be written better testable and better usable for it's clients.
- The test double API can be simplified and the stub can become a fake.
I continue next week.
| // END: isAllNamespaceKeyFake -------------------------------------------------- | ||
|
|
||
| // START: consoleFetchStreamStub ----------------------------------------------- | ||
| // Test double for the SSE stream consumed by useBuildStatus. Mirrors the |
There was a problem hiding this comment.
Pls remove all the zero value comments.
I'll tag them with "remove comment" from now on.
| // hook's cross-read buffering. | ||
|
|
||
| let frames: string[] = []; | ||
| let streamError: unknown = null; |
There was a problem hiding this comment.
Use a concrete type.
unknown is as good as any type. Only use it if there is really no other way.
In the fixtures setup for useK8WatchResourceStub I was able to refactor towards known types everywhere.
| const pat = sessionStorage.getItem(PAT_KEY); | ||
| if (!pat) return; |
There was a problem hiding this comment.
This should be done via the context. This is for a follow up. We have a ticket for it https://redhat.atlassian.net/browse/SRVOCF-842. Can you add to the ticket that this hooks will need to be updated too.
| const [statuses, setStatuses] = useState<ReadonlyMap<string, BuildStatus>>(() => new Map()); | ||
|
|
||
| useEffect(() => { | ||
| let cancelled = false; |
There was a problem hiding this comment.
Maybe name it streaming or running? Then the statements read a bit nicer down below e.g. while (streaming).
| if (isAuthError(err)) { | ||
| // A bad or expired PAT will not recover on retry, so stop rather | ||
| // than reconnect in a tight loop. | ||
| console.error( |
There was a problem hiding this comment.
We log errors but do nothing about it. Same below.
I'd rather return the error like useCluster does and handle it in the client.
| const { result } = renderHook(() => useBuildStatus()); | ||
|
|
||
| await waitFor(() => expect(result.current.size).toBe(1)); | ||
| expect(streamStub.streamFetchLastArgs()[2]).toBe(0); |
There was a problem hiding this comment.
This is a contract test -> timeout must be 0 or it disconnects after 60s. It's important that we keep it. Just don't spy on internals. Use a fake implementation, then fake timers and assert on the result.
The type (no unknown):
type ConsoleFetch = (url: string, options?: RequestInit, timeout?: number) => Promise<Response>;The stub:
export const consoleFetchStub = (
url: string,
options?: RequestInit,
timeout?: number,
): Promise<Response> => {
if (streamError) return Promise.reject(streamError);
const stream = new ReadableStream<Uint8Array>({
start(controller) {
const encoder = new TextEncoder();
for (const chunk of frames) {
controller.enqueue(encoder.encode(chunk));
}
// Keep the stream open like a real SSE connection.
// End it only when the caller aborts or the timeout fires.
options?.signal?.addEventListener('abort', () => controller.close());
if (timeout) {
setTimeout(() => controller.error(new Error('timeout')), timeout);
}
},
});
return Promise.resolve(new Response(stream, { status: 200 }));
};The test:
it('keeps the stream alive beyond the default consoleFetch timeout', async () => {
vi.useFakeTimers();
streamStub.setStreamFrames([
streamStub.buildStatusFrame({ 'alice/fn': { buildStatus: 'Building' } }),
]);
const { result, unmount } = renderHook(() => useBuildStatus());
await waitFor(() => expect(result.current.size).toBe(1));
// Advance past the default 60s consoleFetch timeout.
// If the hook didn't disable it, the stream would be dead and data cleared on reconnect error.
await vi.advanceTimersByTimeAsync(120_000);
expect(result.current.get('alice/fn')?.buildStatus).toBe('Building');
unmount();
});| it('ignores a frame with no event name', async () => { | ||
| // An unnamed frame is a default "message" event, not our build-status event. | ||
| vi.useFakeTimers(); | ||
| streamStub.setStreamFrames(['data: {"functions":{"a/b":{"buildStatus":"Building"}}}\n\n']); | ||
|
|
||
| const { result, unmount } = renderHook(() => useBuildStatus()); | ||
| // Reconnecting proves the frame was read and dropped, not merely unread yet. | ||
| await vi.advanceTimersByTimeAsync(10_000); | ||
|
|
||
| expect(streamStub.streamFetchCalls()).toBeGreaterThan(1); | ||
| expect(result.current.size).toBe(0); | ||
|
|
||
| unmount(); | ||
| }); | ||
|
|
There was a problem hiding this comment.
Test on the visible behaviour and the result. I'd send an unnamed event followed by named event and only the named should appear in the results. With the refactored test, the proof that the unnamed frame was consumed is that the valid frame after it was processed.
it('ignores a frame with no event name', async () => {
streamStub.setStreamFrames([
'data: {"irrelevant":"not a build-status event"}\n\n',
streamStub.buildStatusFrame({ 'c/d': { buildStatus: 'Succeeded' } }),
]);
const { result } = renderHook(() => useBuildStatus());
await waitFor(() => expect(result.current.size).toBe(1));
expect(result.current.get('c/d')?.buildStatus).toBe('Succeeded');
});| it('stops reconnecting after an auth failure', async () => { | ||
| vi.useFakeTimers(); | ||
| streamStub.setStreamError(Object.assign(new Error('unauthorized'), { code: 401 })); | ||
| const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); | ||
|
|
||
| const { unmount } = renderHook(() => useBuildStatus()); | ||
| // Advance well past the 3s backoff window; a stopped stream must not retry. | ||
| await vi.advanceTimersByTimeAsync(10_000); | ||
|
|
||
| expect(streamStub.streamFetchCalls()).toBe(1); | ||
| expect(errorSpy).toHaveBeenCalled(); | ||
|
|
||
| unmount(); | ||
| }); |
There was a problem hiding this comment.
You need two tests here, one for 401 and one for 403 and you must return error from the hook so that you can assert against it here. No spy needed then.
it('surfaces an auth error and stops the stream', async () => {
streamStub.setStreamError(Object.assign(new Error('unauthorized'), { code: 401 }));
const { result } = renderHook(() => useBuildStatus());
await waitFor(() => expect(result.current.error).toBeTruthy());
expect(result.current.statuses.size).toBe(0);
});| ); | ||
| return; | ||
| } | ||
| console.error('useBuildStatus: build status stream error, reconnecting', err); |
There was a problem hiding this comment.
This doesn't look right. Any error except 401 and 403 is a reconnect. Are you sure about that? I would rather indicate somehow in the UI that we're reconnecting instead of doing it silently then we can also test easier against it.
So this should probably set some state which is returned from this hook like { reconnecting }.
There was a problem hiding this comment.
Are you sure about that?
TBH I am not. I am only sure that with 401 and 403 there is no point in retry.
What error should be retryable? Connection dial error? 503 or any 5XX?
There was a problem hiding this comment.
I'll stop for now. I'm not yet done with reviewing all the tests.
What is clear so far is that the useBuildStatus implementation is difficult to test, so you'll need to refactor it a bit.
Also the consoleFetchStub test double needs refactoring towards a fake and the spy API for consoleFetchStub needs to be removed. After a refactoring of useBuildStatus you won't need it. You should always be able to test against observable behaviour.
watchPollInterval, watchRediscoverInterval and buildHeartbeatInterval were package-level vars so tests could speed up the loops they exercise. Each test wrote them from the spec goroutine while the code under test read them from the background goroutine it had already started, which is a data race that -race reports. Running specs serially does not help, since both goroutines belong to the same spec. Make the cadence per instance. The github client keeps pollInterval and rediscoverInterval as ghClient fields, set by NewWithBaseURL from consts and overridable through the new WithWatchIntervals option. They are written before the watch goroutine starts, so the go statement supplies the happens-before edge. New stays non-variadic to remain assignable to scm.ClientFactory, so options go through NewWithBaseURL. HandleBuildWatch becomes BuildWatch, a function returning an http.HandlerFunc that closes over its heartbeat. It read no Handlers field, and adding one would have made six unrelated handlers carry a heartbeat they never use, so it comes off the struct entirely and main wires it as handler.BuildWatch(). Both constructors now reject a non-positive interval with a panic at construction. Previously time.NewTicker panicked inside a background goroutine instead, taking the process down rather than failing the call that caused it. With the tunables no longer package state, the tests no longer reach into either package. watch_test.go and build_test.go move to github_test and handler_test, and the status mapping is now asserted through the SSE frame the frontend consumes rather than by calling the unexported deriveBuildStatus. Signed-off-by: Matej Vašek <matejvasek@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BuildWatch advertised one tunable and reached for a second dependency through config.SCMRegistry, a mutable package-level global. Add WithSCMFactory next to WithHeartbeat so the factory is a named part of the handler's configuration, defaulting to the registry lookup the handler used to make inline. Production wiring is unchanged, handler.BuildWatch() with no options, and only tests pass a factory. That takes the handler's tests off the global. build_test.go no longer swaps config.SCMRegistry from a spec goroutine while a request goroutine reads it, which was safe only because Ginkgo happened to run the restore after httptest.Server.Close had drained in-flight requests. Nothing enforced that ordering. Its local withSCMStub is replaced by buildWatchWithStub, which builds a handler backed by a stub and mutates nothing. The other five handlers still read the registry directly; converting them means putting the factory on Handlers and touching every spec that calls withSCMStub, so it is left for a follow-up. They also keep covering the DefaultPlatform lookup, which no BuildWatch spec exercises any more. Signed-off-by: Matej Vašek <matejvasek@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ETag cache added earlier on this branch only helps when it actually stores a response, and httpcache stores one only once its body reaches EOF. go-github decodes with json.Decoder, which stops as soon as the top-level value is complete and need never read that far, so whether a response lands in the cache depends on how the body is framed. That makes it a size lottery rather than a threshold. Measured against this client, 8 KB and 16 KB bodies were cached while 14 KB and 100 KB were not, and the payload only has to grow by a few bytes to flip. A lost poll costs a full 200 against the primary rate limit instead of a free 304. Production against api.github.com is unaffected, because gzip drains to EOF, so the exposure is GitHub Enterprise Server, the fake used in dev, or anything that strips Content-Encoding in between. A workflow_runs entry embeds whole repository objects and runs 10-20 KB, right in the range where this bites. The fix wraps each response body so Close drains whatever the caller left unread. It lives in the transport we already wrap for forced revalidation, because upstream cannot be fixed: gregjones/httpcache#104 is an open fix for this from 2020 and the repository was archived in 2023. The existing revalidation spec passed for the wrong reason. Its fixture was a few dozen bytes, a size that happens to cache either way, so it never exercised the failure. It now runs at three sizes to state the property that matters, that caching must not depend on payload size, and two of the three fail without this change. Signed-off-by: Matej Vašek <matejvasek@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
/test all |
|
/test e2e-aws |
twoGiants
left a comment
There was a problem hiding this comment.
Now I understood useBuildStatus. Lets separate the concerns there in a clean way and then we're good. You're very close! Great job on the low level implementaton and great job on putting everything close together -> the extraction will be simple because you prepared it so well. 👍
I'll finish the tests next time I review.
| function toMap(snap: BuildSnapshot): ReadonlyMap<string, BuildStatus> { | ||
| return new Map( | ||
| Object.entries(snap.functions ?? {}).map(([key, f]) => [ | ||
| key, | ||
| { | ||
| buildStatus: f.buildStatus, | ||
| conclusion: f.conclusion, | ||
| runURL: f.runURL, | ||
| }, | ||
| ]), | ||
| ); | ||
| } |
There was a problem hiding this comment.
This is not needed, use Record<string, BuildStatus>. The BuildSnapshot already comes in the shape we can work with it. In FunctionListPage.tsx you use const build = buildStatuses.get(${item.owner}/${item.repoName}); but this can be done with a record too -> buildStatuses[key].
Then you use setStatuses(snap.functions) directly.
| async function readStream( | ||
| body: ReadableStream<Uint8Array>, | ||
| onSnapshot: (snap: BuildSnapshot) => void, | ||
| ): Promise<void> { | ||
| const reader = body.getReader(); | ||
| const decoder = new TextDecoder(); | ||
| let buffer = ''; | ||
| for (;;) { | ||
| const { done, value } = await reader.read(); | ||
| if (done) return; | ||
| buffer += decoder.decode(value, { stream: true }); | ||
| let idx: number; | ||
| while ((idx = buffer.indexOf('\n\n')) !== -1) { | ||
| const frame = buffer.slice(0, idx); | ||
| buffer = buffer.slice(idx + 2); | ||
| const snap = parseFrame(frame); | ||
| if (snap) onSnapshot(snap); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| function parseFrame(frame: string): BuildSnapshot | null { | ||
| let event = ''; | ||
| const dataLines: string[] = []; | ||
| for (const line of frame.split('\n')) { | ||
| if (line.startsWith(':')) continue; // heartbeat / comment | ||
| if (line.startsWith('event:')) event = line.slice('event:'.length).trim(); | ||
| else if (line.startsWith('data:')) dataLines.push(line.slice('data:'.length).trim()); | ||
| } | ||
| if (event !== 'build-status') return null; | ||
| if (dataLines.length === 0) return null; | ||
| try { | ||
| return JSON.parse(dataLines.join('\n')) as BuildSnapshot; | ||
| } catch { | ||
| return null; | ||
| } | ||
| } |
There was a problem hiding this comment.
Now after a good night sleep I understand this much better. These two functions plus the consoleFetch are the low level implementation of the SSE receiving. If we keep SSE, we might reuse it in more client hooks, but no need to extract now, just refactor properly. (Or extract if you feel like it, it'll come anyway.)
Ideally this entire part would live in functionsClient.ts as an async function* receiveFunctionsBuildStatus() which after the useCluster to backend migration probably will become a receiveFunctionsStatus or even receiveFunctions. In functionsClient.ts is also the scmHeaders function which reads the pat. The receiving and deserialization bits would be unexported functions in that module, like now in this hook module.
There are three main things happening for the receiving implementation:
- Setting up the stream via consoleFetch.
- Receiving incoming data in chunks.
- Deserializing.
I think they should all live in one generic generator function and be provided from functionsClient.ts via receiveFunctionsBuildStatus and consumed by useBuildStatus hook in the useEffect block. The deserialization should be generic, I wouldn't couple it to BuildSnapshot. I'd also pass in the event type as an argument.
// in functionsClient.ts
export async function* receiveFunctionsBuildStatus(
abortSignal: AbortSignal,
): AsyncGenerator<BuildSnapshot> {
yield* receiveSSE<BuildSnapshot>(
`${PROXY_BASE}/api/v1/func/build/watch`,
'build-status',
abortSignal,
);
}
async function* receiveSSE<T>(
url: string,
eventName: string,
abortSignal: AbortSignal,
): AsyncGenerator<T> {
const res = await consoleFetch(url, { headers: scmHeaders(), signal: abortSignal }, 0);
if (!res.body) return;
const decoder = new TextDecoder();
let buffer = '';
for await (const chunk of res.body) {
buffer += decoder.decode(chunk, { stream: true });
let idx: number;
while ((idx = buffer.indexOf('\n\n')) !== -1) {
const raw = buffer.slice(0, idx);
buffer = buffer.slice(idx + 2);
const parsed = deserializeSSEFrame(raw);
if (parsed) yield parsed;
}
}
function deserializeSSEFrame(frame: string): T | null {
let event = '';
const dataLines: string[] = [];
for (const line of frame.split('\n')) {
if (line.startsWith(':')) continue;
if (line.startsWith('event:')) event = line.slice('event:'.length).trim();
else if (line.startsWith('data:')) dataLines.push(line.slice('data:'.length).trim());
}
if (event !== eventName) return null;
if (dataLines.length === 0) return null;
try {
return JSON.parse(dataLines.join('\n')) as T;
} catch {
return null;
}
}
}The try block in useBuildStatus would be reduced to:
useEffect(() => {
let cancelled = false;
const controller = new AbortController();
async function run() {
while (!cancelled) {
const pat = sessionStorage.getItem(PAT_KEY);
if (!pat) return;
try {
for await (const snap of receiveFunctionsBuildStatus(controller.signal)) {
if (!cancelled) setStatuses(snap.functions);
}
} catch (err) {
if (cancelled) return;
//...…urce cleanup Introduce a Kubernetes-style watch pattern with explicit Stop() method for resource cleanup. The BuildWatch handler now calls defer watch.Stop() to ensure the polling goroutine halts when the request exits, preventing resource leaks. Handle errors emitted from the watch channel to gracefully close the stream on watch errors. Signed-off-by: Matej Vašek <matejvasek@gmail.com>
|
/test e2e-aws |
|
@matejvasek: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Summary
/api/v1/func/build/watch), user-scoped like/listscm.Client.WatchWorkflowRunspolls GitHub Actions behind one channel per connection, scoped tofunc-deploy.yaml, ETag-cached to keep 304s free against the rate limituseBuildStatus, list merge); Playwright e2e against the real backend and fakegithubdocs/design/2026-08-26-SRVOCF-1038-build-status-design.md, implementation plan indocs/plans/completed/2026-08-26-SRVOCF-1038-build-status.mdFixes SRVOCF-1038
How the build status stream works
The backend polls GitHub for workflow runs and the browser subscribes to the result. Server-Sent Events (SSE) is a long-lived HTTP response of
Content-Type: text/event-streamthat the server keeps open and appends text frames to. It is one-way (server to client), which is all we need here.Why polling and not webhooks. GitHub can push
workflow_runevents to a webhook, and that would be lower latency, but it is a much bigger system: a publicly reachable route into the cluster, a webhook plus signing secret registered and kept in sync on every function repo, signature verification, and server-side state to fan each event out to the right browser session. Polling needs none of that. It runs inside the existing user-scoped request, holds no state beyond the life of the connection, and unchanged polls are 304s, so the steady-state cost is close to zero. The push we do need, backend to browser, is the one SSE gives us.Wire format. Frames are separated by a blank line. A line starting with
:is a comment (our heartbeat, which keeps proxies from closing an idle connection). Everything else the client ignores unless the frame'sevent:isbuild-status:Each frame is a full snapshot, not a delta: the map is keyed by
owner/repoand the client replaces its state wholesale. That makes the client stateless with respect to ordering and missed frames, and it means a reconnect needs no catch-up protocol.Auth failures stay ordinary HTTP. Once a response is a stream you can no longer change its status code, so
HandleBuildWatchdoes repo discovery before writing any headers. A revoked PAT is therefore a plain401, not a half-written stream (backend/handler/build.go:50-59).Change-only emission. The poller compares each new snapshot to the previous one and only sends on the channel when it differs, so an idle list produces nothing but heartbeats (
backend/scm/github/watch.go:42-53). Polls that find nothing new are304 Not Modifiedthanks to an ETag-caching transport, and 304s do not count against GitHub's primary rate limit.Why not
EventSource? The browser's built-in SSE client cannot set request headers, which would force the PAT into the URL (where it lands in logs and history). So the client usesconsoleFetchwithtimeout: 0and readsresponse.bodyas aReadableStream, splitting frames on\n\nitself (src/common/clients/useBuildStatus.ts). The cost is that we implement reconnect by hand: 3s backoff on a dropped stream, and a hard stop on 401/403 since a bad token will not fix itself. Full rationale in the design doc under "Transport decision: SSE over consoleFetch stream".Where it lands in the UI.
useBuildStatusreturns a map thatFunctionsListPagemerges with the cluster status per function. The merge is non-destructive: a function the cluster knows about keeps its cluster badge and the build shows only as a secondary indicator (spinner or red warning icon linking to the run).Suggested review order
The change is easier to follow outside-in rather than by diff order:
docs/design/2026-08-26-SRVOCF-1038-build-status-design.mdfor the what and why, especially the status merge table.backend/scm/github/watch.gofor the polling loop, change detection, and per-repo error carry-forward.backend/handler/build.gofor the SSE framing and the auth-before-headers ordering.src/common/clients/useBuildStatus.tsfor the client-side frame parsing and reconnect.src/pages/function-list/FunctionsListPage.tsx(mergeBuild) andcomponents/FunctionTable.tsx(StatusCell) for how the two statuses combine and render.e2e/use-cases/list/build-status.test.tsis the end-to-end story in one file.SSE resources
EventSource(the API we deliberately do not use, see above)response.bodygives us instead)http.Flusher(why every frame is followed by aFlush())proxy_buffering/X-Accel-Buffering(why the handler sets that header)Checklist
docs/ARCHITECTURE.md(if there are relevant changes to our layered architecture)Additional Info
Deployingfor a moment mid-rollout and gating onRunning/ScaledToZeromade every redeploy flicker throughBuilding.🤖 Generated with Claude Code