Stop the stale-job reaper from revoking jobs that are created but not yet started

[mihow]

Summary

Jobs can be created without starting them immediately — for example, to pre-configure several jobs and launch them at a later time. A job created this way waits in the CREATED state until someone starts it. A periodic background task revokes jobs that look stale or hung, and it was also revoking these created-but-unstarted jobs, so a pre-configured job would be cancelled before it ever ran. This change leaves CREATED jobs alone, so they wait to be started instead of being reaped.

Closes #1354

List of Changes

Change (user-facing effect)	How (implementation)
A job that is created but not yet started is no longer revoked by the stale-job reaper, regardless of how long it waits.	Removed CREATED from JobState.running_states(), the set the reaper and reconcilers select jobs from.
Added a regression test pinning that a CREATED job past the staleness cutoff is left untouched.	New test_skips_created_but_unstarted_jobs in ami/jobs/tests/test_update_stale_jobs.py.

Detailed Description

The stale-job reaper (check_stale_jobs in ami/jobs/tasks.py) finds jobs that are still in a running state and whose updated_at is older than Job.STALLED_JOBS_MAX_MINUTES (10 minutes), then revokes them. It selected candidates via JobState.running_states(), which included CREATED. A job that was pre-configured and left for later therefore qualified once it was older than the cutoff, and was revoked.

A CREATED job has no Celery task and no async (NATS/Redis) resources, so there is nothing to reap or reconcile. Job.enqueue() flips a job to PENDING the moment it is dispatched, and a job that gets stuck mid-dispatch is already PENDING — so genuinely stuck jobs are still caught. Removing CREATED from running_states() also keeps the lost-images reconciler and the async snapshot check (the other callers of running_states()) from touching jobs that have not started.

finalizable_states() is intentionally left unchanged — it keeps CREATED because a synchronous job can legitimately transition CREATED → SUCCESS through the normal completion path; that is separate from time-based reaping.

The full stale-job test suite passes (7/7).

Out-of-scope observations (left for follow-up)

While sweeping for code that depends on running_states(), I found a few pre-existing places in the frontend that classify the CREATED state independently of the backend. They are not consumers of running_states() (which is Python-only), so this change does not affect them, and they are not modified here. Noting them for a later pass:

• ui/src/data-services/models/capture-set.ts and ui/src/data-services/models/capture-details.ts — hasJobInProgress counts a CREATED job as "in progress". A created-but-unstarted job has not been dispatched yet, so this may overstate activity. It could also be intentional (signalling that a job is already set up for the capture set, to avoid duplicates) — worth a deliberate decision rather than a silent change.
• ui/src/data-services/models/job.ts — canCancel only allows STARTED/PENDING, but the backend's finalizable_states() permits cancelling a CREATED job too. The result is a harmless asymmetry: the cancel button is hidden for CREATED jobs even though the backend would accept the request.

These are flagged only; no behavior change is intended in this PR.

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	a8ac1a9
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6a3f0a5face2b60007e47d54

[netlify]

✅ Deploy Preview for antenna-ssec canceled.

Name	Link
🔨 Latest commit	a8ac1a9
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-ssec/deploys/6a3f0a5f72a68c0008ce09c7

[coderabbitai]

Warning

Review limit reached

@mihow, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 33 minutes and 52 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 323dbe37-b9ce-4f98-ada1-bedd796a59e4

📥 Commits

Reviewing files that changed from the base of the PR and between 08ca0a4 and a8ac1a9.

📒 Files selected for processing (2)

• ami/jobs/models.py
• ami/jobs/tests/test_update_stale_jobs.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/1354-dont-revoke-created-jobs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

This PR updates the job “stale/hung” cleanup logic so that jobs left in CREATED (pre-configured but not enqueued) are not considered “running” and therefore won’t be revoked by periodic reaping tasks. It also adds a regression test to lock in this behavior, but includes an additional migration that appears unrelated to the stated PR scope.

Changes:

• Remove CREATED from JobState.running_states() so reapers/reconcilers don’t select created-but-unstarted jobs.
• Add a regression test asserting a stale CREATED job is left untouched by check_stale_jobs.
• Add a migration that globally enables async_pipeline_workers for all existing projects (appears out of scope vs. PR description/title).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
ami/jobs/models.py	Updates the definition of “running” job states so CREATED jobs are excluded from reaping/reconciliation selection.
ami/jobs/tests/test_update_stale_jobs.py	Adds regression coverage ensuring CREATED-but-unstarted jobs are not reaped.
ami/main/migrations/0094_enable_async_pipeline_workers.py	Introduces a global feature-flag rollout migration for async pipeline workers (scope discrepancy with PR metadata).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mihow]

Claude says: Retracted — this comment was wrong. Please disregard it.

I originally argued that a production incident on 2026-07-29 was a counter-case to this PR's rationale, on the grounds that jobs sitting in PENDING with no progress were reaped despite being healthy. That premise is false.

MLJob.run() sets job.update_status(JobState.STARTED) as its first statement, before any images are queued (ami/jobs/models.py:471). The jobs in that incident were STARTED well before they were reaped — one within about twelve seconds of creation, the other already mid-processing with tens of thousands of images tracked. PENDING is only the brief window between enqueue and a worker picking up run_job, and the queue serving that had healthy consumers throughout. The incident never entered the window this PR's reasoning is about.

I believe the error came from conflating a stage status of CREATED 0% with the job status. They are different fields.

So this PR's rationale for stopping at CREATED and leaving PENDING reapable is untouched by that incident, and the merge was correct. Apologies for the noise on a merged PR.

The point I was reaching for — that a no-progress timer cannot distinguish "this job is stuck" from "this job is fine and something downstream of it has failed" — is real, but it belongs to the reaper's predicate generally and not to this change. It is being tracked in #1383.