Skip to content

feat(dgw): run AI analysis of a recording as a provisioner task - #2044

Open
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 34 commits into
feat/ai-session-language-summaryfrom
feat/recording-ai-analysis
Open

irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 34 commits into
feat/ai-session-language-summaryfrom
feat/recording-ai-analysis

Conversation

@irvingoujAtDevolution

@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

AI analysis of a recording, started by DVLS with a plain HTTP request (no new token type), tracked as a provisioner Task and run as a job on the existing job queue.

Provisioner tasks run through a ProvisionerTaskRunner, the one place that keeps a task's record (#2041's ProvisionerTaskRecord) and its job in step. Each kind of work is a ProvisionerTask (what one attempt does, what to drop once its record ends); the runner creates the record, queues its job, runs each attempt with its timeout, retries transient failures and records how the Task ends. The Task is the source of truth and the job only wakes it up: each Task names the one job allowed to run it (#2041), so a stale or duplicate job does nothing. A Task is reported created only once its job is stored in the job queue. At startup, before the job runner claims anything, an unfinished Task whose job is no longer stored (such as after a crash between recording the Task and storing its job) gets a new one; a Task whose job is still stored keeps it. The analysis itself is the AI crate's analyze_recording (#2043); Gateway hands it the manifest of the finished recording and the language of the request, and adds the log it returns to the session as its own file.

API (stable, no enable_unstable)

  • POST /jet/jrec/{session_id}/ai-analysis, scope gateway.tasks.recording-ai-analysis.start. Body: taskId, provider, model, baseUrl (needed for openai-compatible), maxOutputTokens, language, apiKey.
    • 202 + Task when created, 200 + Task for the same request again
    • 400 bad settings, body or language, 404 no recording
    • 409 task_id_conflict / recording_active (a clip is still pushed) / analysis_in_progress (+ the running taskId)
  • GET /jet/jrec/{session_id}/ai-analysis/{task_id}, scope gateway.tasks.recording-ai-analysis.read: the Task (state, payload, attempts, times), or 404. The 202 answer of the POST names it in its Location header. A Task of another kind, or the analysis of another session, is not found under that path.

Each provisioner task declares its own scopes, gateway.tasks.<kind>.start and gateway.tasks.<kind>.read, so a token for one kind of Task can neither start nor read another kind. The kind of AI analysis is recording-ai-analysis.

The two new scopes in Devolutions.Gateway.Utils and the regenerated OpenAPI clients are in #2054, stacked on this PR.

The first commit hands the provisioner task store (#2041) to DgwState on its own gateway.db connection; DgwState::mock becomes async for that, hence the .await in a few tests. It also passes job_queue::MAX_ATTEMPTS (5, same as the current default) to the job queue explicitly, so the retry check in the AI analysis job can't drift from it.

How it behaves

  • The API key and base URL live in memory only, encrypted, as the Task's secret in the SecretStore (feat(dgw): add an encrypted in-memory secret store #2058); never in the job, gateway.db or our logs. After a Gateway restart the Task fails with "the API key is no longer available".
  • Transient provider errors are retried by the job queue (up to 5 attempts); permanent ones fail the Task. 30 min per attempt, 2 h per Task.
  • Terminal (.cast, .trp) and video (.webm) recordings are analyzed (feat(dgw): analyze any recording in the AI crate #2043); video needs XMF v2026.9.25.0 or later.
  • language is any text of 1 to 10 characters, such as fr-CA or français; anything else is a 400. Without it the analysis is in English. It's part of the Task parameters, so the same taskId with another language is a task_id_conflict.
  • The result is its own log, ai-analysis-0.slog under artifacts.ai-analysis, with the summary on its session.end (feat(ai): write the AI analysis in the requested language, with a summary #2118). Gateway reads nothing the producer pushed.
  • Every attempt of a new Task first takes the session lock and removes the earlier AI analysis, even a succeeded one. So a session holds at most one, and a Task that fails, even on a lost key, leaves none.
  • The Task params keep the AI settings without the key and the base URL, plus the language; the OpenAPI document describes them as AiAnalysisParams.
  • The Task payload names the log, the model, the prompt versions and the token usage.

Where to look hardest

  • provisioner_tasks/runner/mod.rs: run_attempt (cancel on timeout, retry vs. fail, on_end on every end) and resume (called from service.rs before the job runner starts)
  • provisioner_tasks/recording_ai_analysis/mod.rs: the AI ProvisionerTask, and lock_session: the working files belong to the session, so analyses of one session run one at a time (a Task past its deadline may still have an attempt running when the next Task starts)
  • start() in the same file: the order of the checks

Known, accepted

  • At VerbosityProfile All, reqwest/hyper's own TRACE lines show the provider's address (not the key). Our code never logs it.
  • If Gateway crashes right after adding the log to the manifest, and before moving it in, the manifest lists a missing ai-analysis-0.slog.
  • A Task that fails before any attempt runs (its job can't be queued, or no time is left for one) keeps the earlier analysis, since only an attempt holds the session lock.

Tests

  • cargo test -p devolutions-gateway --lib: 249 passed.
  • cargo test -p testsuite --test integration_tests ai_analysis: 10 passed. These endpoint tests drive the Gateway binary with a mock AI provider, and check that the key and the base URL reach neither the logs nor gateway.db. The testsuite gains a time dev-dependency to check the RFC 3339 times.
  • clippy (--features openapi --tests) and nightly rustfmt are clean.
  • Not run against a real AI provider at this head.

🤖 Generated with Claude Code

@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) changed the title feat/recording ai analysis feat(dgw): run AI analysis of a recording as a provisioner task Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Let maintainers know that an action is required on their side

  • Add the label release-required Please cut a new release (Devolutions Gateway, Devolutions Agent, Jetsocat, PowerShell module) when you request a maintainer to cut a new release (Devolutions Gateway, Devolutions Agent, Jetsocat, PowerShell module)

  • Add the label release-blocker Follow-up is required before cutting a new release if a follow-up is required before cutting a new release

  • Add the label publish-required Please publish libraries (`Devolutions.Gateway.Utils`, OpenAPI clients, etc) when you request a maintainer to publish libraries (Devolutions.Gateway.Utils, OpenAPI clients, etc.)

  • Add the label publish-blocker Follow-up is required before publishing libraries if a follow-up is required before publishing libraries

@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2045 October 6, 2026 21:26
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2046 October 7, 2026 20:36
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2049 October 7, 2026 22:19
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) removed this pull request from stack #2051 October 7, 2026 22:20
`as_of_v2022_3_0_0` lists what DVLS 2022.3 sends; later scopes such as
`gateway.recording.delete` were never added there. The AI analysis scopes
are covered by the `ai_analysis` tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`AiSettings` and `AiAccess` (and so `StartRequest`) derived `Debug`, which
prints the base URL that must never reach the logs. Nothing logs them today;
this keeps it that way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The server only ever sends 0 to 100, as documented.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failure to list or hand over the unfinished Tasks stopped the whole
service. Unfinished Tasks still fail at their deadline, so the error is
logged and startup goes on; one Task that cannot get a new job no longer
stops the others.

Also follows the provisioner task store changes below this PR:
`update_running` takes `now` and `job_token` is no longer optional.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A newer DVLS may send a new optional field to an older Gateway. With
deny_unknown_fields the older Gateway refused the whole request with 400,
so the analysis never ran. Every other Gateway endpoint ignores unknown
fields; this one now does too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…deadline

An attempt started near the 2-hour deadline could keep calling the AI for
30 more minutes, add its log to the recording, and still leave the Task
failed as timed out.

Attempts now start only during the first 2 hours, and the deadline is one
attempt timeout after that, so an attempt started in time can finish and
its result counts. The runner fails a Task that has no time left for a
full attempt as timed out, without running it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ress

`percent` is `done / total`, and `message` is English that clients
translate anyway, so neither belongs in the API.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ads it

A recording is remuxed 60 seconds after it ends, and the remux replaces
the video file. An analysis started right after the session ended read
the video before it was remuxed.

The analysis now waits until the job queue stores no remux job for a file
of the session, checking every second, before it reads the recordings.
The job queue lists remux jobs that are scheduled or running, including
those stored before a Gateway restart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- The working files of a session were reused by any later Task with the
  same model, and the late job of a Task that ended without it, such as one
  failed past its deadline on a status read, could delete the files of a
  newer Task, which then paid for every chunk again. The analysis now names
  its Task as the owner of the files: the attempts of one Task resume from
  them, a new Task starts fresh, and a Task's cleanup leaves another Task's
  files in place.
- An attempt starts only with one minute to spare past its timeout: it ends a
  little later while it stops, and stored deadlines lose their fraction of a
  second. This also keeps the runner test clear of rounding.
- While describing, progress is written whenever another chunk is done, not
  when the percent changes, since the payload no longer has a percent.
- `deadlineAt` says a Task can time out earlier, and the failure reasons are
  listed in full.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Analysing a recording again kept the earlier results: each success added
another `ai-analysis-N.slog` to the manifest, so one recording could hold
several analyses.

A new Task is now a hard restart. Its first attempt removes every earlier
AI analysis result of the session, the file and its manifest entry, even a
succeeded one, and the working files whoever left them. Its later attempts
are job retries and resume from the working files. A recording holds at
most one AI analysis.

The recording manager gains `ClearArtifacts`, which drops the manifest
entries first, so the manifest never lists a file that is gone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The caller of `POST /jet/jrec/{session_id}/ai-analysis` now picks the
language in an optional `language` field: any text of 1 to 10
characters, such as `fr-CA` or `français`; anything else is a 400.
Without it, the analysis is in English. The language is part of the
Task parameters, so the same Task ID with another language is a
`task_id_conflict`.

Gateway no longer reads anything the producer pushed: there is no
events log to take a `locale` from.

- The tests count the summary request the AI crate now sends.
- "AI analysis queued" is a debug line, since the runner already logs
  the queued Task.
- The jrec and provisioner tasks intents state the language, how many
  analyses a session keeps, and how retries and restarts behave.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The working files of an analysis now always name their language, so a fixture without one is not a valid workspace.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Clearing on the first attempt only missed two paths. A Task that failed before its first attempt cleared, such as on a
lost key, kept the earlier analysis. A first attempt cancelled while waiting for the session retried as attempt 2 and
never cleared, so the session ended with two analyses. Each attempt now takes the session lock, then clears, before
anything else can fail.

The Task params keep the language as a `Language`, so it's checked once, when the request comes in.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…document

`params` of a `recording-ai-analysis` Task now has the `AiAnalysisParams` schema, like its payloads. The lint attributes
on these schema structs are gone, since the lint never fires on them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The tests move from the crate's own `tests/` to the testsuite, as AGENTS.md asks, and talk to a running Gateway and a
mock AI provider instead of linking Gateway internals. They read the key leak checks from the log files and
`gateway.db` in the config folder.

The check that a Task of another kind is not found at this endpoint is gone: the binary only knows one kind.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…st does

The full path made the OpenAPI document reference a schema that does not exist.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…a failed removal

A session without recording has no AI analysis to remove, so clearing it succeeds and the attempt then reports
"session has no recording".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Windows can't delete the database and log files a running Gateway holds open, so each test left its folder behind.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…er accepts

The server reads a `u32`, which an `int32` schema can't hold, so the clients now get a 64-bit integer from 0 to
4294967295.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The AI access of a Task now lives in the standalone `SecretStore`,
keyed by the Task ID, instead of the provisioning store, which is
about sessions only. Gateway holds one store in `DgwState`, hands it
to the provisioner tasks, and runs its expiry cleanup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Task secrets live in the standalone `SecretStore`, not in the
provisioning store, which is about sessions only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@irvingoujAtDevolution

Copy link
Copy Markdown
Contributor Author

Intent drafts: handoff

Read at a10bbd6b0 (feat/recording-ai-analysis), written in b1318c855.

Provenance

# Commitment as written Human's words (if tidied) Level
1 They are kept encrypted in memory, in the SecretStore, until the Task ends or reaches its deadline. Chose to name the standalone store, then adopted the candidate sentence with "good". 3

Left out on purpose

  • Lines 23 and 25 of the Secrets section (never on disk; gone after a restart) still match the code, so they were not reopened.
  • Who holds the store (DgwState.secrets) is wiring, recorded in DECISIONS.md, not intent.

Placement

devolutions-gateway/src/provisioner_tasks/INTENT.md, section "Secrets", line 24, replacing "in the provisioning store".

Pending experiments

None.

Note

Human-tuned, LLM-assisted content.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants