Repository navigation
feat(dgw): run AI analysis of a recording as a provisioner task - #2044
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 34 commits into
Conversation
Let maintainers know that an action is required on their side
|
c81d3f1 to
5b16e3a
Compare
5b16e3a to
faf9419
Compare
faf9419 to
55fef54
Compare
55fef54 to
5ff453b
Compare
5ff453b to
f2948c9
Compare
f2948c9 to
633c904
Compare
633c904 to
6083262
Compare
6083262 to
ed522eb
Compare
`as_of_v2022_3_0_0` lists what DVLS 2022.3 sends; later scopes such as `gateway.recording.delete` were never added there. The AI analysis scopes are covered by the `ai_analysis` tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`AiSettings` and `AiAccess` (and so `StartRequest`) derived `Debug`, which prints the base URL that must never reach the logs. Nothing logs them today; this keeps it that way. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The server only ever sends 0 to 100, as documented. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failure to list or hand over the unfinished Tasks stopped the whole service. Unfinished Tasks still fail at their deadline, so the error is logged and startup goes on; one Task that cannot get a new job no longer stops the others. Also follows the provisioner task store changes below this PR: `update_running` takes `now` and `job_token` is no longer optional. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A newer DVLS may send a new optional field to an older Gateway. With deny_unknown_fields the older Gateway refused the whole request with 400, so the analysis never ran. Every other Gateway endpoint ignores unknown fields; this one now does too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…deadline An attempt started near the 2-hour deadline could keep calling the AI for 30 more minutes, add its log to the recording, and still leave the Task failed as timed out. Attempts now start only during the first 2 hours, and the deadline is one attempt timeout after that, so an attempt started in time can finish and its result counts. The runner fails a Task that has no time left for a full attempt as timed out, without running it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ress `percent` is `done / total`, and `message` is English that clients translate anyway, so neither belongs in the API. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ads it A recording is remuxed 60 seconds after it ends, and the remux replaces the video file. An analysis started right after the session ended read the video before it was remuxed. The analysis now waits until the job queue stores no remux job for a file of the session, checking every second, before it reads the recordings. The job queue lists remux jobs that are scheduled or running, including those stored before a Gateway restart. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- The working files of a session were reused by any later Task with the same model, and the late job of a Task that ended without it, such as one failed past its deadline on a status read, could delete the files of a newer Task, which then paid for every chunk again. The analysis now names its Task as the owner of the files: the attempts of one Task resume from them, a new Task starts fresh, and a Task's cleanup leaves another Task's files in place. - An attempt starts only with one minute to spare past its timeout: it ends a little later while it stops, and stored deadlines lose their fraction of a second. This also keeps the runner test clear of rounding. - While describing, progress is written whenever another chunk is done, not when the percent changes, since the payload no longer has a percent. - `deadlineAt` says a Task can time out earlier, and the failure reasons are listed in full. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Analysing a recording again kept the earlier results: each success added another `ai-analysis-N.slog` to the manifest, so one recording could hold several analyses. A new Task is now a hard restart. Its first attempt removes every earlier AI analysis result of the session, the file and its manifest entry, even a succeeded one, and the working files whoever left them. Its later attempts are job retries and resume from the working files. A recording holds at most one AI analysis. The recording manager gains `ClearArtifacts`, which drops the manifest entries first, so the manifest never lists a file that is gone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The caller of `POST /jet/jrec/{session_id}/ai-analysis` now picks the
language in an optional `language` field: any text of 1 to 10
characters, such as `fr-CA` or `français`; anything else is a 400.
Without it, the analysis is in English. The language is part of the
Task parameters, so the same Task ID with another language is a
`task_id_conflict`.
Gateway no longer reads anything the producer pushed: there is no
events log to take a `locale` from.
- The tests count the summary request the AI crate now sends.
- "AI analysis queued" is a debug line, since the runner already logs
the queued Task.
- The jrec and provisioner tasks intents state the language, how many
analyses a session keeps, and how retries and restarts behave.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The working files of an analysis now always name their language, so a fixture without one is not a valid workspace. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Clearing on the first attempt only missed two paths. A Task that failed before its first attempt cleared, such as on a lost key, kept the earlier analysis. A first attempt cancelled while waiting for the session retried as attempt 2 and never cleared, so the session ended with two analyses. Each attempt now takes the session lock, then clears, before anything else can fail. The Task params keep the language as a `Language`, so it's checked once, when the request comes in. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…document `params` of a `recording-ai-analysis` Task now has the `AiAnalysisParams` schema, like its payloads. The lint attributes on these schema structs are gone, since the lint never fires on them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The tests move from the crate's own `tests/` to the testsuite, as AGENTS.md asks, and talk to a running Gateway and a mock AI provider instead of linking Gateway internals. They read the key leak checks from the log files and `gateway.db` in the config folder. The check that a Task of another kind is not found at this endpoint is gone: the binary only knows one kind. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…st does The full path made the OpenAPI document reference a schema that does not exist. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…a failed removal A session without recording has no AI analysis to remove, so clearing it succeeds and the attempt then reports "session has no recording". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Windows can't delete the database and log files a running Gateway holds open, so each test left its folder behind. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…er accepts The server reads a `u32`, which an `int32` schema can't hold, so the clients now get a 64-bit integer from 0 to 4294967295. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The AI access of a Task now lives in the standalone `SecretStore`, keyed by the Task ID, instead of the provisioning store, which is about sessions only. Gateway holds one store in `DgwState`, hands it to the provisioner tasks, and runs its expiry cleanup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Task secrets live in the standalone `SecretStore`, not in the provisioning store, which is about sessions only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Intent drafts: handoffRead at Provenance
Left out on purpose
Placement
Pending experimentsNone. Note Human-tuned, LLM-assisted content. |
AI analysis of a recording, started by DVLS with a plain HTTP request (no new token type), tracked as a provisioner Task and run as a job on the existing job queue.
Provisioner tasks run through a
ProvisionerTaskRunner, the one place that keeps a task's record (#2041'sProvisionerTaskRecord) and its job in step. Each kind of work is aProvisionerTask(what one attempt does, what to drop once its record ends); the runner creates the record, queues its job, runs each attempt with its timeout, retries transient failures and records how the Task ends. The Task is the source of truth and the job only wakes it up: each Task names the one job allowed to run it (#2041), so a stale or duplicate job does nothing. A Task is reported created only once its job is stored in the job queue. At startup, before the job runner claims anything, an unfinished Task whose job is no longer stored (such as after a crash between recording the Task and storing its job) gets a new one; a Task whose job is still stored keeps it. The analysis itself is the AI crate'sanalyze_recording(#2043); Gateway hands it the manifest of the finished recording and the language of the request, and adds the log it returns to the session as its own file.API (stable, no
enable_unstable)POST /jet/jrec/{session_id}/ai-analysis, scopegateway.tasks.recording-ai-analysis.start. Body:taskId,provider,model,baseUrl(needed for openai-compatible),maxOutputTokens,language,apiKey.task_id_conflict/recording_active(a clip is still pushed) /analysis_in_progress(+ the runningtaskId)GET /jet/jrec/{session_id}/ai-analysis/{task_id}, scopegateway.tasks.recording-ai-analysis.read: the Task (state, payload, attempts, times), or 404. The 202 answer of thePOSTnames it in itsLocationheader. A Task of another kind, or the analysis of another session, is not found under that path.Each provisioner task declares its own scopes,
gateway.tasks.<kind>.startandgateway.tasks.<kind>.read, so a token for one kind of Task can neither start nor read another kind. The kind of AI analysis isrecording-ai-analysis.The two new scopes in
Devolutions.Gateway.Utilsand the regenerated OpenAPI clients are in #2054, stacked on this PR.The first commit hands the provisioner task store (#2041) to
DgwStateon its owngateway.dbconnection;DgwState::mockbecomes async for that, hence the.awaitin a few tests. It also passesjob_queue::MAX_ATTEMPTS(5, same as the current default) to the job queue explicitly, so the retry check in the AI analysis job can't drift from it.How it behaves
SecretStore(feat(dgw): add an encrypted in-memory secret store #2058); never in the job,gateway.dbor our logs. After a Gateway restart the Task fails with "the API key is no longer available"..cast,.trp) and video (.webm) recordings are analyzed (feat(dgw): analyze any recording in the AI crate #2043); video needs XMFv2026.9.25.0or later.languageis any text of 1 to 10 characters, such asfr-CAorfrançais; anything else is a 400. Without it the analysis is in English. It's part of the Task parameters, so the sametaskIdwith another language is atask_id_conflict.ai-analysis-0.slogunderartifacts.ai-analysis, with the summary on itssession.end(feat(ai): write the AI analysis in the requested language, with a summary #2118). Gateway reads nothing the producer pushed.paramskeep the AI settings without the key and the base URL, plus the language; the OpenAPI document describes them asAiAnalysisParams.Where to look hardest
provisioner_tasks/runner/mod.rs:run_attempt(cancel on timeout, retry vs. fail,on_endon every end) andresume(called fromservice.rsbefore the job runner starts)provisioner_tasks/recording_ai_analysis/mod.rs: the AIProvisionerTask, andlock_session: the working files belong to the session, so analyses of one session run one at a time (a Task past its deadline may still have an attempt running when the next Task starts)start()in the same file: the order of the checksKnown, accepted
All, reqwest/hyper's own TRACE lines show the provider's address (not the key). Our code never logs it.ai-analysis-0.slog.Tests
cargo test -p devolutions-gateway --lib: 249 passed.cargo test -p testsuite --test integration_tests ai_analysis: 10 passed. These endpoint tests drive the Gateway binary with a mock AI provider, and check that the key and the base URL reach neither the logs norgateway.db. The testsuite gains atimedev-dependency to check the RFC 3339 times.--features openapi --tests) and nightly rustfmt are clean.🤖 Generated with Claude Code