fix(zetesis,paroche,archon): credentialed enqueue-by-reference with a results cache#614
Merged
Merged
Conversation
… results cache
Search redacts indexer credentials from download_url on every response
(correct — credentials must not reach clients), but there was no
server-side path to resolve the real URL at enqueue time, so a
credentialed Torznab/Newznab hit could not be submitted back to
POST /api/v1/downloads. Magnets were unaffected. Search results were
also fully ephemeral: SearchIndexerService::search minted a QueryId,
fanned out, deduped, and returned a bare Vec<SearchResult> — nothing
was stored, and SearchResult carried no stable server key. The
existing `releases` table was unusable for this (want_id NOT NULL FK,
zero production callers of insert_release).
The cache (crates/zetesis/src/results_cache.rs): a hand-rolled
ResultsCache(std::sync::Mutex<CacheInner>) — never locked across an
await, matching the crate's existing swap-behind-std-lock discipline.
CacheInner tracks insertion order (also age order), a QueryId -> deduped
results map, and a release Uuid -> (QueryId, index) map. search() mints
one themelion::ReleaseId per deduped result and inserts the whole query
under its QueryId; insert purges expired queries and evicts the oldest
beyond result_cache_max_queries (dropping the evicted query's release
entries with it). Lookups are lazy-TTL: a stale entry reads as absent
without being physically swept until the next insert. No new
dependencies, no DB — credentials never leave process memory or touch
disk.
New types (crates/zetesis/src/types.rs): CataloguedResult (a
SearchResult plus its release_id, Serialize, flattened on the wire) and
SearchOutcome (a QueryId plus its CataloguedResults) are what
SearchIndexerService::search now returns instead of a bare Vec —
additive JSON, no client breakage. ResolvedRelease (download_url,
protocol, info_hash, indexer_id) is deliberately NOT Serialize: a
compile error is the guard against ever placing a credentialed URL on
an HTTP response path. Its Debug redacts download_url; SearchOutcome
and CataloguedResult both Deref for ergonomic Vec/struct-like access at
existing call sites.
SearchIndexerService gained cached_results(query_id) and
resolve_release(release_id) — both idempotent and non-consuming, so a
retry after a failed enqueue still resolves.
The enqueue resolve branch (crates/paroche/src/routes/download.rs):
EnqueueRequest.download_url is now Option<String>. A present non-empty
value takes today's path verbatim (client-supplied protocol/info_hash);
an absent value resolves the release server-side via
state.search.resolve_release(release_id) and uses the RESOLVED
download_url, protocol, and info_hash (falling back to the body's
info_hash only when the resolved release carries none) — the resolved
protocol always wins over the body's default. A present-but-empty or
whitespace-only download_url still 422s, unchanged. A resolve miss maps
ServiceError::NotFound -> 404 via the existing error.rs conversion.
DynSearchService gained cached_results/resolve_release (paroche::state
carries its own decoupled ResolvedRelease mirror, same pattern as
EnqueueItem/DynQueueManager staying decoupled from syntaxis types);
archon's SearchAdapter implements both against the live zetesis
service. NullSearch and every test stub were updated to match.
Security invariant: an unredacted download_url crosses the paroche HTTP
boundary only in REQUESTS, never in responses or logs. The only
unredacted egress from the cache is resolve_release, consumed
exclusively inside enqueue_download, and its type cannot be
accidentally serialized. net_validate::validate_download_url runs on
the RESOLVED url exactly as it does on a client-supplied one — by-
reference enqueue must not become an SSRF bypass (covered by
enqueue_download_rejects_resolved_url_to_private_space). Every route
test that touches a credentialed URL asserts the response body never
contains the literal secret.
Also wired the dead GET /api/v1/search/{query_id}/results route
(previously always 404 via a hack in search_query_from_json, whose only
caller was this same dead route — deleted along with its now-obsolete
test): it parses the path UUID (InvalidId -> 400), calls
cached_results, 404s on a miss, and keeps the existing credential-
redaction walk over the response byte-for-byte, since cached results
carry unredacted URLs and this endpoint is member-visible.
horismos: SearchSubsystemConfig gained result_cache_ttl_seconds
(default 1800) and result_cache_max_queries (default 32), both LIVE —
read straight from the Section on every cache operation, no rebuild.
Classified in diff.rs's LIVE list and documented in
config-reload.md/configuration.md.
Tests: zetesis unit coverage for the cache (insert/resolve round-trip,
cached_results parity, TTL=0 immediate miss, cap eviction including
by_release cleanup, ResolvedRelease Debug redaction) plus an end-to-end
test through a mock credentialed Torznab indexer proving search() ->
resolve_release returns the exact credentialed URL and cached_results
agrees. paroche route tests cover the done-when scenario (admin
enqueues by release_id with no download_url, the queue receives the
credentialed URL, the HTTP response is redacted), resolve miss (404,
nothing enqueued), SSRF-on-resolved (private-space resolved URL is
422, nothing enqueued), resolved-protocol-wins, and the wired GET route
(redacted hit, 404 miss, 400 on a garbage id).
Closes #608
Gate-Passed: kanon 0.1.6 +stages:fmt,check,clippy,nextest,lint sha:79fd0df24cc47d8b883611d01dcd7a1f9cf00943
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #608. A private-tracker search hit whose
download_urlembeds credentials can now be enqueued through the API without the client ever seeing the credential.Search results were ephemeral (dropped on response), so a redacted
download_urlhad no server-side path back to the real URL. This adds an in-memory TTL results cache in zetesis (bounded, no new deps, credentials never on disk): each result gets a server-mintedrelease_id;POST /api/v1/downloadsacceptsrelease_idwith nodownload_urland resolves the unredacted URL server-side (SSRF-validated on the resolved URL too — by-reference is not an SSRF bypass). Rawdownload_urlinput still works for magnets/manual URLs. The previously-deadGET /api/v1/search/{query_id}/resultsroute is wired (redacted).Security invariant (compile-enforced + tested): the unredacted URL crosses the HTTP boundary only in requests —
ResolvedReleasehas noSerializederive (a leak is a compile error), every response path stays redacted, and route tests assert the credential never appears in a response body. Gate green (full workspace); 3-lens adversarial security review clean (0 findings).