schedule leaf searches by (priority, remaining_splits) lexicographically

[Abdul-Andha]

Description

• Add priority to SearchRequest
• LeafPermitRequest was ordered by remaining_splits. Now it is ordered by (priority, remaining_splits)
  • This lets a query with more splits and a higher priority (smaller int is higher priority) skip the line against a query with less splits and a lower priority

How was this PR tested?

• Unit tests
• test_search_permit_priority_precedes_remaining_splits

[chatgpt-codex-connector]

💡 Codex Review

https://github.com/quickwit-oss/quickwit/blob/8305498a4764978cc4576db53ae0cefc4c5624bd/quickwit-proto/src/codegen/quickwit/quickwit.search.rs#L213-L215
Exclude scheduling priority from the leaf cache key

Because SearchRequest derives Hash/Eq and LeafSearchCache stores the entire request in its CacheKey (leaf_cache.rs:84-114), this new field makes otherwise identical searches with different priorities occupy separate cache entries. Priority affects only scheduling, not search results, so workloads issuing the same query at different priorities lose valid cache hits, increase cache churn, and can make high-priority requests perform unnecessary split searches; normalize priority when constructing the cache key.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

quickwit/quickwit/quickwit-search/src/search_permit_provider.rs

Lines 234 to 240 in cf7ac28

	other.priority.cmp(&self.priority).then_with(|| {
	other
	.single_split_permit_requests
	.as_slice()
	.len()
	.cmp(&self.single_split_permit_requests.as_slice().len())
	})

Skip unserviceable priority heads

When the highest-priority request's next split exceeds the currently available memory, this ordering keeps that request at the heap head, and pop_next_request_if_serviceable returns without examining lower-priority requests whose splits would fit. For example, with one warmup slot free and 40 MB available, a promoted 50 MB split blocks a queued 10 MB split, leaving usable capacity idle until another active search releases memory. The scheduler should preserve priority among serviceable requests rather than allowing an unserviceable priority head to block the entire queue.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[trinity-1686a]

it's a bit strange that all tasks must have the same priority, but there's still a per-task priority field

[Abdul-Andha]

i agree its a bit strange. i think the alternative is introducing a new struct for per-leafsearch metadata. what do you think?

struct LeafSearchTaskMetadata {
    priority: i32,
    splits: Vec<SplitSearchTaskMetadata>,
}

[trinity-1686a]

i think it looks better