Skip to content

[Evaluation] Configure adaptive native AOAI output result page size - #49034

Merged
Rabah B (RabsB) merged 4 commits into
Azure:mainfrom
RabsB:rboubchir-microsoft-aoai-output-page-size
Sep 16, 2026
Merged

Rabah B (RabsB) merged 4 commits into
Azure:mainfrom
RabsB:rboubchir-microsoft-aoai-output-page-size

Conversation

@RabsB

@RabsB Rabah B (RabsB) commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • adds the optional public aoai_output_items_page_size keyword to evaluate, preserving existing calls with a default of 100
  • validates an integer value from 1 through 100 before preprocessing, client calls, or evaluation work begins; booleans, strings, floats, fractional values, and out-of-range values raise the existing EvaluationException user-input shape
  • threads the value through every native Azure OpenAI grader run and limits output items per HTTP response/request page, not response bytes, latency, or dataset rows evaluated; all result pages are still fetched
  • gives each output-items page three total attempts (the initial request plus two retries); timeouts and HTTP 408/504 retry the same cursor with max(1, (page_size + 1) // 2) and retain a successfully reduced size for later pages in that run
  • preserves the page size for other ordinarily retryable connection/status failures and delegates retry eligibility, Retry-After, retry headers, and delay calculation to the OpenAI client within the same attempt budget
  • disables OpenAI automatic retries only for output-items requests through a scoped with_options(max_retries=0) client; shared-client ownership and unrelated inference, grader creation, and run polling retry behavior are unchanged
  • advances the cursor and appends results only after a complete successful page, preserving complete retrieval, ordering/row alignment, grader-name mapping, metrics, and final-error propagation without partial success
  • records the user-facing behavior through the Chronus change fragment for the current unreleased package version

Validation

  • python -m pytest tests\unittests\test_aoai_evaluation_pagination.py -q — 25 passed
  • python -m pytest tests\unittests\test_aoai_evaluation_pagination.py tests\unittests\test_aoai_alignment_missing_rows.py tests\unittests\test_evaluate_aoai.py tests\unittests\test_evaluate.py -q — 246 passed, 7 skipped
  • azpysdk black . — passed
  • azpysdk breaking . — passed; package is currently opted out of breaking-change comparison
  • azpysdk apistub . — generated the expected public signature, then the wrapper failed during metadata extraction because current main does not contain eng\scripts\Extract-APIViewMetadata-Python.ps1; only the intended API signature update is included
  • azpysdk changelog verify — could not run because the pinned Chronus dependencies could not be installed in this environment due npm authentication/TLS failures; the fragment follows the repository's .chronus/changes format

Tests use controlled responses and mocked sleep; no live evaluation qualification is claimed.

Scope

This is SDK Part 1 only. Vienna/ACA integration, ingress changes, adaptive timing thresholds, page-size growth, dataset sampling, and deployment are intentionally unchanged and remain separate follow-up work.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added the Evaluation Issues related to the client library for Azure AI Evaluation label Sep 16, 2026
@RabsB
Rabah B (RabsB) marked this pull request as ready for review September 16, 2026 11:41
Copilot AI balanced review requested due to automatic review settings September 16, 2026 11:41
@RabsB
Rabah B (RabsB) requested a review from a team as a code owner September 16, 2026 11:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI review requested due to automatic review settings September 16, 2026 11:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 16, 2026 16:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

@RabsB Rabah B (RabsB) changed the title [Evaluation] Configure native AOAI output result page size [Evaluation] Configure adaptive native AOAI output result page size Sep 16, 2026
@RabsB
Rabah B (RabsB) marked this pull request as draft September 16, 2026 16:32
@RabsB
Rabah B (RabsB) marked this pull request as ready for review September 16, 2026 17:58
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI review requested due to automatic review settings September 16, 2026 17:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

@RabsB
Rabah B (RabsB) merged commit 3e96bca into Azure:main Sep 16, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Evaluation Issues related to the client library for Azure AI Evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants