[Agent evals] Add Vally coverage for Java skills and onboard the shared eval pipeline

[helen229]

Context

This tracks the Java rollout for Azure/azure-sdk-tools#16348

Before enabling the shared language-repository eval pipeline, add current-format Vally coverage for the Java-specific Agent Skills. Shared azsdk-common-* skills already carry centrally synced evals and should not be duplicated or independently modified here.

Use these sources of truth:

• Skill eval authoring workflow
• Eval authoring guide
• Skill eval layout example
• Vally configuration example
• Shared skill-eval entrypoint
• Shared eval archetype

Do not copy the eval-authoring-* meta-skills into this repository.

Skill eval work

Add or update Vally coverage for every Java-owned skill under .github/skills/:

• agentic-workflows
• .github/skills/azsdk-java-patch-mergeback (use the canonical skill name from its SKILL.md in graders and tags)
• create-package-skill
• find-package-skill
• management-autopr-review

For each skill:

• Read its SKILL.md, especially its activation boundaries, guardrails, and invoked tools or commands.
• Put new routing and capability coverage in .github/skills/<skill>/evals/eval.yaml. Split into additional <behavior>.eval.yaml files only when coverage is genuinely large.
• Add at least three realistic triggers and three anti-triggers using skill-invocation.
• For a boundary prompt, mount the closest competing skill, require that skill, and disallow the skill under test.
• Add capability or guardrail stimuli using tool-calls and deterministic output, file, or command graders where applicable.
• Supply concrete Maven coordinates, repository paths, PR details, patch branch names, or other identifiers needed to complete the request.
• Use hermetic fixtures and the mock MCP. Do not use secrets, network-dependent writes, or production targets.
• Set tags.area to the exact canonical skill name, use defaults:, and set scoring.threshold: 0.8.

Cover meaningful boundaries, including:

• create-package-skill versus find-package-skill and ordinary package modification.
• Patch merge-back versus triggering a release, starting a patch cycle, or general SDK generation.
• management-autopr-review versus APIView feedback resolution, implementation, and general pipeline repair.
• agentic-workflows versus Java SDK CI, package, or release workflows.

Audit the existing management-autopr-review/evals/ suite (trigger, true-negatives, findings, and orchestration) against the current four-layer guidance. Preserve its useful fixtures and behavior coverage. Reconcile its nested .vally.yaml with the new repository-level configuration so local commands and CI use one clear environment contract.

Repository configuration

Add .github/skills/.vally.yaml following the tools-repo layout:

• Register every local and synced skill eval directory, including the existing management review suite.
• Define azsdk-mcp-mock using ../../artifacts/mcp/mock/azsdk-mock.dll.
• Set cwd: . for the MCP server so paths remain stable in Vally trial workspaces.
• Add .github/skills/.gitignore for Vally results and caches.

Pipeline

Add eng/pipelines/skill-eval.yml as a thin entrypoint extending:

/eng/common/pipelines/templates/stages/archetype-eval.yml

Required configuration:

• Declare Azure/azure-sdk-tools as repository resource azure-sdk-tools, using endpoint Azure and refs/heads/main.
• Use /eng/common/pipelines/templates/steps/eval-mcp-setup.yml.
• Set TestType: mock and toolsRepo: azure-sdk-tools.
• Set vallyRoot: .github/skills.
• Discover both */evals/eval.yaml and */evals/*.eval.yaml so local and synced suites run.
• Use threshold: 0.8, failOnFailedTests: false, and shardTimeoutInMinutes: 20.
• Reference /eng/pipelines/templates/variables/image.yml and AzSDK_Eval_Variable_group.
• Start manual/scheduled on main with trigger: none and pr: none. Add selective PR triggers or blocking only after the pilot is stable.

Do not copy or customize shared eng/common framework files in this repository. Framework fixes belong in azure-sdk-tools and must arrive through the normal sync.

Register the YAML as the java - skill eval Azure DevOps pipeline and authorize its repository resource and variable group.

Acceptance criteria

• Every current Java-owned skill has current-format Vally routing coverage.
• Existing management AutoPR review coverage and fixtures are retained and hardened.
• Supported capabilities and critical guardrails have deterministic graders.
• Shared azsdk-common-* evals remain centrally sourced.
• Strict Vally lint passes for every added or changed eval.
• Focused local eval runs pass and their commands and pass counts are recorded in the PR.
• The scheduled, report-only pipeline completes and publishes standard summary, JUnit, and failure-debug artifacts.
• No production writes, credentials, or real customer identifiers are used.
• Any deferred live-only behavior is documented in a follow-up issue.

[github-actions]

🎯 Agentic Issue Triage — Needs Manual Review

🏷️ Label Decision

• Candidate labels considered: "Central-EngSys" was the only plausible category, since the issue is entirely about adding Vally eval coverage for .github/skills/* Agent Skills, creating .github/skills/.vally.yaml, and registering a new eng/pipelines/skill-eval.yml Azure DevOps pipeline (java - skill eval). No specific com.azure/com.azure.resourcemanager/com.azure.provisioning package or /sdk/ service directory is referenced, so no Client/Mgmt/Provisioning + service label pair is applicable. The closest matching service label would be tied to individual skill names (e.g., azsdk-java-patch-mergeback), but there is no single Azure service this maps to.
• Confidence blockers: The only viable category, "Central-EngSys", is explicitly excluded from automatic assignment per the triage rules (requires human judgment). There is also no unambiguous single service label — the issue spans multiple skills (agentic-workflows, azsdk-java-patch-mergeback, create-package-skill, find-package-skill, management-autopr-review) and pipeline/eng-systems work, not a specific Azure service.
• Outcome: Requested needs-triage — could not confidently predict labels because the only applicable category ("Central-EngSys") is excluded from automated assignment and no single service label applies.

👥 Owner Routing

• CODEOWNERS scan: Not performed. Per the decision flow, Steps 4 and 5 (deprecated package check and CODEOWNERS owner lookup) are skipped whenever the final label set is needs-triage, since label prediction did not confidently resolve to a category + service pair.
• Matched entry: N/A — owner lookup was not attempted.
• Owners found: none — lookup skipped.
• Outcome: Requested needs-triage only; no owner routing was requested. A human triager should manually determine the appropriate Central-EngSys routing/ownership for this eng-systems/Vally-eval onboarding task.

Generated by Agentic Triage for #50496 · auto · 38.6 AIC · ⌖ 9.24 AIC · ⊞ 15.9K · ◷