Repository navigation
feat: add report-only community contribution insights - #1524
Open
Wassim Chegham (manekinekko) wants to merge 3 commits into
Open
Wassim Chegham (manekinekko) wants to merge 3 commits into
Wassim Chegham (manekinekko) wants to merge 3 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00c70221-6615-45d2-b39d-3ae77292b59c
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00c70221-6615-45d2-b39d-3ae77292b59c
github-actions
Bot
added
type: documentation
Wassim Chegham (manekinekko)
added
topic: dx
Test Results (Node.js 22)test: Run #306
🎉 All tests passed! |
Make Microsoft-Decision-1 the sole final review-action selector while Copilot supplies all insights and contributor feedback. Add isolated Azure configuration, production-backed decision evaluations, and concise report summaries with expandable evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 00c70221-6615-45d2-b39d-3ae77292b59c
Test Results (Node.js 22)test: Run #378
🎉 All tests passed! |
Wassim Chegham (manekinekko)
marked this pull request as ready for review
October 23, 2026 23:12
Wassim Chegham (manekinekko)
requested a review
from Cedric Vidal (cedricvidal)
as a code owner
October 23, 2026 23:12
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved workflow trust-boundary gaps, CI trigger omissions, and grading-context mismatch block approval.
4 open issues
1 file not reviewed
pnpm-lock.yaml: Generated file
What changed in this PR
Adds report-only community contribution triage to Scope’s repository tooling, separating Copilot-generated insights from optional Microsoft-Decision-1 review-action selection.
Changes:
- Adds hosted and local assessment flows with snapshot validation, isolated credentials, and artifact-only reports.
- Integrates separate insight and decision families into the static-prompt evaluation framework.
- Documents configuration, trust boundaries, and operating limits without changing Scope’s public API or distributed CLI.
| File | Description |
|---|---|
scripts/license-headers.ts |
Includes .mts files in license checks. |
scripts/contribution-insights/workflow.test.ts |
Tests workflow boundaries and report execution. |
scripts/contribution-insights/report.test.ts |
Tests validation, citations, and report layout. |
scripts/contribution-insights/report.mts |
Validates assessments and renders reports. |
scripts/contribution-insights/decision.test.ts |
Tests decision configuration and transport contracts. |
scripts/contribution-insights/decision.mts |
Implements the dedicated Decision-1 client. |
scripts/contribution-insights/collect.test.ts |
Tests bounded collection and evidence handling. |
pnpm-lock.yaml |
Records the developer-only SDK dependency. |
package.json |
Exposes the local assessment command. |
evaluations/static-prompts/tests/quality/test_runner.py |
Tests decision-only evaluation execution. |
evaluations/static-prompts/tests/quality/test_deterministic.py |
Tests exact decision labels and schema checks. |
evaluations/static-prompts/tests/quality/test_data.py |
Tests report and decision normalization. |
evaluations/static-prompts/tests/quality/test_configuration.py |
Tests the new evaluator configurations. |
evaluations/static-prompts/src/static_prompt_evals/quality/runner.py |
Skips unnecessary Azure graders. |
evaluations/static-prompts/src/static_prompt_evals/quality/models.py |
Retains native decision requests. |
evaluations/static-prompts/src/static_prompt_evals/quality/deterministic.py |
Adds decision accuracy and exact-value checks. |
evaluations/static-prompts/src/static_prompt_evals/quality/data.py |
Normalizes contribution evaluation output. |
evaluations/static-prompts/src/static_prompt_evals/quality/configuration.py |
Allows deterministic-only evaluation families. |
evaluations/static-prompts/scripts/validate-dataset.ts |
Validates contribution cases and action coverage. |
evaluations/static-prompts/scripts/validate-coverage.test.ts |
Updates expected quality-target coverage. |
evaluations/static-prompts/scripts/local-copilot.ts |
Provides an isolated local Copilot session. |
evaluations/static-prompts/scripts/local-copilot.test.ts |
Tests isolation, tool restrictions, and cleanup. |
evaluations/static-prompts/scripts/harvest.ts |
Adds reproducible contribution case curation. |
evaluations/static-prompts/scripts/harvest.test.ts |
Checks committed fixture reproducibility. |
evaluations/static-prompts/scripts/dataset-adapter-contract.test.ts |
Exercises contribution adapter contracts. |
evaluations/static-prompts/scripts/contribution-insights-local.ts |
Runs local collection, inference, and reporting. |
evaluations/static-prompts/scripts/contribution-insights-local.test.ts |
Tests local options and GitHub reads. |
evaluations/static-prompts/scripts/contribution-insights-fixtures.ts |
Defines eight contribution scenarios. |
evaluations/static-prompts/scripts/contribution-decision-fixtures.ts |
Supplies fixed insights for decision evaluation. |
evaluations/static-prompts/README.md |
Documents local operation and evaluation families. |
evaluations/static-prompts/package.json |
Adds the local runner and SDK dependency. |
evaluations/static-prompts/evaluators/rubrics.yaml |
Defines insight grading and decision accuracy. |
evaluations/static-prompts/evaluation-manifest.yaml |
Registers both contribution families. |
evaluations/static-prompts/datasets/v1/contribution-insights.jsonl |
Adds curated insight evaluation cases. |
evaluations/static-prompts/datasets/manifest.json |
Registers contribution datasets and hashes. |
evaluations/static-prompts/adapter/validation.ts |
Adds explicit transport-error classification. |
evaluations/static-prompts/adapter/transport.ts |
Adds real and fixed-fake decision transports. |
evaluations/static-prompts/adapter/targets/contribution-insights.ts |
Reuses production insight composition and validation. |
evaluations/static-prompts/adapter/targets/contribution-decision.ts |
Builds native decision evaluation requests. |
evaluations/static-prompts/adapter/runner.ts |
Classifies dedicated transport failures. |
evaluations/static-prompts/adapter/registry.ts |
Registers the contribution adapters. |
evaluations/static-prompts/adapter/protocol.ts |
Adds contribution families and decision types. |
evaluations/static-prompts/adapter/contribution-insights.test.ts |
Tests insight composition and submission contracts. |
evaluations/static-prompts/adapter/contribution-decision.test.ts |
Tests decision separation and production transport. |
evaluations/static-prompts/adapter/adapter.test.ts |
Updates adapter registry expectations. |
ENV_VARIABLES.md |
Documents pilot and decision settings. |
docs/architecture/prompt-evaluations.md |
Documents contribution prompt contracts and evaluation. |
CONTRIBUTING.md |
Explains the report-only pilot and compiler version. |
AGENTS.md |
Links the contribution-insights architecture guide. |
.github/workflows/contribution-insights.md |
Defines hosted orchestration and assessment instructions. |
.github/workflows/ci.yml |
Adds downstream contribution workflow filters. |
.github/aw/actions-lock.json |
Pins the new workflow compiler setup action. |
Review effort: Balanced
| required: false | ||
| default: "3" | ||
|
|
||
| if: github.repository == 'microsoft/scope' && github.ref == 'refs/heads/main' && (github.event_name == 'workflow_dispatch' || vars.CONTRIBUTION_INSIGHTS_ENABLED == 'true') |
| contribution-report: | ||
| description: "Save one report-only contribution assessment as validated JSON and Markdown workflow artifacts. Does not post feedback or mutate GitHub entities." | ||
| max: 1 | ||
| if: needs.agent.result == 'success' && needs.detection.result == 'success' |
Comment on lines
+848
to
+849
| - '.github/workflows/contribution-insights.md' | ||
| - '.github/workflows/contribution-insights.lock.yml' |
| query = stable_text(query_messages) if query_messages else stable_text(case_input) | ||
| if family == "developer-feedback" and query_messages: | ||
| context = stable_text(query_messages) | ||
| if family == "contribution-decision": |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The workflow remains read-only and artifact-only. No comments, labels, reviewer requests, approvals, branch writes, closures, or merges are enabled. Hosted schedules remain opt-in, and this PR provisions no Azure resources.
Demos
N/A: repository automation and developer tooling; no Portal or distributed Scope CLI UX changed. Live reports, Azure settings, and credentials remain local and are not committed.
Before
N/A.
After
N/A.
Testing
pnpm exec vitest run scripts/contribution-insights/collect.test.ts scripts/contribution-insights/decision.test.ts scripts/contribution-insights/report.test.ts scripts/contribution-insights/workflow.test.ts scripts/ci-workflow.test.ts --reporter=dotpnpm --filter static-prompt-evals typecheckpnpm --filter static-prompt-evals test:tspnpm --filter static-prompt-evals validate-datapnpm --filter static-prompt-evals exec uv run pytest tests/quality --basetemp=results/pr-1524-verification/pytest -o addopts='' --tb=short -qgh aw compile contribution-insights --no-check-update --actionlint --approvegh aw lint .github/workflows/contribution-insights.lock.ymlpnpm headers:checkCached Python/offline evaluation environment, set from the repository root:
Explicit prompt-evaluation commands also passed:
Both passed as plumbing/contract smoke runs, not model-quality baselines. The fake decision transport always abstains and never reads expected labels to choose its answer.
Live manual verification: a fresh native run on public PR #1492 collected current evidence, generated new Copilot insights (
gpt-6-luna), and obtained a real Microsoft-Decision-1 result. Separate live decision-only calls verified both Entra ID and API-key authentication. The concise report was then re-rendered without changing its recorded JSON, evidence snapshot, or final decision. No Actions workflow was dispatched or contributor feedback posted.Hosted Actions execution, its sandbox/firewall and threat-detection path, paid Azure-assisted quality grading, and production calibration were not run.
Documentation and compatibility
No Scope REST API, distributed CLI, or database compatibility changes. Report schema v2 replaces the earlier Copilot-authored recommendation with insights plus trusted Decision-1 metadata; rerun historical v1 JSON through the new pipeline rather than treating it as a Decision-1 result. Existing historical Markdown remains unchanged.
Local
--decisionrequires an existing deployment and explicit configuration before Copilot inference. Azure inference is billed separately from Copilot budgets. Hosted configuration uses an Actions API-key secret; injected Entra tokens are supported locally, but no hosted OIDC provisioning is added. Existing human approval, CLA, and merge requirements remain unchanged.The Copilot SDK dependency is developer-only and already used elsewhere in the workspace. The lockfile is updated; production license payloads are unchanged, so no NOTICE regeneration is required.
Agentic Workflow credential review
This update adds the
CONTRIBUTION_DECISION_API_KEYsecret reference; none are removed. Relative to the existing draft, no action dependencies/pins are added or removed and no redirect policy changes.Reviewed: the key is supplied only to the trusted report-rendering step, outside the Copilot job. It is used in an authentication header, never in prompts or decision input. The client validates an HTTPS Azure resource root, rejects redirects, bounds input/response bytes and request time, and retries only explicit HTTP 429 rejections. GitHub permissions remain read-only/empty, and shared compiler metadata retains older workflow pins and the lockfile merge policy.
Maintainers must still verify the configured Azure resource's ownership, authorization, billing, and hosted secret configuration before rollout. No real key, tenant/subscription ID, private endpoint, or live report is committed.
Checklist
up()/down()and keep it CosmosDB-compatible. N/A: no database changes.NOTICE/NOTICE-REVIEW.txtwithpnpm noticeas needed. Lockfile updated; the SDK addition is developer-only and introduces no new production notice content.