Skip to content

Explain mean and variation in Statistics results #1550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Explain mean and variation in Statistics results #1550

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions apps/portal/src/pages/Statistics.stories.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.

import type { Meta, StoryObj } from "@storybook/react-vite";
import { expect } from "storybook/test";

import { PerformanceTable } from "./Statistics";
import type { AnalysisResponse } from "@/types";

const analysis: AnalysisResponse = {
groups: [{
task: "Build a small API",
taskPromptId: "task-1",
workerType: "coder-acp-copilot",
total: 4,
completed: 4,
passed: 3,
rejected: 0,
passAtK: { 1: 0.75 },
successAtT: [0.5, 0.75],
iterationStats: { mean: 2, stdDev: 0.8, min: 1, max: 3 },
durationStats: { mean: 90_000, stdDev: 30_000, min: 60_000, max: 120_000 },
}],
kValues: [1],
maxT: 3,
summary: { totalRuns: 4, completedRuns: 4, passedRuns: 3, overallPassRate: 0.75, avgIterationsToPass: 2 },
availableCriteria: [],
selectedCriteria: [],
availableFeatures: [],
selectedFeatures: [],
};

const meta = {
component: PerformanceTable,
args: { data: analysis },
} satisfies Meta<typeof PerformanceTable>;

export default meta;
type Story = StoryObj<typeof meta>;

export const WithVariation: Story = {
play: async ({ canvas }) => {
await expect(canvas.getByText("Mean 2.0")).toBeVisible();
await expect(canvas.getByText("Std dev 0.8")).toBeVisible();
await expect(canvas.getByText("Std dev 30s")).toBeVisible();
},
};
29 changes: 14 additions & 15 deletions apps/portal/src/pages/Statistics.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -428,7 +428,7 @@ function InsightsRow({ insights }: { insights: DerivedInsights }) {

// ─── Performance breakdown (merged iteration + duration) ────────────────────

function PerformanceTable({ data }: { data: AnalysisResponse }) {
export function PerformanceTable({ data }: { data: AnalysisResponse }) {
const { groups } = data;

if (groups.length === 0) {
Expand Down Expand Up @@ -472,8 +472,9 @@ function PerformanceTable({ data }: { data: AnalysisResponse }) {
Performance by task & worker
</CardTitle>
<CardDescription>
Pass rate, iteration distribution, and run duration — sorted by pass
rate.
Pass rate and run variation, sorted by pass rate. Mean and standard
deviation use passed runs only; standard deviation shows how much
individual runs vary from the mean.
</CardDescription>
</CardHeader>
<CardContent>
Expand All @@ -484,8 +485,8 @@ function PerformanceTable({ data }: { data: AnalysisResponse }) {
<TableHead>Worker</TableHead>
<TableHead className="text-center">Runs</TableHead>
<TableHead className="w-[200px]">Pass rate</TableHead>
<TableHead className="text-center">Avg iter</TableHead>
<TableHead className="text-center">Avg duration</TableHead>
<TableHead className="text-center">Iterations to pass</TableHead>
<TableHead className="text-center">Run duration</TableHead>
<TableHead className="w-[1%]" />
</TableRow>
</TableHeader>
Expand Down Expand Up @@ -551,22 +552,20 @@ function PerformanceTable({ data }: { data: AnalysisResponse }) {
</TableCell>
<TableCell className="text-center font-mono tabular-nums text-xs">
{group.iterationStats ? (
<span
title={`min ${group.iterationStats.min} · max ${group.iterationStats.max} · σ ${formatNumber(group.iterationStats.stdDev)}`}
>
{formatNumber(group.iterationStats.mean)}
</span>
<div title={`Minimum ${group.iterationStats.min} · Maximum ${group.iterationStats.max}`}>
<span className="block">Mean {formatNumber(group.iterationStats.mean)}</span>
<span className="block text-muted-foreground">Std dev {formatNumber(group.iterationStats.stdDev)}</span>
</div>
) : (
<span className="text-muted-foreground">—</span>
)}
</TableCell>
<TableCell className="text-center font-mono tabular-nums text-xs">
{group.durationStats ? (
<span
title={`min ${formatDuration(group.durationStats.min)} · max ${formatDuration(group.durationStats.max)} · σ ${formatDuration(group.durationStats.stdDev)}`}
>
{formatDuration(group.durationStats.mean)}
</span>
<div title={`Minimum ${formatDuration(group.durationStats.min)} · Maximum ${formatDuration(group.durationStats.max)}`}>
<span className="block">Mean {formatDuration(group.durationStats.mean)}</span>
<span className="block text-muted-foreground">Std dev {formatDuration(group.durationStats.stdDev)}</span>
</div>
) : (
<span className="text-muted-foreground">—</span>
)}
Expand Down
5 changes: 5 additions & 0 deletions docs/architecture/app-design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -702,6 +702,11 @@ The Statistics page (`apps/portal/src/pages/Statistics.tsx`) is backed by `GET /
| `criteria` | Success criteria (comma-separated `criterionId`s) | A run counts as a pass only if **all** selected criteria pass; runs lacking a selected criterion are excluded |
| `features` | Task prompt features (comma-separated `featureId`s) | Keep a run iff its task prompt was **detected** to have **every** selected feature (AND) |

The performance table labels each passed-run distribution with its mean and
standard deviation. Iteration statistics describe iterations to pass; duration
statistics sum the measured turn durations for each passed run. Missing values
appear when no passed run has the required measurement.

The response exposes the option lists and current selection for each filter so the Portal can render filter bars: `availableCriteria` / `selectedCriteria` and `availableFeatures` / `selectedFeatures`. Both `available*` lists are computed over the full valid-run set **before** filtering, so the bars stay populated even when a filter combination matches zero runs (the Portal then shows a "no runs match" empty state instead of the "no data yet" state).

**Detected-based feature semantics (intentional divergence).** A task prompt's `features[]` stores a `{featureId, detected}` row for *every* evaluated feature, so presence is near-universal and meaningless as a filter. `availableFeatures` is therefore the sorted union of featureIds with `detected === true` across valid runs, and the feature filter matches on `detected === true`. This is deliberately stricter than the MDP route (`GET /api/v1/criteria/mdp`), whose feature filter is presence-based. Runs join to task prompts by effective id (`taskPromptId || computeTaskPromptId(scenario.task)`) so legacy runs without a stored `taskPromptId` still resolve their features.
Expand Down