feat: add teacher-configurable submission matching (P-673)

[Acture]

When submissions use nested directories, duplicate filenames or nonstandard function names, teachers need to resolve the association before grading. This adds shared assignment.toml rules for student ownership, item files, function aliases and per-student overrides, plus scriptmark match to preview candidates and their rule evidence without importing student code.

Exact matches and unique declared aliases can execute. Ambiguous matches, fuzzy suggestions and invalid explicit overrides require review and withhold grades as teacher matching faults. Results retain original file/archive provenance and runtime function decisions; CLI and Python run/grade use the same configuration. Canvas source identities remain authoritative.

Includes a runnable matching bundle and regression coverage for nested/shared files, conflicting rules, aliases, overrides, archives and Canvas provenance. The matching guide and validation record follow the P-850 notes workflow; this PR records the published notes commit instead of restoring a root documentation tree. Static preview defers dynamic exports and syntax failures to execution. Versioned result/regrading work remains separate.

Validation:

• cargo test --offline --workspace: 379 tests passed with Python 3.13.
• After rebasing onto current master, cargo test --offline -p scriptmark --test matching --test harness: all 41 tests passed.
• cargo fmt --check, cargo clippy --offline --all-targets -- -D warnings, git diff --check, Ruff and ty passed.
• Shipped CLI preview/grading and native Python 3.13 run/grade smoke checks passed.

Related to P-673

Summary by CodeRabbit

• New Features
  • Added scriptmark match to preview student, file, and function matches and export candidates as JSON without running submissions.
  • Added assignment settings for student ownership, file patterns, function aliases, and per-student overrides.
  • Matching decisions are included in grading reports; unresolved or ambiguous matches are flagged for teacher review.
  • The Python API now accepts an optional assignment configuration.
• Documentation
  • Expanded the CLI and assignment configuration guidance, including how to preview matches and resolve conflicts.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: d83100f5-fea5-44b4-80f1-da8b8d7a2e23

📥 Commits

Reviewing files that changed from the base of the PR and between 3d2ca8b and 1961069.

📒 Files selected for processing (25)

• README.md
• crates/scriptmark-py/src/lib.rs
• crates/scriptmark/src/assignment.rs
• crates/scriptmark/src/discovery.rs
• crates/scriptmark/src/lib.rs
• crates/scriptmark/src/main.rs
• crates/scriptmark/src/matching.py
• crates/scriptmark/src/matching.rs
• crates/scriptmark/src/models/config.rs
• crates/scriptmark/src/models/result.rs
• crates/scriptmark/src/models/submission.rs
• crates/scriptmark/src/runner/executor.rs
• crates/scriptmark/src/runner/harness.py
• crates/scriptmark/src/runner/judge.rs
• crates/scriptmark/src/runner/oracle.rs
• crates/scriptmark/src/runner/orchestrator.rs
• crates/scriptmark/src/runner/prepare.rs
• crates/scriptmark/src/runner/python.rs
• crates/scriptmark/tests/harness.rs
• crates/scriptmark/tests/input_equivalence.rs
• crates/scriptmark/tests/matching.rs
• examples/bundles/matching/assignment.toml
• examples/bundles/matching/submissions/alice/deep/solution.py
• examples/bundles/matching/tests/test_double.toml
• notes

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The change adds assignment-based matching for students, files, and functions. A new CLI command previews matching decisions as JSON. Grading uses matching rules and records unresolved decisions in reports.

Changes

Assignment matching

Layer / File(s)	Summary
Matching rules and result contracts crates/scriptmark/src/matching.rs, crates/scriptmark/src/assignment.rs, crates/scriptmark/src/models/config.rs, crates/scriptmark/src/models/result.rs, crates/scriptmark/src/models/submission.rs, crates/scriptmark/src/lib.rs	Adds matching configuration, validation, candidate decisions, and item-match data. Assignment configuration and result models carry matching settings, decisions, and ownership diagnostics.
Submission discovery and ownership crates/scriptmark/src/discovery.rs	Local discovery scans nested directories and archives. It resolves owners with matching rules and attaches ownership decisions to files.
Matching preview command crates/scriptmark/src/matching.py, crates/scriptmark/src/main.rs, README.md, examples/bundles/matching/*	Adds scriptmark match and Python static function analysis. The command prepares preview inputs, writes matching decisions as JSON, and avoids saving normalized Canvas input. README and example files describe and demonstrate matching configuration.
Matching-aware grading crates/scriptmark-py/src/lib.rs, crates/scriptmark/src/main.rs, crates/scriptmark/src/runner/*, crates/scriptmark/tests/*	Grading uses file and function decisions, records matching evidence, and reports unresolved matching outcomes as teacher errors. The Python API accepts an optional assignment path. Tests cover preview, ownership, overrides, grading outcomes, and matching validation.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant CLI as cmd_match
  participant Prep as prepare_batch
  participant Preview as matching::preview
  participant Python as matching.py
  participant Output as JSON output
  CLI->>Prep: prepare inputs in preview mode
  CLI->>Preview: pass inputs, specs, rules, and Python command
  Preview->>Python: send selected-file function requests
  Python-->>Preview: return function decisions
  Preview-->>CLI: return matching preview
  CLI->>Output: write pretty-printed preview JSON

Loading

Merge Risk: ⚪ Minimal · up to 19610

The notes checkout still requires repository access, but this change does not introduce that access problem. No issue identified here needs to block the merge.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 19610

Matching is now an explicit grading control rather than an automatic fuzzy substitution. The inspected paths keep file selection within each student's submission, preserve Canvas identity, and withhold grades for unresolved matching. No introduced security vulnerability was established, but the review does not establish complete protection across every filesystem and execution environment.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The demonstrated exposure is the local grading workflow: submission filenames and runtime exports influence matching and execution, while teacher policy can affect attribution and grading across the assignment. The inspected file-selection flow does not grant an override authority to select another student's collection or an arbitrary host path.

Security Findings and Attack Paths

• observed — The exact-lookup bypass raised during inspection is an existing reference-implementation behavior, not a demonstrated student matching bypass. The payload producer chooses matching for student subjects and exact lookup for reference subjects, as it did before this PR.

Trust Boundaries and Controls

• observed — Requested targets originate in teacher specification plans, while callable candidates come from the student module. Dynamic lookup and invocation run inside the guarded call. Non-matched configured decisions raise MatchingRequired instead of invoking a suggestion.
• observed — Matching outcomes in both setup and scored calls become teacher faults. Item scoring withholds on teacher faults, and any withheld item withholds the student's overall grade, preserving the review gate through the final grading decision.

Resilience and Maintainability Implications

• observed — Failed student imports do not proceed to setup or steps. Per-call evidence is reset, guarded-call cleanup restores execution state, and process deadlines and group cleanup contain unfinished units. Fresh unit processes also prevent failed-import module state from being reused by subsequent units.
• observed — New nested discovery does not descend through directory symlinks or revisit extraction caches. File metadata still follows file symlinks, a pre-existing behavior; the inspected traversal is therefore not evidence of a general filesystem sandbox.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 45.78% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 83 functions across 21 files. (4 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding teacher-configurable matching for submissions.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 45.78% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 83 functions across 21 files. (4 skipped: 4 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}