Skip to content

FIX Honor score feedback and handle zero-vector similarity - #2403

Open
Roman Lutz (romanlutz) wants to merge 1 commit into
microsoft:mainfrom
romanlutz:romanlutz-daily-audit-2026-08-16
Open

FIX Honor score feedback and handle zero-vector similarity#2403
Roman Lutz (romanlutz) wants to merge 1 commit into
microsoft:mainfrom
romanlutz:romanlutz-daily-audit-2026-08-16

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

use_score_as_feedback=False was preserved in attack configuration but ignored by Crescendo and Tree of Attacks with Pruning. This change keeps prior target-response context available to the adversarial model while suppressing score values and rationales; TAP also skips the unnecessary score lookup. Default score-feedback behavior remains unchanged.

The audit also found that cosine similarity returned nan for zero vectors, so zero-norm comparisons now return 0.0. Stale PyRIT shell examples are migrated from the deprecated print-scenario alias to scenario-results and updated to match the current history output.

This does not change Crescendo refusal, backtracking, or terminal success semantics.

Tests and Documentation

  • Full unit suite: 15,228 passed, 121 skipped.
  • Changed analytics, Crescendo, and TAP modules: 196 passed.
  • Ruff, changed-file type checks, async-suffix validation, documentation validation, and all changed-file pre-commit hooks passed.
  • Added regression coverage for disabled score feedback in Crescendo and TAP, including the absence of TAP score lookup.
  • Added zero-vector cosine-similarity coverage.
  • Updated doc/scanner/2_pyrit_shell.md; JupyText is not applicable to this Markdown-only documentation change.

Handle zero-vector cosine similarity deterministically and migrate deprecated shell examples to scenario-results.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants