Skip to content

FIX PlagiarismScorer: verbatim fast path matched sub-word substrings - #2388

Open
WatchTree-19 wants to merge 1 commit into
microsoft:mainfrom
WatchTree-19:fix-plagiarism-scorer-subword-substring
Open

FIX PlagiarismScorer: verbatim fast path matched sub-word substrings#2388
WatchTree-19 wants to merge 1 commit into
microsoft:mainfrom
WatchTree-19:fix-plagiarism-scorer-subword-substring

Conversation

@WatchTree-19

Copy link
Copy Markdown
Contributor

Description

PlagiarismScorer._plagiarism_score has a fast path that returns 1.0 when the reference appears verbatim in the response. That check used a raw string test (reference in response), but every metric in the scorer is word-level: the text is tokenized with lowercasing and punctuation removal before LCS / Levenshtein / Jaccard are computed. The raw check was inconsistent with that tokenization in both directions:

  • False positive: a short reference that is only a substring of a longer response word scored full plagiarism. For example reference="cat" against response="concatenate the results" returned 1.0 for all three metrics, because "cat" is a substring of "concatenate" even though cat never appears as a word.
  • Missed match: a word-level verbatim copy that differed only in case or punctuation did not take the fast path (it happened to fall through to the metric, which usually recovered, but the shortcut itself was wrong).

The fix compares the tokenized sequences instead: the reference tokens must appear as a contiguous run inside the response tokens. This keeps the fast path consistent with the case/punctuation-insensitive word-level semantics the metrics already use.

This continues the small correctness pass on the converters/scorers (cf. #2133, #2137, #2278, #2279).

Tests and Documentation

Added regression tests in tests/unit/score/test_plagiarism_scorer.py:

  • a sub-word substring ("cat" vs "concatenate the results") now scores 0.0 for every metric,
  • a word-level verbatim copy differing in case/punctuation still scores 1.0,
  • a direct unit test for the new _is_contiguous_sublist helper.

Full file passes: pytest tests/unit/score/test_plagiarism_scorer.py -> 37 passed. ruff check and black clean. No documentation changes needed (internal scoring behavior only).

…strings

The verbatim-match shortcut in _plagiarism_score used a raw string check
(reference in response). The rest of the scorer is word-level: it tokenizes
with lowercasing and punctuation removal before computing LCS / Levenshtein /
Jaccard. The raw check was inconsistent with that in both directions:

  - false positive: a short reference that is only a substring of a longer
    response word scored 1.0 (e.g. reference 'cat' vs response 'concatenate
    the results' returned full plagiarism for every metric).
  - missed match: a word-level verbatim copy differing only in case or
    punctuation did not take the fast path.

Compare the tokenized sequences instead, so the fast path matches the same
word-level semantics the metrics use. Adds regression tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant