[Improve] Loosen the addressee gate for unmentioned replies and log its scores - #3131
Merged
Merged
Conversation
The gate was too conservative in real threads: a reply routed only when the judgment model put at least 0.85 on Roomote and 0.5 on expecting a response, so short questions such as "Which test?" and remarks aimed at Roomote stayed silent, and the fixed bar did not transfer between judgment backends with different calibration. Route when Roomote is the majority addressee (0.5). Replace the expects-response question with one that detects only closing acknowledgements, so banter aimed at Roomote routes while "ok thanks" does not. Log every decision's probabilities, without message text, so the gate can be tuned from ordinary logs.
Contributor
|
No code issues found. See task
Reviewed 4c3bc98 |
The lower threshold only means something over a real distribution, so reject probability vectors that do not sum to one and compute the likeliest addressee from the probabilities instead of trusting the reported choice.
A thread with one human and nobody else mentioned has only one possible addressee, so the judgment call there was latency and a false-silence risk for nothing. Consult the model only when another human has posted, somebody else was mentioned earlier, or the reply itself mentions somebody else. Slack and Discord pass the reply's own peer-mention flag; Teams already refuses those replies before routing.
The majority bar requires a strict winner; a tie between Roomote and another option no longer routes.
Divide the reported probabilities by their sum before comparing, so the validator's rounding tolerance cannot lift a sub-majority Roomote score over the bar.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
0.5) instead of requiring0.85. Human-to-human traffic scores far below this, and a majority bar holds up across judgment backends whose absolute probabilities are calibrated differently. Routing is still not a reply: Fast keeps its own rule to stay silent in a multi-human thread when Roomote was not the last speaker.[UnmentionedThreadReply], with the message id but never the text, so the gate can be tuned from ordinary logs.Why this change was made
In a real multi-person thread the gate stayed silent on "Which test?" asked right after a Roomote message and on a joke aimed at Roomote, which read as only answering explicit mentions. The 0.85 bar was set on one judgment backend and does not transfer to another.
Impact