Skip to content

[Improve] Loosen the addressee gate for unmentioned replies and log its scores - #3131

Merged
daniel-lxs merged 6 commits into
developfrom
feat/addressee-gate-calibration
Sep 22, 2026
Merged

daniel-lxs merged 6 commits into
developfrom
feat/addressee-gate-calibration

Conversation

@daniel-lxs

@daniel-lxs daniel-lxs commented Sep 22, 2026

Copy link
Copy Markdown
Member

What changed

  • The unmentioned-reply gate routes when Roomote is the majority addressee (probability at least 0.5) instead of requiring 0.85. Human-to-human traffic scores far below this, and a majority bar holds up across judgment backends whose absolute probabilities are calibrated differently. Routing is still not a reply: Fast keeps its own rule to stay silent in a multi-human thread when Roomote was not the last speaker.
  • The second question no longer asks whether the reply expects a response. It asks whether the reply is only a closing acknowledgement ("ok thanks", "got it", an emoji), and a reply is dropped only when that is more likely than not. Remarks, jokes, and short follow-up questions aimed at Roomote now route; sign-offs still do not.
  • The model is consulted only when another human is part of the conversation: someone else has posted in the thread, somebody else was mentioned earlier, or the reply itself mentions somebody else. A sender alone with Roomote has only one possible addressee, so those replies route on the heuristic with no judgment call. Slack and Discord pass the reply's own peer-mention flag; Teams already refuses such replies before routing.
  • Addressee probabilities must form a distribution (sum to one within a small tolerance) and the likeliest addressee is computed from them rather than read from the reported choice, so the lower bar cannot be satisfied by a malformed vector.
  • Every decision logs its probabilities and outcome under [UnmentionedThreadReply], with the message id but never the text, so the gate can be tuned from ordinary logs.
  • Docs on Slack, Discord, Teams, and personal settings describe the acknowledgement rule, with a changeset.

Why this change was made

In a real multi-person thread the gate stayed silent on "Which test?" asked right after a Roomote message and on a joke aimed at Roomote, which read as only answering explicit mentions. The 0.85 bar was set on one judgment backend and does not transfer to another.

Impact

  • Deployments with a judgment backend route more unmentioned replies aimed at Roomote; peer discussion and acknowledgements stay silent. Deployments without one are unchanged.
  • Evidence: 126 routing and judgment tests, API typecheck, formatting, and lint pass. A real 24-message multi-person thread replayed through the gate against Jev: every message aimed at Roomote routes (0.87 to 0.99), every human-to-human line stays below 0.45, and the only sign-off is caught as an acknowledgement at 0.94; one ambiguous line ("Can we get the scores?") routes at 0.56. Live on a local stack through the mock Slack harness: in a multi-person thread a remark aimed at Roomote routed and got a reply, a thanks stayed silent, and a message to a colleague stayed silent; in a thread with a single person, two unmentioned replies routed with no judgment call at all. The gate's own judgment call measured about 300 ms per message on Jev via OpenRouter. A who-spoke-last hint was tried and dropped: it pushed post-answer commentary between humans to 0.78. Teams has no harness and is covered by unit tests only.

The gate was too conservative in real threads: a reply routed only when
the judgment model put at least 0.85 on Roomote and 0.5 on expecting a
response, so short questions such as "Which test?" and remarks aimed at
Roomote stayed silent, and the fixed bar did not transfer between judgment
backends with different calibration.

Route when Roomote is the majority addressee (0.5). Replace the
expects-response question with one that detects only closing
acknowledgements, so banter aimed at Roomote routes while "ok thanks" does
not. Log every decision's probabilities, without message text, so the
gate can be tuned from ordinary logs.
@roomote-community

roomote-community Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

No code issues found. See task

  • apps/api/src/handlers/shared/unmentioned-thread-reply.ts:410 Majority gate routes a tied addressee distribution as Roomote.
  • apps/api/src/handlers/shared/unmentioned-thread-reply.ts:415 Sum tolerance lets a non-majority Roomote score route.
  • apps/api/src/handlers/shared/unmentioned-thread-reply.ts:431 Inclusive cutoff routes Roomote when it only ties the alternatives combined.

Reviewed 4c3bc98

Comment thread apps/api/src/handlers/shared/unmentioned-thread-reply.ts Outdated
The lower threshold only means something over a real distribution, so
reject probability vectors that do not sum to one and compute the
likeliest addressee from the probabilities instead of trusting the
reported choice.
A thread with one human and nobody else mentioned has only one possible
addressee, so the judgment call there was latency and a false-silence
risk for nothing. Consult the model only when another human has posted,
somebody else was mentioned earlier, or the reply itself mentions
somebody else. Slack and Discord pass the reply's own peer-mention flag;
Teams already refuses those replies before routing.
The majority bar requires a strict winner; a tie between Roomote and
another option no longer routes.
Divide the reported probabilities by their sum before comparing, so the
validator's rounding tolerance cannot lift a sub-majority Roomote score
over the bar.
@daniel-lxs
daniel-lxs merged commit 73235fc into develop Sep 22, 2026
18 checks passed
@daniel-lxs
daniel-lxs deleted the feat/addressee-gate-calibration branch September 22, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant