Skip to content

Facts and conflicts are governed like duplicates - #699

Closed
WaylandYang wants to merge 3 commits into
devfrom
feat/every-queue-is-governed
Closed

WaylandYang wants to merge 3 commits into
devfrom
feat/every-queue-is-governed

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Cut 1 of #695. Low-confidence facts, stale facts and temporal conflicts are handled by the governor, the same way duplicate pairs are (0025). This is recorded as 0043. Nods (pending_facts, 0015) stay with people. Violations, ontology defects and concept mappings are cut 2.

Why

On the Blackbaud lease bench, the amendments' new deadlines came out at confidence 0.7. A value that doubtful may not take over from the one before it (0022, #679), so the conflict and the fact both waited for a person. The as-of question came back with two answers. Only duplicate pairs had an agent.

What changes

The same governor, more queues (queue_agent.rs in the server). The govern job runs in order:

  1. duplicate rounds;
  2. low-confidence and stale facts, in batches of eight;
  3. open conflicts.

It uses the same switch, the same fuse (reverts of any kind count), and the same hourly scan (governance::due also looks for fact and conflict backlog). Extraction on a governed base enqueues the job after every document.

Each queue's own actions, through the people's paths:

  • Facts: confirm goes through temporal::set_confidence; reject goes through temporal::retract.
  • Conflicts:
    • close_old, keep_both and reject_new go through temporal::resolve_conflict.
    • retime_new goes through correct_interval + reconcile. It is for a successor whose start was taken from the wrong date.

The model decides, the server checks (utopia_extract::queue_agent builds the prompts). Each item carries its evidence (document, the document's own date, quote) and recent human decisions of that queue as precedents. At AUTO_CONF an action is applied; below it, it is proposed. Nothing the model says is taken on trust where it can be checked:

  • A date for close_old or retime_new must be the fact's start, a document's own date, or a day the quote writes out.
  • A stale fact is confirmed only by a quote that appears in the current version of its document; that quote becomes its new evidence.

A failed check holds the item as a proposal that says why.

Every applied action is revertible. agent_decisions.detail records the undo:

Action Undo
Confirm a low-confidence fact Restore the prior confidence
Confirm a stale fact Remove the attached evidence
Reject a fact temporal::restore
close_old, retime_new temporal::undo_rewrite, then the original arrives again so its conflict is recorded again
keep_both, reject_new Reopen the conflict (and restore the fact for reject_new)

summary keeps what the item looked like when decided. A person answers from the Agent queue (POST /review/agent/{id}): accept, override with the queue's own action, or revert.

Confidence is part of the timeline. set_confidence locks the fact's timelines, changes the value and recomputes. graph::confirm_fact, the human button, now uses it too, so confirming a doubtful successor lets it take over. A low_confidence conflict the recompute no longer holds is withdrawn.

  • Migration 0058: widens the agent_decisions target and action checks, and adds summary and detail.
  • Decision records: 0043 is new, the index row is added, and 0025's "other queues" open question points at it.

Interface (minimal, needs a look before merge):

  • Agent rows for facts and conflicts show their summary instead of two names.
  • A proposal offers that queue's own actions.
  • Revert appears on any applied fact or conflict decision.
  • en/zh labels for the new actions.

Checks

  • the_agent_settles_facts_and_conflicts_and_a_person_can_take_it_back (server, scripted model endpoint). The fixture is the lease shape: a 0.9 deadline, a 0.7 successor, an unsupported fact, and two tenants starting on the same day. The later tenant's evidence says "became the tenant on June 1, 2021".
    • After the governor runs: the successor is confirmed and takes over, and its conflict is withdrawn. The unsupported fact is rejected. The later tenant is retimed to 2021-06-01, and the first tenant ends there. No conflicts remain open, and there are three applied rows with summaries.
    • After a person reverts all three: the deadline is back to 0.7 and the predecessor is open again. The fact is back. The tenant's start is back, and the simultaneous conflict is open again. The second revert trips the fuse.
  • Unit tests:
    • dates accepted only from evidence;
    • audit action names;
    • prompt shape for both queues, and verdict parsing;
    • precedent rendering.
  • Suites: cargo test -p utopia-store 264, -p utopia-server 281, -p utopia-extract 57, on a fresh database migrated to 0058.
  • Lint: clippy -D warnings and fmt are clean.
  • Web: style guard, tsc, vitest 51.
  • The lease bench on a build with this, A confident merge waits for no one, and a succession clashes with nothing #696 and A value written in a table is stated #697 follows.

🤖 Generated with Claude Code

WaylandYang and others added 3 commits September 14, 2026 14:09
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
@WaylandYang

Copy link
Copy Markdown
Contributor Author

Review before the dev → main merge. Holding this PR out of the release. It merges cleanly with current dev (0058 and 0043 are free), and the tests are green: 618 passed, clippy and fmt clean, web build and vitest pass. But local experiments with a scripted model reproduced these problems:

  1. After a person reverts, the next run does the same thing again. Reverting retime_new or confirm re-records the conflict under a new id. conflict_queue (store queue_agent.rs:165-179) excludes only conflict ids it has already seen, so the new one goes straight back to the agent, and a revert doesn't count as a precedent. In E1 the second run moved B Corp's start to 2021-06-01 again. Fix: judge "already looked at" by both facts' supersedes lineage, as LOOKED_AT does for facts, and count a revert as a precedent.
  2. Actions are applied from the snapshot taken before the model call, so they can overwrite a person's decision. In E3 a person confirmed Tower 9 to 1.0 while the model was thinking, and the governor then rejected it anyway (server queue_agent.rs:254-285, 427-456). Fix: re-check under the lock that the item is still in its queue (still below threshold, still stale, conflict still open).
  3. tidy wrongly withdraws a conflict on a relation unique on both sides, and nothing brings it back. This path runs on every reconcile, whether governance is on or not. still_held (temporal.rs 550-602) only covers pairs held on the timeline being tidied. In E4: A leads P (0.9, 2020) and B leads P (0.7, 2021) with a low-confidence conflict; then B leads Q (2022). Tidying B's side marks the conflict withdrawn while A–P and B–P still overlap. Re-tidying P's side can't re-record it, because the withdrawn row hits ON CONFLICT DO NOTHING. withdrawn also means "one side was invalidated" in 0051.
  4. One malformed reply blocks the whole queue. "i":"0" fails strict serde parsing, the ? fails the job with no decisions recorded, and every rerun repeats the same call (E2). confidence: 60 is clamped to 1.0 and acted on. Fix: parse item by item, record a bad item as unsure, and treat confidence > 1 as invalid.
  5. A revert after a later rewrite silently does nothing. set_confidence returning None and undo_rewrite returning false are ignored (server queue_agent.rs:672-676). The decision is marked reverted and counts toward the fuse (E5).
  6. The stale-fact check can't fail. current_text picks chunks by the subject's name, so a quote containing the subject's name always passes, even when the current text says the opposite (E6). When no chunk matches, the prompt tells the model "the current version no longer mentions the subject" and the system prompt says to reject. Contract names like "Lease Agreement dated May 16, 2016, as amended" rarely appear verbatim, so this can reject in bulk. Fix: require the quote to contain the object value, say plainly when no chunk matched, and never reject on that alone.
  7. Nodded memory facts can still be rejected by the agent. pending.rs stores the extraction confidence (possibly < 0.75), and the low-confidence queue (store queue_agent.rs:47-67) doesn't exclude them. That goes against nods staying with people.
  8. Reverting reject_new only reopens the target conflict. The other conflicts resolve_conflict closed along with it stay resolved.

Smaller issues:

  • A person's answer in the Agent queue is recorded without subject, predicate and object, so the precedent reads as a bare reason.
  • For a fact that is both stale and low-confidence, confirm adds evidence but doesn't raise confidence.
  • answer() scans conflict_queue(i64::MAX) for a target it can never find.
  • A human decision in the normal queue doesn't settle the matching agent suggestion.
  • Batch runs have no daily budget.
  • In the UI, the unsure conflict card's primary button is the destructive reject_new, there's no way to enter a date for close or retime, confirming a stale fact without a quote errors, and zh agentEmpty still says "一对".

Item 3 needs fixing before anything else merges on top of this, since it changes reconcile for every base.

🤖 Generated with Claude Code

@WaylandYang

Copy link
Copy Markdown
Contributor Author

Closing for now. Since this was opened, extraction moved to the open graph (#731, #735, #736): typed facts come only from alignment (0044 cut 2), so the queues this PR governs (facts, conflicts) will be fed by the alignment layer and the typed path this branch edits is gone. 0043 stays the decision; it re-lands on top of the alignment slices rather than being rebased across the rewrite. The branch is kept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant