feat(cli,agent): expose tiered search + taxonomy via CLI and wire into query/chat agent - #261
Open
sebastianbraun25 wants to merge 12 commits into
Open
sebastianbraun25 wants to merge 12 commits into
sebastianbraun25 wants to merge 12 commits into
Conversation
added 5 commits
August 28, 2026 16:25
Adds a dependency-free BM25 full-text index (openkb/fulltext_index.py) over concepts/entities/summaries pages, exposed as a new search_wiki tool alongside index.md-driven navigation in build_query_agent. Additive hybrid retrieval: surfaces pages whose one-line index summary omits a buried detail, without replacing existing navigation. Resolves VectifyAI#233.
…axonomy accessors - fulltext_index.py: extract shared _BM25Scorer from WikiFullTextIndex (no behavior change), add Locator (line/page) on SearchHit, add TieredWikiSearch with three independent tiers over summaries/ (briefs + full body) and sources/ (whole-file .md + per-page PageIndex .json, never the whole long doc as one BM25 unit). - frontmatter.py: add resolve_description()/body_only() shared helpers (kept separate from agent.compiler._resolve_description, which is under active unrelated development). - agent/tools.py: add list_taxonomy_items()/get_taxonomy_item() for semantic browsing of persisted concepts/entities (pending candidates in PendingTopicsStore are structurally excluded). - No wiring into CLI/MCP/query-agent yet (follow-up PRs); WikiFullTextIndex and agent.tools.search_wiki keep their existing signature/behavior.
…o query/chat agent - cli.py: new 'openkb list-taxonomy [--kind concept|entity] [--json]' and 'openkb search <query> [--scope briefs,summaries,sources] [--top-k N] [--json]' commands. - agent/tools.py: search_wiki now searches the new tiered briefs/summaries/sources index instead of the old combined concepts+entities+summaries index (concepts/entities move to the new list_taxonomy tool - semantic browsing, not keyword search); new list_taxonomy() text-formatting wrapper over list_taxonomy_items(). - agent/query.py (+ chat.py via tool inheritance): wires list_taxonomy and the retiered search_wiki in as agent tools; search strategy instructions updated to browse taxonomy first, then use scope-restricted search_wiki as a keyword fallback. - README.md: updated hybrid-retrieval paragraph and command table. - Intentional behavior change to agent.tools.search_wiki (scope param, concepts/entities out of scope, output grouped by tier) - safe since VectifyAI#234/VectifyAI#259, which introduced it, are not yet merged upstream; existing tests updated to the new contract.
sebastianbraun25
pushed a commit
to sebastianbraun25/OpenKB
that referenced
this pull request
Sep 11, 2026
- skills/openkb/SKILL.md: 'See what's available' now leads with list_taxonomy (MCP) / 'openkb list-taxonomy' (CLI) before falling back to reading the full index.md. - 'Read content' table adds search_wiki (MCP) / 'openkb search' (CLI) rows ahead of the existing grep fallback, with a note on why BM25 ranking beats raw grep occurrence count. - 'When the KB doesn't have the answer' and the openkb-query guidance updated to reference the new search options alongside grep. - Documentation-only change; no behavior change to the underlying tools/CLI/MCP server (VectifyAI#259/VectifyAI#261/VectifyAI#263).
…earch tier - agent/tools.py: get_taxonomy_item, read_wiki_file, get_wiki_page_content were four differently-shaped ways to read wiki content (slug+kind vs. path vs. doc_name+pages). Replace with a single get_content(slug, wiki_root, kind=None, pages=None) covering all seven content kinds (concept/entity/summary/exploration/source/report/index). kind=None fans out and returns one ContentEntry per match (mirrors list_taxonomy_items' kind=None semantics) instead of a first-match-wins precedence that would silently drop e.g. a source when a summary shares its slug. - read_wiki_file/get_wiki_page_content (already released, predate this branch) now delegate to get_content — kept for backward compatibility, existing tests unchanged. get_taxonomy_item is removed (unreleased, no callers outside its own tests). - New DocumentItem/list_documents (summaries + explorations), mirroring TaxonomyItem/list_taxonomy_items for a different pair of kinds. - New get_kb_status/KbStatus: structured KB counts (basis for a future MCP get_status tool), without pulling in cli.py's heavier import chain. - fulltext_index.py: add an explorations BM25 tier (TIERED_SCOPES now briefs/summaries/sources/explorations) — its own tier, not merged into summaries, so a hit stays labeled as a saved answer vs. a document summary. - Split agent/tools.py into agent/tools.py + new agent/content.py (tools.py re-exports for backward compatibility) to stay under the 800-line module gate (tests/test_file_size.py) after these additions. - Tests: tests/test_agent_tools.py (get_content across all kinds/error paths, list_documents, get_kb_status), tests/test_fulltext_index.py (explorations tier). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ts, explorations tier) # Conflicts: # tests/test_agent_tools.py
…ons, list_documents agent tool - openkb list-documents [--kind summary|exploration] [--json]: new command mirroring openkb list-taxonomy, exposing the Core PR's list_documents. - openkb search --scope gains 'explorations' (TieredWikiSearch's 4th tier). - openkb list (print_list): deprecated in its help text in favor of list-taxonomy/list-documents; its Summaries/Concepts/Entities sections now call list_documents/list_taxonomy_items internally instead of duplicating directory-glob logic (Documents-registry-table and Reports listing stay as their own logic — they don't map onto the 5/7-kind content model). Output format unchanged for existing scripts. - agent/query.py: new list_documents tool (mirrors list_taxonomy's browse-list style) registered on the query agent — and therefore also the chat agent, which builds on top of it. Instructions updated to recognize an explorations search hit as a previously-saved answer, distinct from a summaries/sources hit, and to check list_documents before re-synthesizing an answer that may already exist. Deferred (not part of this PR, see plan): replacing the internal read_file/ get_page_content tools with the unified get_content — a separate, higher-risk change to an already-productive agent, to be assessed on its own. - tests/test_query.py: tool count/name assertions updated for the new list_documents tool. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
sebastianbraun25
pushed a commit
to sebastianbraun25/OpenKB
that referenced
this pull request
Sep 11, 2026
…kbs, explorations tier - 'First: find where the KB lives': add get_status() as the MCP path for learning the KB's absolute path - previously the only way was 'openkb status' (shell), leaving a pure-MCP client (no shell access) unable to complete this step at all, despite list_taxonomy being documented as MCP-usable right below it. - New 'Multiple knowledge bases' note: list_kbs()/the kb parameter (MCP), --kb-dir (CLI). - 'See what's available': add list_documents/openkb list-documents alongside list_taxonomy/list-taxonomy; mark openkb list as deprecated. - 'Read content' table: add get_content(slug, kind, pages) as the MCP-only alternative to filesystem reads for every row (concept/entity/ summary/exploration/source, incl. paginated PageIndex docs) - for clients without their own filesystem tool (remote MCP, pure chat clients). Search rows updated to mention the explorations tier and --scope/scope list value. - Explorations search hits called out as their own category (a previously-saved answer, not a document summary) to reuse instead of re-synthesizing. - references/commands.md: document list-taxonomy/list-documents/search (previously undocumented there since PR VectifyAI#261/VectifyAI#259 introduced them); mark list as deprecated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
added 3 commits
September 21, 2026 12:06
…nly) search-taxonomy shipped as an 'openkb search-taxonomy' CLI command and later as an MCP tool, but the internal query/chat agent (build_query_agent) never got a matching function_tool -- an omission, not an intentional scope cut. Add agent.tools.search_taxonomy (thin formatter over fulltext_index.TaxonomySearch, mirrors list_taxonomy's shape) and wire it into build_query_agent alongside list_taxonomy, with search-strategy instructions covering when to browse (list_taxonomy) vs. rank (search_taxonomy) a large taxonomy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR was created in collaboration between a human and AI: implementation, tests, and
PR text were created by an AI assistant under the guidance and review of the human author.
Problem
search_wiki/list_taxonomy_items/get_taxonomy_item(#258/#259) are only reachable frominside the
openkb query/openkb chatagent's Python process. There is no way for ascript, a CI job, or an external AI assistant (GitHub Copilot, Claude Code, etc.) that isn't
running the OpenKB agent itself to browse the taxonomy or run the tiered search — the only
options today are re-implementing keyword search with
grep, or paying for a fullopenkb queryLLM round-trip just to get a list of candidate pages. Separately, thequery/chat agent itself still navigated concepts/entities via ad-hoc
index.mdreadsrather than the new, purpose-built taxonomy tools.
Solution / Changes
openkb/cli.py:openkb list-taxonomy [--kind concept|entity] [--json]— lists persistedconcept/entity pages with their one-line briefs (semantic browsing, not keyword
search).
openkb search <query> [--scope briefs,summaries,sources] [--top-k N] [--json]— tieredBM25 search over summaries/sources, grouped by tier, printing each hit's
line/page locator when present.
openkb search-taxonomy <query> [--kind concept|entity] [--top-k N] [--json](latest update) — ranks concept/entity pages by BM25 match against each page's slug
(readable form) + one-line brief only, never their full body (
TaxonomySearch, feat(search): tiered BM25 search (briefs/summaries/sources) + taxonomy accessors #259).Complements
list-taxonomy's plain listing for a KB with too many taxonomy items toscan by eye;
--top-kdefaults to 20 (much higher thansearch's5, since abrief is short).
openkb/agent/tools.py:search_wikinow searches the tieredbriefs/summaries/sourcesindex (newscopeparam) instead of the old combined
concepts+entities+summariesindex — conceptsand entities are no longer keyword-searched at all, only browsed via
list_taxonomy.This is an intentional behavior/contract change, safe because feat(agent): hybrid BM25 search_wiki tool for query/chat agent #234/feat(search): tiered BM25 search (briefs/summaries/sources) + taxonomy accessors #258/feat(search): tiered BM25 search (briefs/summaries/sources) + taxonomy accessors #259 (which
introduced/extended
search_wiki) are not yet merged upstream, so there is no releasedconsumer to break. Existing
TestSearchWikitests updated to the new contract(concepts/entities excluded, output grouped by tier,
scopevalidation).list_taxonomy(wiki_root, kind=None)— text-formatting wrapper overlist_taxonomy_itemsfor LLM tool consumption.openkb/agent/query.py(andchat.py, which extends the query agent's tools):build_query_agentnow exposes 5 tools:read_file,get_page_content,list_taxonomy,search_wiki,get_image.list_taxonomyfor concept/entityquestions (pick a slug by meaning, then
read_filethe page), fall back tosearch_wiki(query, scope)for buried details in summaries/sources, narrowingscopeto
["sources"]for source-only facts (author, date, exact field name) a generatedsummary would likely omit.
README.md: updated hybrid-retrieval paragraph and the wiki-commands table with the CLIcommands, including
search-taxonomy.Testing
tests/test_agent_tools.py:TestSearchWikiupdated (concepts/entities no longermatched, tier-grouped output,
scopeparam, locator in output, invalid-scope errormessage) + new
TestListTaxonomy(3 tests: wikilink formatting,kindfilter, emptycase).
tests/test_query.py: tool count/name assertions updated for the new 5-tool set(
list_taxonomyadded).tests/test_cli.py: newTestSearchTaxonomyCmd(ranking output,--kindfilter,--jsonoutput, no-match message).list-taxonomy(plain +--json+--kind),search(plain +--json+--scope+ invalid-scope error path),search-taxonomy(plain +--json+--kind+--top-k) all verified against a throwaway KB.ruff check,ruff format --check,mypy openkb,pytestall green.Dependencies
Depends on #259(feat/issue-233-tiered-search) — this PR is built on top of thatbranch (
TieredWikiSearch,list_taxonomy_items/get_taxonomy_item,TaxonomySearch),which it wires into the CLI and the query/chat agent without changing their internal
behavior. The diff below includes feat(search): tiered BM25 search (briefs/summaries/sources) + taxonomy accessors #259's (and thus feat(agent): hybrid BM25 search_wiki tool for query/chat agent #234's) commits; once feat(search): tiered BM25 search (briefs/summaries/sources) + taxonomy accessors #259 merges
first, this PR's diff will shrink to just this PR's own commits.
search-taxonomyis independent of feat(mcp): expose taxonomy browsing + tiered search as an MCP server #263 (MCP server) — both exposeTaxonomySearchseparately (CLI here, an MCP tool there) and don't touch the same code.
Issues
Resolves #260