From fbeaf06b95cd74a0c51c07c652d438aafa8947ef Mon Sep 17 00:00:00 2001 From: Wolfvin Date: Mon, 29 Jun 2026 16:15:21 +0000 Subject: [PATCH] fix(#38): single-source-of-truth for command + MCP tool counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Problem (issue #38): the CodeLens command count was hardcoded in 8 different places across the repo and had drifted to 5 different values (41, 45, 56, 57, 60). The actual runtime count is 64 commands. The existing regression test (tests/test_integration.py L281) used a loose `>= 41` assertion that would not catch silent command loss — every command could disappear and the test would still pass. Solution — single source of truth + strict enforcement: 1. scripts/sync_command_count.py (new) — reads COMMAND_REGISTRY and _TOOL_DEFINITIONS at runtime and rewrites every doc/metadata file with the correct counts. Supports --check (CI mode, exit 1 on drift) and --apply (write mode). KISS: one helper, regex-driven, no codegen/build system. 2. scripts/codelens.py — added --command-count flag that prints the runtime count and exits; --help description now includes the count so introspection works without parsing subcommand lists. 3. tests/test_integration.py — the loose `>= 41` assertion is now strict `== EXPECTED_COMMAND_COUNT` (== 64). Updated with a clear docstring explaining the update procedure when the count intentionally changes. 4. tests/test_command_count.py (new) — four enforcement tests: - test_command_count_helper_matches_runtime_registry - test_mcp_tool_count_math_is_consistent (total = cmd - {watch,serve}; static + dynamic = total) - test_all_docs_in_sync_with_command_registry (runs sync --check; fails on any drift) - test_sync_helper_idempotent_after_apply (running --apply twice is a no-op the second time) 5. CONTRIBUTING.md — added 'Syncing Command Counts' section so contributors know the procedure when adding/removing commands. Files synced by sync_command_count.py --apply (8 files, 19 substitutions): - README.md (4 locations: feature blurb, --help comment, file tree x2) - SKILL.md (front-matter description + MCP tool count) - SKILL-QUICK.md (All N Commands, Total line, MCP Server section x4) - pyproject.toml (description field) - skill.json (description field) - scripts/mcp_server.py (module docstring 'all N+ commands') - scripts/graph_model.py (docstring 'All N existing CLI commands') - tests/test_integration.py (module docstring 'all N commands') Actual counts (runtime): - CLI commands: 64 (len(COMMAND_REGISTRY)) - MCP tools: 62 total = 51 static (len(_TOOL_DEFINITIONS)) + 11 dynamic (excluded from MCP: watch, serve — long-running) Test results: - tests/test_command_count.py: 4/4 pass - tests/test_integration.py::TestModuleStructure: 4/4 pass (incl. strict assertion) - tests/test_integration.py (full file): 95/95 pass - tests/ (full suite minus test_integration.py): 843 passed, 12 skipped, 1 pre-existing failure (test_architecture::test_auto_scans_on_fresh_workspace — fixture bug unrelated to issue #38; verified failing on pristine main) Closes #38. --- CONTRIBUTING.md | 37 +++- README.md | 12 +- SKILL-QUICK.md | 12 +- SKILL.md | 4 +- pyproject.toml | 2 +- scripts/codelens.py | 30 ++- scripts/graph_model.py | 2 +- scripts/mcp_server.py | 2 +- scripts/sync_command_count.py | 338 ++++++++++++++++++++++++++++++++++ skill.json | 2 +- tests/test_command_count.py | 135 ++++++++++++++ tests/test_integration.py | 34 +++- 12 files changed, 588 insertions(+), 22 deletions(-) create mode 100644 scripts/sync_command_count.py create mode 100644 tests/test_command_count.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 60a327f..69c714f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -39,10 +39,45 @@ CodeLens uses a modular engine architecture. To add a new analysis capability: 4. **Implement the engine** following the pattern of existing engines (return `{status, workspace, findings, summary}`) 5. **Add a command module** in `commands/yourfeature.py` with `add_args(subparser)` and `execute(args)` functions 6. **Add tests** in `tests/` -7. **Update documentation** in `SKILL.md`, `SKILL-QUICK.md`, `README.md`, and `CHANGELOG.md` +7. **Sync command counts** — see "Syncing Command Counts" below; do NOT hand-edit the count in `README.md`, `SKILL.md`, `SKILL-QUICK.md`, `pyproject.toml`, `skill.json`, or `scripts/mcp_server.py` +8. **Update documentation** in `SKILL.md`, `SKILL-QUICK.md`, `README.md`, and `CHANGELOG.md` Commands auto-register via `commands/__init__.py` — no manual wiring needed. +### Syncing Command Counts (issue #38) + +The number of CLI commands and MCP tools must never be hand-edited in +documentation or metadata files — it drifts every time a command is added or +removed. The single source of truth is the runtime `COMMAND_REGISTRY` (and +`_TOOL_DEFINITIONS` for MCP static tools). The `scripts/sync_command_count.py` +helper propagates the runtime count into every doc/metadata file. + +When you add or remove a command: + +```bash +# 1. Run the sync helper in --check mode to see what would change: +PYTHONPATH=scripts python3 scripts/sync_command_count.py --check + +# 2. Apply the changes: +PYTHONPATH=scripts python3 scripts/sync_command_count.py --apply + +# 3. Update the strict regression sentinel in tests/test_integration.py +# (TestModuleStructure.test_command_registry_has_all_commands) +# to match the new len(COMMAND_REGISTRY). This is the ONE place where +# the count is intentionally hardcoded — it is the regression anchor. + +# 4. Verify: +PYTHONPATH=scripts python3 -m pytest tests/test_command_count.py tests/test_integration.py::TestModuleStructure -v +``` + +The test suite enforces this in CI: + +- `tests/test_command_count.py::test_all_docs_in_sync_with_command_registry` + fails if any doc/metadata file mentions a stale count. +- `tests/test_integration.py::TestModuleStructure::test_command_registry_has_all_commands` + fails if `len(COMMAND_REGISTRY)` changes in either direction (strict `==`, + not `>=`). + ### Adding New Language Parsers 1. **Check tree-sitter support** for the language diff --git a/README.md b/README.md index 2a023bf..5d16e62 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,12 @@ > **Before an AI writes a new class/id/function, CodeLens must be checked. This is not optional.** -CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 57 CLI commands, an MCP server with 55 tools (50 static + 5 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). +CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 64 CLI commands, an MCP server with 62 tools (51 static + 11 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). ## Features -- **57 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection -- **MCP Server (55 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 50 statically-defined tools + 5 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) +- **64 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection +- **MCP Server (62 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 51 statically-defined tools + 11 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) - **Token-Efficient Compact Output (v8.2, issue #17)** — `--format compact` produces single-char-key JSON with abbreviated types, omitted null fields, and relative paths — ~50% smaller than `json` on real trace output. Combined with `--limit`/`--offset` pagination, 5 structural queries now cost <5k tokens (down from 30-80k) - **AST Taint Engine** — Tree-sitter based taint analysis with return-value propagation, scope hierarchy, and branch condition refinement - **Live CVE/OSV Scanning** — Real-time vulnerability data from OSV.dev API with SQLite cache, 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex) @@ -225,8 +225,8 @@ codelens/ │ ├── changelog.md # Older changelog (per-version highlights) │ └── agent-integration.md # AI agent integration guide ├── scripts/ -│ ├── codelens.py # CLI entry point (56 commands registered) -│ ├── mcp_server.py # MCP JSON-RPC server (54 tools) +│ ├── codelens.py # CLI entry point (64 commands registered) +│ ├── mcp_server.py # MCP JSON-RPC server (62 tools) │ ├── registry.py # Registry read/write/build │ ├── persistent_registry.py # SQLite persistent storage (WAL mode) │ ├── base_parser.py # Base tree-sitter parser @@ -283,7 +283,7 @@ codelens/ │ ├── plugin_system.py # Plugin system & marketplace │ ├── pre_commit_hook.py # Git pre-commit hook integration │ ├── utils.py # Shared utilities (version, helpers) -│ ├── commands/ # One file per CLI command (auto-registered, 57 commands incl. graph-schema) +│ ├── commands/ # One file per CLI command (auto-registered, 64 commands) │ ├── formatters/ # Output formatters (markdown, sarif, compact) │ ├── parsers/ # Tree-sitter + fallback parsers │ │ ├── html_parser.py, css_parser.py, js_frontend_parser.py, js_backend_parser.py diff --git a/SKILL-QUICK.md b/SKILL-QUICK.md index 31b8f31..724ad8d 100755 --- a/SKILL-QUICK.md +++ b/SKILL-QUICK.md @@ -113,7 +113,7 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co | "Cross-file taint" | `dataflow` | `taint` (taint is single-file, AST-deep) | | "Auto-fix issues" | `fix` | `check` (check just gates, doesn't fix) | -## All 56 Commands +## All 64 Commands ### Setup & Lifecycle (8+) `init` · `scan [--incremental] [--max-files N] [--full]` · `validate` · `detect` · `watch [--debounce SECS] [--git-mode] [--interval SECS]` · `git-status` · `migrate` · `serve` · `lsp-status` @@ -145,9 +145,9 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co ### Tooling (1) `plugin ` -**Total: 60 commands** (56 original + `graph-schema` #17 + `architecture` #19 + `resolve-types` #13 + `git-status` #14; verified via `commands/__init__.py` auto-registration) +**Total: 64 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) -## MCP Server (58 Tools) +## MCP Server (62 Tools) Start the MCP server for AI agent integration: @@ -155,9 +155,9 @@ Start the MCP server for AI agent integration: python3 scripts/codelens.py serve ``` -Exposes 58 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): -- 51 statically-defined tools (full JSON schemas in `mcp_server.py`) including `codelens_graph_schema` (#17), `codelens_architecture` (#19), `codelens_resolve_types` (#13), and `codelens_git_status` (#14) -- 7 dynamically-discovered tools (`benchmark`, `dashboard`, `history`, `lsp-status`, `migrate`, `diff`, `resolve-types`) +Exposes 62 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): +- 51 statically-defined tools (full JSON schemas in `mcp_server.py`) +- 11 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) - Every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`). Use `format: "compact"` for token-efficient responses (~50% smaller than `json`). - `watch` and `serve` itself are excluded (long-running) diff --git a/SKILL.md b/SKILL.md index 4435803..8106e7e 100755 --- a/SKILL.md +++ b/SKILL.md @@ -1,10 +1,10 @@ --- name: codelens description: > - CodeLens — AI-Native Code Intelligence. 56 commands for AI-powered code analysis, + CodeLens — AI-Native Code Intelligence. 64 commands for AI-powered code analysis, security auditing, quality scoring, AST-based taint analysis, live CVE scanning, and pre-write safety checks. Supports 28+ languages with tree-sitter + regex - fallback parsing. MCP server exposes 54 tools for AI agent integration. + fallback parsing. MCP server exposes 62 tools for AI agent integration. For quick command reference with validated output schemas, see SKILL-QUICK.md. For version history, see CHANGELOG.md. --- diff --git a/pyproject.toml b/pyproject.toml index 59f5ad9..c1d186b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta" [project] name = "codelens" version = "8.2.0" -description = "Live Codebase Reference Intelligence — 45 commands for AI-powered code analysis, security auditing, and quality scoring" +description = "Live Codebase Reference Intelligence — 64 commands for AI-powered code analysis, security auditing, and quality scoring" readme = "README.md" license = {text = "MIT"} requires-python = ">=3.8" diff --git a/scripts/codelens.py b/scripts/codelens.py index 83b63a6..e8deb40 100755 --- a/scripts/codelens.py +++ b/scripts/codelens.py @@ -769,8 +769,28 @@ def compute_confidence_distribution_flat(result: Dict[str, Any]) -> Dict[str, in # ─── CLI Entry Point ────────────────────────────────────────── def main(): + # Command count is derived from COMMAND_REGISTRY at runtime so it can never + # drift from the actual number of registered commands (issue #38). The + # `--command-count` flag below prints it for scripts / CI; the description + # also includes it so `--help` is self-documenting. + from commands import COMMAND_REGISTRY as _cli_registry_for_count + _command_count = len(_cli_registry_for_count) + parser = argparse.ArgumentParser( - description=f"CodeLens v{CODELENS_VERSION} — Live Codebase Reference Intelligence (Tree-sitter Edition)" + description=( + f"CodeLens v{CODELENS_VERSION} — Live Codebase Reference Intelligence " + f"(Tree-sitter Edition). {_command_count} commands available; run " + f"`python3 scripts/codelens.py --command-count` to print just the count." + ) + ) + # Quick introspection flag — prints the runtime command count and exits. + # Used by tests / CI / sync_command_count.py to verify the registry size. + parser.add_argument( + "--command-count", + action="store_true", + default=False, + help="Print the runtime command count (len(COMMAND_REGISTRY)) and exit. " + "Single source of truth for issue #38 reconciliation.", ) subparsers = parser.add_subparsers(dest="command", help="Available commands") @@ -836,6 +856,14 @@ def main(): print(json.dumps({"status": "error", "error": str(e)}, indent=2)) sys.exit(0) + # Handle --command-count as a special top-level flag (issue #38): + # prints just the runtime command count and exits. Used by tests, CI, + # and sync_command_count.py to verify the registry size without parsing + # the full --help output. + if "--command-count" in sys.argv: + print(_command_count) + sys.exit(0) + # Pre-parse to capture global flags before subparser overwrites them global_format = None global_top = None diff --git a/scripts/graph_model.py b/scripts/graph_model.py index 1df96a5..ef008b9 100644 --- a/scripts/graph_model.py +++ b/scripts/graph_model.py @@ -13,7 +13,7 @@ - New tables `graph_nodes` and `graph_edges` are additive (prefixed `graph_` to avoid colliding with any existing table name). - The flat registry tables and JSON files are untouched. -- All 56 existing CLI commands continue to work unchanged. +- All 64 existing CLI commands continue to work unchanged. Schema: graph_nodes( diff --git a/scripts/mcp_server.py b/scripts/mcp_server.py index b27bfb0..817c93a 100644 --- a/scripts/mcp_server.py +++ b/scripts/mcp_server.py @@ -4,7 +4,7 @@ Implements the MCP specification (2025-03-26) over stdio (JSON-RPC 2.0). Provides persistent server mode with in-memory registry caching, sub-millisecond -query latency after initial scan, and automatic tool discovery for all 45+ CodeLens commands. +query latency after initial scan, and automatic tool discovery for all 64 CodeLens commands. Usage: python3 codelens.py serve # Start MCP server (stdio transport) diff --git a/scripts/sync_command_count.py b/scripts/sync_command_count.py new file mode 100644 index 0000000..fe3218e --- /dev/null +++ b/scripts/sync_command_count.py @@ -0,0 +1,338 @@ +#!/usr/bin/env python3 +""" +sync_command_count.py — Single-source-of-truth synchronizer for command counts. + +Why this exists +--------------- +Issue #38: CodeLens command count was hardcoded in 7+ places across the repo +(README, SKILL, SKILL-QUICK, pyproject.toml, skill.json, mcp_server.py, +graph_model.py, test_integration.py). The numbers drifted (41, 45, 56, 57, 60) +and none matched the actual runtime count. Updating them by hand every time a +command is added is exactly what caused the drift in the first place. + +This script reads the runtime ``COMMAND_REGISTRY`` (the canonical source of +how many commands CodeLens actually has) and ``_TOOL_DEFINITIONS`` (the +canonical source of MCP static tool definitions) and rewrites every +documentation / metadata file to use the correct numbers. + +Usage +----- +:: + + python3 scripts/sync_command_count.py # alias for --check + python3 scripts/sync_command_count.py --check # exit 1 if any file would change + python3 scripts/sync_command_count.py --apply # write changes to disk + +When to run +----------- +- After adding or removing a command module in ``scripts/commands/`` +- In CI (via ``tests/test_command_count.py``) to catch drift before merge +- Manually, before tagging a release + +This script is the ONLY place that knows the full list of files that mention +command / MCP tool counts. The documentation files themselves never hardcode +the number — they always get it from ``COMMAND_REGISTRY`` via this script. + +The strict regression sentinel (``EXPECTED_COMMAND_COUNT``) lives in +``tests/test_integration.py`` and is intentionally NOT touched by this script — +that sentinel is the test's job, not the docs' job. +""" + +from __future__ import annotations + +import argparse +import os +import re +import sys +from typing import Dict, List, Tuple + +# Make scripts/ importable when this file is run directly (no PYTHONPATH set). +_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) +if _SCRIPT_DIR not in sys.path: + sys.path.insert(0, _SCRIPT_DIR) + +from commands import COMMAND_REGISTRY # noqa: E402 (path inserted above) + +# ``_TOOL_DEFINITIONS`` is the static MCP tool registry. Importing it also +# validates that the MCP server module is intact. If the import fails (e.g. +# tree-sitter missing in a stripped-down environment), MCP count sync is +# skipped — but command count sync still runs. +try: + from mcp_server import _TOOL_DEFINITIONS # type: ignore # noqa: E402 + _MCP_AVAILABLE = True +except Exception: # pragma: no cover - defensive + _TOOL_DEFINITIONS = {} # type: ignore + _MCP_AVAILABLE = False + + +# Commands that are NOT exposed as MCP tools (long-running, would block the +# JSON-RPC server). Must match ``MCPServer._get_dynamic_tools`` exclusion. +_MCP_EXCLUDED_COMMANDS = {"watch", "serve"} + + +def get_command_count() -> int: + """Return the canonical CodeLens CLI command count. + + This is the single source of truth. Every doc, metadata file, and test + sentinel must reconcile to this number. + """ + return len(COMMAND_REGISTRY) + + +def get_mcp_counts() -> Tuple[int, int, int]: + """Return ``(total, static, dynamic)`` MCP tool counts. + + - ``total`` = every command except the long-running exclusions + (``watch`` + ``serve``) + - ``static`` = commands with an explicit schema in + ``mcp_server._TOOL_DEFINITIONS`` + - ``dynamic`` = ``total - static`` (auto-discovered from COMMAND_REGISTRY) + """ + if not _MCP_AVAILABLE: + return (0, 0, 0) + total = sum(1 for name in COMMAND_REGISTRY if name not in _MCP_EXCLUDED_COMMANDS) + static = len(_TOOL_DEFINITIONS) + dynamic = total - static + return (total, static, dynamic) + + +# ─── Replacement rules ──────────────────────────────────────────────────── +# +# Each entry is ``(relative_path, regex, template)``. +# +# - ``regex`` — compiled with ``re.MULTILINE``; must match the FULL phrase +# containing the digit(s) to be replaced, so that no other +# digits on the same line get clobbered. +# - ``template``— uses ``{cmd}`` / ``{mcp}`` / ``{static}`` / ``{dynamic}`` +# placeholders, filled from the runtime counts above. +# +# When adding a new place that mentions command/MCP counts, ADD A RULE HERE. +# Do not introduce a second sync mechanism. + +_PROJECT_ROOT = os.path.dirname(_SCRIPT_DIR) + + +def _rel(*parts: str) -> str: + return os.path.join(_PROJECT_ROOT, *parts) + + +def _build_rules() -> List[Tuple[str, str, str]]: + """Return the list of ``(file, regex, template)`` sync rules.""" + return [ + # ─── README.md ─────────────────────────────────────────────── + # L5: "through N CLI commands, an MCP server with M tools (S static + D dynamic)" + (_rel("README.md"), + r"through \d+ CLI commands, an MCP server with \d+ tools \(\d+ static \+ \d+ dynamic\)", + "through {cmd} CLI commands, an MCP server with {mcp} tools ({static} static + {dynamic} dynamic)"), + # L9: "- **N CLI Commands** — From basic ..." + (_rel("README.md"), + r"\*\*\d+ CLI Commands\*\*", + "**{cmd} CLI Commands**"), + # L10: "- **MCP Server (M Tools)** — ... S statically-defined tools + D dynamically discovered" + (_rel("README.md"), + r"\*\*MCP Server \(\d+ Tools\)\*\* — Native AI agent integration via Model Context Protocol \(JSON-RPC over stdio\), \d+ statically-defined tools \+ \d+ dynamically discovered", + "**MCP Server ({mcp} Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), {static} statically-defined tools + {dynamic} dynamically discovered"), + # L228: "CLI entry point (N commands registered)" + (_rel("README.md"), + r"CLI entry point \(\d+ commands registered\)", + "CLI entry point ({cmd} commands registered)"), + # L229: "MCP JSON-RPC server (M tools)" + (_rel("README.md"), + r"MCP JSON-RPC server \(\d+ tools\)", + "MCP JSON-RPC server ({mcp} tools)"), + # L286: "(auto-registered, N commands incl. graph-schema)" — drop the + # stale "incl. graph-schema" suffix; the count already includes it. + (_rel("README.md"), + r"\(auto-registered, \d+ commands incl\. graph-schema\)", + "(auto-registered, {cmd} commands)"), + + # ─── SKILL.md ──────────────────────────────────────────────── + # L4: "N commands for AI-powered code analysis" + (_rel("SKILL.md"), + r"\d+ commands for AI-powered code analysis", + "{cmd} commands for AI-powered code analysis"), + # L7: "MCP server exposes M tools for AI agent integration." + (_rel("SKILL.md"), + r"MCP server exposes \d+ tools for AI agent integration", + "MCP server exposes {mcp} tools for AI agent integration"), + + # ─── SKILL-QUICK.md ────────────────────────────────────────── + # L116: "## All N Commands" + (_rel("SKILL-QUICK.md"), + r"^## All \d+ Commands", + "## All {cmd} Commands"), + # L148: "**Total: N commands** ..." — collapse the entire stale + # parenthetical (the per-issue breakdown is wrong and not maintained). + (_rel("SKILL-QUICK.md"), + r"\*\*Total: \d+ commands\*\* \(.*?\)", + "**Total: {cmd} commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command)"), + # L150: "## MCP Server (M Tools)" + (_rel("SKILL-QUICK.md"), + r"^## MCP Server \(\d+ Tools\)", + "## MCP Server ({mcp} Tools)"), + # L158: "Exposes M tools as `codelens_` ..." + (_rel("SKILL-QUICK.md"), + r"Exposes \d+ tools as `codelens_`", + "Exposes {mcp} tools as `codelens_`"), + # L159: "- N statically-defined tools (full JSON schemas ...)" + (_rel("SKILL-QUICK.md"), + r"^- \d+ statically-defined tools \(full JSON schemas in `mcp_server.py`\) including .*$", + "- {static} statically-defined tools (full JSON schemas in `mcp_server.py`)"), + # L160: "- D dynamically-discovered tools (...)" + (_rel("SKILL-QUICK.md"), + r"^- \d+ dynamically-discovered tools \(.*\)$", + "- {dynamic} dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded)"), + + # ─── pyproject.toml ────────────────────────────────────────── + # L8: description = "... N commands for AI-powered code analysis ..." + (_rel("pyproject.toml"), + r"\d+ commands for AI-powered code analysis", + "{cmd} commands for AI-powered code analysis"), + + # ─── skill.json ────────────────────────────────────────────── + # L4: "description": "Live Codebase Reference Intelligence. N commands ..." + (_rel("skill.json"), + r"\d+ commands for AI-powered code analysis", + "{cmd} commands for AI-powered code analysis"), + + # ─── scripts/mcp_server.py ─────────────────────────────────── + # L7: "... automatic tool discovery for all N+ CodeLens commands." + (_rel("scripts", "mcp_server.py"), + r"automatic tool discovery for all \d+\+ CodeLens commands", + "automatic tool discovery for all {cmd} CodeLens commands"), + + # ─── scripts/graph_model.py ────────────────────────────────── + # L16 (docstring): "All N existing CLI commands continue to work unchanged." + (_rel("scripts", "graph_model.py"), + r"All \d+ existing CLI commands continue to work unchanged", + "All {cmd} existing CLI commands continue to work unchanged"), + + # ─── tests/test_integration.py ─────────────────────────────── + # L2 (module docstring): "Integration smoke tests for all N CodeLens commands." + # NOTE: the strict sentinel `EXPECTED_COMMAND_COUNT = 64` lower in this + # file is intentionally NOT touched — that is the test's regression + # anchor, not a doc. + (_rel("tests", "test_integration.py"), + r"Integration smoke tests for all \d+ CodeLens commands", + "Integration smoke tests for all {cmd} CodeLens commands"), + ] + + +# ─── Engine ─────────────────────────────────────────────────────────────── + + +def _format_values() -> Dict[str, str]: + cmd = get_command_count() + mcp_total, mcp_static, mcp_dynamic = get_mcp_counts() + return { + "cmd": str(cmd), + "mcp": str(mcp_total), + "static": str(mcp_static), + "dynamic": str(mcp_dynamic), + } + + +def _apply_rule(content: str, regex: str, template: str, values: Dict[str, str]) -> Tuple[str, int]: + """Apply one rule to ``content``. Returns (new_content, num_substitutions).""" + pattern = re.compile(regex, re.MULTILINE) + replacement = template.format(**values) + new_content, n = pattern.subn(replacement, content) + return new_content, n + + +def sync(apply: bool = False) -> int: + """Run all sync rules. + + Args: + apply: If True, write changes to disk. If False, only report drift. + + Returns: + Number of files that had (or would have) changes. + """ + values = _format_values() + rules = _build_rules() + files_changed = 0 + drift_report: List[str] = [] + + # Group rules by file so we read/write each file exactly once. + by_file: Dict[str, List[Tuple[str, str]]] = {} + for filepath, regex, template in rules: + by_file.setdefault(filepath, []).append((regex, template)) + + for filepath, file_rules in sorted(by_file.items()): + if not os.path.exists(filepath): + drift_report.append(f" MISSING: {filepath} (skipped)") + continue + with open(filepath, "r", encoding="utf-8") as f: + original = f.read() + content = original + rule_hits = 0 + for regex, template in file_rules: + content, n = _apply_rule(content, regex, template, values) + rule_hits += n + if rule_hits == 0: + # No rule matched — either the file already matches (good) or the + # regex is stale (bad). We can't tell which, so we just report. + drift_report.append(f" NO_MATCH: {filepath} (no rule matched — already in sync, or regex is stale)") + continue + if content != original: + files_changed += 1 + rel_path = os.path.relpath(filepath, _PROJECT_ROOT) + drift_report.append(f" DRIFT: {rel_path} ({rule_hits} substitution(s))") + if apply: + with open(filepath, "w", encoding="utf-8") as f: + f.write(content) + + # ─── Report ───────────────────────────────────────────────────── + cmd = values["cmd"] + mcp, static, dynamic = get_mcp_counts() + print(f"Command count (runtime COMMAND_REGISTRY): {cmd}") + if _MCP_AVAILABLE: + print(f"MCP tools: {mcp} total = {static} static + {dynamic} dynamic " + f"(excluded from MCP: {sorted(_MCP_EXCLUDED_COMMANDS)})") + else: + print("MCP tools: skipped (mcp_server import failed)") + print() + + if files_changed == 0: + print("All documentation files are in sync with COMMAND_REGISTRY.") + return 0 + + print(f"{files_changed} file(s) {'updated' if apply else 'out of sync'}:") + for line in drift_report: + print(line) + if not apply: + print() + print("Run `python3 scripts/sync_command_count.py --apply` to fix.") + return files_changed + + +# ─── CLI ────────────────────────────────────────────────────────────────── + + +def main(argv: List[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description=( + "Synchronize command/MCP-tool counts in documentation & metadata " + "with the runtime COMMAND_REGISTRY. Single source of truth for " + "issue #38." + ) + ) + mode = parser.add_mutually_exclusive_group() + mode.add_argument( + "--apply", + action="store_true", + help="Write the synchronized counts to disk.", + ) + mode.add_argument( + "--check", + action="store_true", + default=True, + help="Check-only mode (default). Exit 1 if any file would change.", + ) + args = parser.parse_args(argv) + return 1 if sync(apply=args.apply) > 0 and not args.apply else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skill.json b/skill.json index 6961028..06dabf6 100755 --- a/skill.json +++ b/skill.json @@ -1,7 +1,7 @@ { "name": "codelens", "version": "8.2.0", - "description": "Live Codebase Reference Intelligence. 45 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", + "description": "Live Codebase Reference Intelligence. 64 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", "author": "codelens", "command_categories": { "setup": [ diff --git a/tests/test_command_count.py b/tests/test_command_count.py new file mode 100644 index 0000000..84ef42b --- /dev/null +++ b/tests/test_command_count.py @@ -0,0 +1,135 @@ +""" +Doc-sync enforcement tests for command / MCP tool counts (issue #38). + +These tests fail in CI before a PR can merge if any documentation or metadata +file mentions a stale command count. The single source of truth is the runtime +``COMMAND_REGISTRY`` (and ``_TOOL_DEFINITIONS`` for MCP static tool count); the +``scripts/sync_command_count.py`` helper propagates the runtime count into all +docs. + +If any test in this file fails, run:: + + PYTHONPATH=scripts python3 scripts/sync_command_count.py --apply + +and commit the result. The strict sentinel for the command count itself lives +in ``tests/test_integration.py`` (``TestModuleStructure.test_command_registry_has_all_commands``). +""" + +import os +import subprocess +import sys + +import pytest + +# Path to the project root and scripts dir. +_THIS_DIR = os.path.dirname(os.path.abspath(__file__)) +_PROJECT_ROOT = os.path.dirname(_THIS_DIR) +_SCRIPTS_DIR = os.path.join(_PROJECT_ROOT, "scripts") + + +def _run_sync_check() -> subprocess.CompletedProcess: + """Run sync_command_count.py --check as a subprocess and return its result.""" + return subprocess.run( + [sys.executable, os.path.join(_SCRIPTS_DIR, "sync_command_count.py"), "--check"], + capture_output=True, + text=True, + cwd=_PROJECT_ROOT, + env={**os.environ, "PYTHONPATH": _SCRIPTS_DIR}, + timeout=60, + ) + + +def test_command_count_helper_matches_runtime_registry(): + """The sync helper's reported count must equal len(COMMAND_REGISTRY). + + This guards against the helper accidentally hardcoding a different number. + """ + sys.path.insert(0, _SCRIPTS_DIR) + from commands import COMMAND_REGISTRY # type: ignore + from sync_command_count import get_command_count # type: ignore + + assert get_command_count() == len(COMMAND_REGISTRY) + + +def test_mcp_tool_count_math_is_consistent(): + """MCP total = (commands not in {watch, serve}); static + dynamic = total. + + Catches the kind of drift that caused issue #38's MCP tool count + inconsistency (README said 55, SKILL said 54, SKILL-QUICK said 58). + """ + sys.path.insert(0, _SCRIPTS_DIR) + from commands import COMMAND_REGISTRY # type: ignore + from sync_command_count import get_mcp_counts, _MCP_EXCLUDED_COMMANDS # type: ignore + + total, static, dynamic = get_mcp_counts() + expected_total = sum(1 for c in COMMAND_REGISTRY if c not in _MCP_EXCLUDED_COMMANDS) + assert total == expected_total, ( + f"MCP total {total} != expected {expected_total} " + f"(commands minus excluded {sorted(_MCP_EXCLUDED_COMMANDS)})" + ) + assert static + dynamic == total, ( + f"static({static}) + dynamic({dynamic}) != total({total})" + ) + assert dynamic >= 0, f"dynamic count went negative: {dynamic} (static > total)" + + +def test_all_docs_in_sync_with_command_registry(): + """Every doc/metadata file must mention the current command/MCP counts. + + Runs ``sync_command_count.py --check`` and fails if any file would be + changed. This is the test that catches stale numbers before merge. + + Fix failures by running:: + + PYTHONPATH=scripts python3 scripts/sync_command_count.py --apply + """ + result = _run_sync_check() + assert result.returncode == 0, ( + "Documentation is out of sync with COMMAND_REGISTRY.\n" + "--- sync_command_count.py --check stdout ---\n" + f"{result.stdout}\n" + "--- sync_command_count.py --check stderr ---\n" + f"{result.stderr}\n" + "Fix: run `PYTHONPATH=scripts python3 scripts/sync_command_count.py --apply` " + "and commit the result." + ) + + +def test_sync_helper_idempotent_after_apply(): + """Running --apply twice must produce zero changes on the second run. + + Guards against the sync script introducing drift of its own (e.g. a regex + that mangles a previously-synced line into a form the regex no longer + matches). + """ + # First apply to ensure baseline (we use --check first to confirm state; + # if --check passes we're already in sync; if it fails the previous test + # would have failed too). + apply1 = subprocess.run( + [sys.executable, os.path.join(_SCRIPTS_DIR, "sync_command_count.py"), "--apply"], + capture_output=True, + text=True, + cwd=_PROJECT_ROOT, + env={**os.environ, "PYTHONPATH": _SCRIPTS_DIR}, + timeout=60, + ) + assert apply1.returncode == 0, f"first --apply failed: {apply1.stderr}" + + # Second apply must report 0 files changed. + apply2 = subprocess.run( + [sys.executable, os.path.join(_SCRIPTS_DIR, "sync_command_count.py"), "--apply"], + capture_output=True, + text=True, + cwd=_PROJECT_ROOT, + env={**os.environ, "PYTHONPATH": _SCRIPTS_DIR}, + timeout=60, + ) + assert apply2.returncode == 0, f"second --apply failed: {apply2.stderr}" + assert "0 file(s)" in apply2.stdout or "All documentation files are in sync" in apply2.stdout, ( + "Second --apply should be a no-op but reported changes:\n" + f"{apply2.stdout}" + ) + + +if __name__ == "__main__": + pytest.main([__file__, "-v", "--tb=short"]) diff --git a/tests/test_integration.py b/tests/test_integration.py index b4811b3..c9274d4 100644 --- a/tests/test_integration.py +++ b/tests/test_integration.py @@ -1,5 +1,5 @@ """ -Integration smoke tests for all 41 CodeLens commands. +Integration smoke tests for all 64 CodeLens commands. Tests that every command: 1. Runs without crash (valid JSON output) @@ -276,9 +276,39 @@ def test_formatters_dir_exists(self): assert os.path.isdir(os.path.join(SCRIPT_DIR, 'formatters')) def test_command_registry_has_all_commands(self): + """Strict regression sentinel for command count (issue #38). + + The previous assertion (``>= 41``) was trivially satisfied and would + not catch silent command loss — every command could disappear and the + test would still pass. This strict assertion fails whenever the + command count changes in either direction. + + When this test fails, it means a command was added or removed. To fix: + + 1. Confirm the change is intentional (you meant to add/remove a command). + 2. Update ``EXPECTED_COMMAND_COUNT`` below to match the new count. + 3. Run ``python3 scripts/sync_command_count.py --apply`` to propagate + the new count to all documentation and metadata files + (README.md, SKILL.md, SKILL-QUICK.md, pyproject.toml, skill.json, + scripts/mcp_server.py, scripts/graph_model.py, this file's docstring). + 4. Re-run the test suite to confirm green. + + The sentinel is intentionally a literal — it is the one place where + the count is allowed to be hardcoded, because the test's whole purpose + is to detect drift against a frozen reference. + """ sys.path.insert(0, SCRIPT_DIR) from commands import COMMAND_REGISTRY - assert len(COMMAND_REGISTRY) >= 41, f"Expected at least 41 commands, got {len(COMMAND_REGISTRY)}" + # Regression sentinel — see docstring above for update procedure. + EXPECTED_COMMAND_COUNT = 64 + actual = len(COMMAND_REGISTRY) + assert actual == EXPECTED_COMMAND_COUNT, ( + f"Command count drift detected: expected {EXPECTED_COMMAND_COUNT}, " + f"got {actual}. If intentional: (1) update EXPECTED_COMMAND_COUNT " + f"in this test, (2) run " + f"`PYTHONPATH=scripts python3 scripts/sync_command_count.py --apply` " + f"to sync all docs/metadata." + ) def test_fallback_parsers_exist(self): fallback_dir = os.path.join(SCRIPT_DIR, 'parsers')