Skip to content

cratis prologue interpret uses the global language model when the local configuration disables it, and offers no way to turn the model off #241

Description

@woksin

Problem

cratis prologue interpret sends a summary of the captured system to a hosted language model whenever a model is configured anywhere on the machine, even when the project's cratis-prologue.json says llm.enabled: false. There is no flag to force heuristics-only interpretation, and the command does not tell the user which provider it is about to use.

The summary includes concrete HTTP paths (with ids and query strings), table and column names, telemetry span names and schema table names from the capture. For a production system this can be sensitive. A person who has disabled the model for a specific system, or a script that must never contact a hosted service, cannot rely on the local setting.

Evidence

At cli a327e89:

  • LlmOptionsResolver.Resolve returns the local options only when prologueConfiguration?.Llm.Enabled == true. When the local value is false (or there is no local file), it falls through to the llm section of ~/.cratis/config.json (written by cratis llm use) and enables that provider.
  • InterpretPrologueSettings defines only PATH, --file and --prologue-id. cratis prologue interpret --help (3.27.0) lists no option to disable the model.
  • InterpretPrologueCommand prints a message only when no model is configured. When one is, the only trace is a "Refining with …" progress line shown in table/plain output (ConsoleInterpreterCallbacks.cs:77); -o json and -q show nothing, and the model id and endpoint are never printed.
  • Documentation/reference/prologue.md documents the order (local file when enabled, then global, then heuristics) but does not say that a local enabled: false is ignored.
  • The evidence sent to the model is assembled in Prologue's EvidenceFormatter (up to 50 entries per category).

Expected behaviour

  1. cratis prologue interpret --no-llm never creates a chat client and never contacts a model, whatever the local or global configuration says. The output says heuristics-only mode was used.
  2. A local llm.enabled: false in cratis-prologue.json is respected: it disables the model and does not fall back to the global configuration. (Absence of the setting keeps today's fallback.)
  3. Before any evidence is sent, the command states the effective provider: kind, model id, endpoint host, and where the setting came from (local file or global config). In -o json this is part of the result (for example llm: { used: true, kind, model, endpointHost, source }), also when no model is used.
  4. When running non-interactively (-y, -o json, --quiet, or no TTY) and a model would be used because of the global configuration only, the command either refuses unless the model was explicitly requested, or prints the notice to stderr. The maintainers pick one; the notice is the minimum.

Suggested approach

  • Add --no-llm to InterpretPrologueSettings; short-circuit before LlmOptionsResolver.Resolve.
  • Change the resolver to return Enabled = false when the local file explicitly sets enabled: false. The configuration type has to distinguish "not set" from "false" for this; if LlmOptions.Enabled cannot, read the raw JSON property.
  • Add the effective-provider block to the result object and the table output.
  • Update prologue.md and the [LlmDescription]/[LlmOption] attributes so tool-using assistants see the option.

Acceptance criteria

  • --no-llm produces a heuristic-only Screenplay even when a global provider is configured, and makes no outbound model request.
  • A local llm.enabled: false beats a configured global provider; an unset local value still falls back to the global one.
  • The effective provider (or "none") and its source appear in table, plain and JSON output.
  • Specs cover the resolver for: local on, local off, local unset with global set, nothing set, and --no-llm.
  • Documentation/reference/prologue.md describes the new option and the local-off rule.

Status

State: Done
Decisions: --no-llm forces heuristics-only; an explicit local false blocks the machine-wide fallback; an absent local setting keeps it; the provider and destination are announced before use, and an unnameable endpoint is rejected.
Delivered: - #289 (merged).

Activity

  1. added theissue type on Oct 5, 2026
  2. self-assigned this
    on Oct 8, 2026
  3. woksin commented on Oct 8, 2026

    @woksin
    CollaboratorAuthor

    Delivered in #289: prologue interpret honours the project's llm.enabled: false, --no-llm forces heuristics-only interpretation, and the provider and destination are named before anything is sent. Follow-ups: none.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions