Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions Documentation/reference/prologue.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,13 @@ Reads Prologue capture (`.jsonl`) files and interprets them into a Screenplay. D
|---|---|
| `--file <FILE>` | File to write the generated Screenplay to. Defaults to `<SystemName>.play` in the current directory. |
| `--prologue-id <ID>` | The Prologue the captures belong to. Defaults from `cratis-prologue.json` when one is present. |
| `--no-llm` | Force heuristics-only interpretation. Never creates a chat client or sends capture evidence to a model, regardless of local or global configuration. |

```bash
cratis prologue interpret
cratis prologue interpret ./captures
cratis prologue interpret ./captures --file MySystem.play
cratis prologue interpret ./captures --no-llm
```

On success it prints a panel with the written path, the derived system name, and the module/feature/slice counts — and the natural next step is `cratis run` to boot the Screenplay in a local [Stage](run.md) sandbox.
Expand All @@ -58,9 +60,18 @@ On success it prints a panel with the written path, the derived system name, and

The language model is resolved in this order:

1. The `llm` section of a found `cratis-prologue.json`, when it is enabled there.
2. The `llm` section of `~/.cratis/config.json`, written by [`cratis llm use`](llm.md) — `anthropic`, `openai`, or `local` (OpenAI-compatible).
3. Neither configured — interpretation runs with heuristics only and an info line points at `cratis llm use`.
1. `--no-llm` disables refinement, overriding every configuration setting.
2. An explicit `llm.enabled` in a found `cratis-prologue.json`: `true` uses the local provider; `false` disables refinement **without falling back to the global provider**.
3. When the local `llm.enabled` setting is absent, the `llm` section of `~/.cratis/config.json`, written by [`cratis llm use`](llm.md) — `anthropic`, `openai`, or `local` (OpenAI-compatible).
4. Neither configured — interpretation runs with heuristics only.

With `--no-llm` or local `llm.enabled: false`, no chat client is created and no capture evidence leaves the machine for a model. Use `--no-llm` for scripts that must never contact a model.

Before creating the interpreter session, the command prints the effective provider kind, model id, endpoint host, and setting source to **stderr**. This notice is also printed with `-y/--yes`, JSON output, `--quiet`, or no terminal; a globally configured model is not silently used. Only the endpoint host is shown, never credentials, paths, or query strings. When refinement is disabled, the notice says `none` and heuristics-only mode.

For Anthropic, a custom endpoint takes precedence over `ANTHROPIC_BASE_URL`; an empty endpoint or the default Ollama URL (`http://llm:11434`) is treated as unset. When neither a custom endpoint nor the environment variable is set, the public Anthropic API is used. The resolved endpoint is fixed before the notice and passed to the client. The OpenAI provider uses its public API; other providers use their configured endpoint. An enabled provider's effective endpoint must be an absolute HTTP or HTTPS URL with a host (for example, `http://127.0.0.1:11434`); an invalid endpoint causes a validation error before any capture evidence is sent.

Table and plain results include the provider notice. JSON results include an `llm` object with `used`, `kind`, `model`, `endpointHost`, and `source`, including when no model is used. `source` is `local file`, `global config`, `--no-llm`, or `none`; a disabled model has `used: false`, `kind: "none"`, and empty model and endpoint host values. `used` indicates that refinement was enabled for the run, not that a provider returned a usable refinement. Quiet text output remains the written file path; the notice still appears on stderr.

When the language model is genuinely uncertain about a decision that materially changes the model, it asks questions — one at a time, each with its background context, a list of choices, and always an "Other" entry for typing your own answer. Questions are only asked in an interactive terminal; non-interactive runs (CI, piped output, `-y/--yes`) never ask and finalize with the model's best effort.

Expand All @@ -70,6 +81,7 @@ When the language model is genuinely uncertain about a decision that materially
|---|---|
| `start` without an interactive terminal, or with `--yes` | Validation error — the wizard needs a terminal. |
| `interpret` finds no capture (`.jsonl`) files in the folder | Not-found error with a hint to run the extractor with JSON output. |
| Enabled model's effective endpoint is invalid | Validation error before creating a chat client or sending capture evidence. |
| Interpretation fails | Server error carrying the session's error message. |

## Running Prologue without the CLI
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
// Copyright (c) Cratis. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using System.Diagnostics.CodeAnalysis;
using System.Text.Json;
using Cratis.Prologue.Configuration;
using Cratis.Prologue.Contracts;
using Cratis.Prologue.Interpretation;
using Microsoft.Extensions.AI;
using Spectre.Console;

namespace Cratis.Cli.for_InterpretPrologueCommand.given;

public class an_interpret_command : Specification
{
protected string _folder;
protected InterpretPrologueSettings _settings;
protected InterpretPrologueCommand _command;
protected IChatClientFactory _chatClients;
protected IChatClient _client;
protected LlmConfiguration? _global;
protected int _globalLoads;
protected string _noticeBeforeClient;
protected string _noticeBeforeRequest;
protected string _output;
protected string _notice;
protected int _exitCode;
protected JsonElement _llm;

void Establish()
{
_folder = Directory.CreateTempSubdirectory("cli-241-").FullName;
File.WriteAllText(Path.Combine(_folder, "cratis-prologue.json"), "{}");
File.WriteAllText(Path.Combine(_folder, "http.jsonl"), CaptureFiles.Serialize(new CapturedEntry(
Guid.NewGuid(),
DateTimeOffset.UtcNow,
SourceKind.Http,
new HttpCommandObserved("POST", "/api/orders", 201, string.Empty))));
_settings = new InterpretPrologueSettings
{
Path = _folder,
File = Path.Combine(_folder, "System.play"),
Output = OutputFormats.Json,
Yes = true
};
_global = new() { Kind = "openai", Model = "global-model", ApiKey = "global-secret" };
_noticeBeforeClient = string.Empty;
_noticeBeforeRequest = string.Empty;
_chatClients = Substitute.For<IChatClientFactory>();
_client = Substitute.For<IChatClient>();
_command = new InterpretPrologueCommand(_chatClients, () =>
{
_globalLoads++;
return _global;
});
}

protected void Configure([StringSyntax(StringSyntaxAttribute.Json)] string json) => File.WriteAllText(Path.Combine(_folder, "cratis-prologue.json"), json);

protected async Task Interpret()
{
var previousOutput = Console.Out;
var previousError = Console.Error;
var previousConsole = AnsiConsole.Console;
await using var output = new StringWriter();
await using var error = new StringWriter();
_chatClients.CreateFor(Arg.Any<LlmOptions>()).Returns(_ =>
{
_noticeBeforeClient = error.ToString();
return _client;
});
_client.GetResponseAsync(Arg.Any<IEnumerable<ChatMessage>>(), Arg.Any<ChatOptions>(), Arg.Any<CancellationToken>()).Returns(_ =>
{
_noticeBeforeRequest = error.ToString();
return new ChatResponse(new ChatMessage(ChatRole.Assistant, "{}"));
});
try
{
Console.SetOut(output);
Console.SetError(error);
AnsiConsole.Console = AnsiConsole.Create(new AnsiConsoleSettings { Out = new AnsiConsoleOutput(output), Ansi = AnsiSupport.No, Interactive = InteractionSupport.No });
AnsiConsole.Console.Profile.Width = 300;
_exitCode = await _command.ExecuteAsync(new CommandContext([], Substitute.For<IRemainingArguments>(), "interpret", null), _settings, CancellationToken.None);
}
finally
{
Console.SetOut(previousOutput);
Console.SetError(previousError);
AnsiConsole.Console = previousConsole;
}

_output = output.ToString();
_notice = error.ToString();
if (_exitCode == ExitCodes.Success && _settings.ResolveOutputFormat().StartsWith(OutputFormats.Json, StringComparison.Ordinal))
{
using var result = JsonDocument.Parse(_output);
_llm = result.RootElement.GetProperty("llm").Clone();
}
}

void Destroy() => Directory.Delete(_folder, recursive: true);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
// Copyright (c) Cratis. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using Cratis.Prologue.Configuration;

namespace Cratis.Cli.for_InterpretPrologueCommand.when_interpreting;

[Collection(CliSpecsCollection.Name)]
public class with_an_anthropic_environment_endpoint : given.an_interpret_command
{
[Theory]
[InlineData(null, "https://user:secret@environment.example/path?token=hidden", "environment.example")]
[InlineData("", "https://environment.example", "environment.example")]
[InlineData("http://llm:11434", "https://environment.example", "environment.example")]
[InlineData("https://user:secret@configured.example/path?token=hidden", "https://environment.example", "configured.example")]
[InlineData(null, "http://llm:11434", "llm")]
public async Task should_disclose_and_pin_the_effective_endpoint(string? endpoint, string environmentEndpoint, string expectedHost)
{
var previousEndpoint = Environment.GetEnvironmentVariable("ANTHROPIC_BASE_URL");
try
{
Environment.SetEnvironmentVariable("ANTHROPIC_BASE_URL", environmentEndpoint);
_global = new() { Kind = "anthropic", Endpoint = endpoint, Model = "claude-opus-4-6" };

await Interpret();

_exitCode.ShouldEqual(ExitCodes.Success);
_noticeBeforeClient.ShouldContain($"endpoint host: {expectedHost}");
_noticeBeforeRequest.ShouldContain($"endpoint host: {expectedHost}");
_llm.GetProperty("endpointHost").GetString().ShouldEqual(expectedHost);
_chatClients.Received(1).CreateFor(Arg.Is<LlmOptions>(options => options.Endpoint == new Uri(string.IsNullOrEmpty(endpoint) || endpoint == "http://llm:11434" ? environmentEndpoint : endpoint).AbsoluteUri));
_notice.ShouldNotContain("secret");
_notice.ShouldNotContain("hidden");
}
finally
{
Environment.SetEnvironmentVariable("ANTHROPIC_BASE_URL", previousEndpoint);
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
// Copyright (c) Cratis. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using Cratis.Prologue.Configuration;
using Microsoft.Extensions.AI;

namespace Cratis.Cli.for_InterpretPrologueCommand.when_interpreting;

[Collection(CliSpecsCollection.Name)]
public class with_an_invalid_endpoint : given.an_interpret_command
{
[Theory]
[InlineData("127.0.0.1:11434")]
[InlineData("")]
[InlineData("relative/path")]
[InlineData("file:///private/captures")]
[InlineData("https://user:secret@")]
public async Task should_reject_before_creating_a_client_or_sending_evidence(string endpoint)
{
Configure(System.Text.Json.JsonSerializer.Serialize(new { llm = new { enabled = true, kind = "Ollama", endpoint } }));

await Interpret();

_exitCode.ShouldEqual(ExitCodes.ValidationError);
_notice.ShouldContain("absolute HTTP or HTTPS URL with a host");
_notice.ShouldNotContain("secret");
_chatClients.DidNotReceive().CreateFor(Arg.Any<LlmOptions>());
await _client.DidNotReceive().GetResponseAsync(Arg.Any<IEnumerable<ChatMessage>>(), Arg.Any<ChatOptions>(), Arg.Any<CancellationToken>());
File.Exists(_settings.File).ShouldBeFalse();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
// Copyright (c) Cratis. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

namespace Cratis.Cli.for_InterpretPrologueCommand.when_interpreting;

[Collection(CliSpecsCollection.Name)]
public class with_global_configuration : given.an_interpret_command
{
Task Because() => Interpret();

[Fact] void should_succeed() => _exitCode.ShouldEqual(ExitCodes.Success);
[Fact] void should_announce_the_global_provider_before_sending_evidence() => _noticeBeforeRequest.ShouldContain("OpenAI; model: global-model; endpoint host: api.openai.com; source: global config");
[Fact] void should_report_the_provider_in_json() => _llm.GetProperty("kind").GetString().ShouldEqual("OpenAI");
[Fact] void should_report_the_global_source_in_json() => _llm.GetProperty("source").GetString().ShouldEqual("global config");
[Fact] void should_report_refinement_enabled() => _llm.GetProperty("used").GetBoolean().ShouldBeTrue();
[Fact] void should_not_expose_the_global_secret() => _notice.ShouldNotContain("global-secret");
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright (c) Cratis. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using Cratis.Prologue.Configuration;

namespace Cratis.Cli.for_InterpretPrologueCommand.when_interpreting;

[Collection(CliSpecsCollection.Name)]
public class with_local_disabled : given.an_interpret_command
{
void Establish() => Configure("""{"llm":{"enabled":false}}""");
Task Because() => Interpret();

[Fact] void should_succeed() => _exitCode.ShouldEqual(ExitCodes.Success);
[Fact] void should_write_a_screenplay() => File.ReadAllText(_settings.File).ShouldNotBeEmpty();
[Fact] void should_not_create_a_chat_client() => _chatClients.DidNotReceive().CreateFor(Arg.Any<LlmOptions>());
[Fact] void should_report_no_model_in_json() => _llm.GetProperty("used").GetBoolean().ShouldBeFalse();
[Fact] void should_report_the_local_source() => _llm.GetProperty("source").GetString().ShouldEqual("local file");
[Fact] void should_report_heuristics_only() => _notice.ShouldContain("Interpreting with heuristics only.");
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
// Copyright (c) Cratis. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using Cratis.Prologue.Configuration;
using Microsoft.Extensions.AI;

namespace Cratis.Cli.for_InterpretPrologueCommand.when_interpreting;

[Collection(CliSpecsCollection.Name)]
public class with_local_enabled : given.an_interpret_command
{
void Establish() => Configure("""{"llm":{"enabled":true,"kind":"OpenAICompatible","modelId":"project-model","endpoint":"https://user:password@project.example/v1?secret=hidden","accessToken":"local-secret"}}""");
Task Because() => Interpret();

[Fact] void should_succeed() => _exitCode.ShouldEqual(ExitCodes.Success);
[Fact] void should_use_the_local_provider() => _chatClients.Received(1).CreateFor(Arg.Is<LlmOptions>(options => options.ModelId == "project-model"));
[Fact] void should_call_the_model() => _client.Received(1).GetResponseAsync(Arg.Any<IEnumerable<ChatMessage>>(), Arg.Any<ChatOptions>(), Arg.Any<CancellationToken>());
[Fact] void should_announce_before_creating_the_client() => _noticeBeforeClient.ShouldContain("OpenAICompatible; model: project-model; endpoint host: project.example; source: local file");
[Fact] void should_announce_before_sending_evidence() => _noticeBeforeRequest.ShouldContain("Capture evidence will be sent to this provider.");
[Fact] void should_include_the_model_in_json() => _llm.GetProperty("model").GetString().ShouldEqual("project-model");
[Fact] void should_include_only_the_endpoint_host_in_json() => _llm.GetProperty("endpointHost").GetString().ShouldEqual("project.example");
[Fact] void should_not_expose_the_access_token() => _output.ShouldNotContain("local-secret");
[Fact] void should_not_expose_endpoint_credentials() => _notice.ShouldNotContain("password");
[Fact] void should_not_expose_endpoint_query_strings() => _notice.ShouldNotContain("hidden");
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
// Copyright (c) Cratis. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using Cratis.Prologue.Configuration;

namespace Cratis.Cli.for_InterpretPrologueCommand.when_interpreting;

[Collection(CliSpecsCollection.Name)]
public class with_no_llm : given.an_interpret_command
{
void Establish()
{
_settings.NoLlm = true;
Configure("""{"llm":{"enabled":true,"kind":"Anthropic","accessToken":"local-secret"}}""");
}

Task Because() => Interpret();

[Fact] void should_succeed() => _exitCode.ShouldEqual(ExitCodes.Success);
[Fact] void should_write_a_screenplay() => File.ReadAllText(_settings.File).ShouldNotBeEmpty();
[Fact] void should_not_create_a_chat_client() => _chatClients.DidNotReceive().CreateFor(Arg.Any<LlmOptions>());
[Fact] void should_not_load_the_global_configuration() => _globalLoads.ShouldEqual(0);
[Fact] void should_report_no_model_in_json() => _llm.GetProperty("used").GetBoolean().ShouldBeFalse();
[Fact] void should_report_none_as_the_provider() => _llm.GetProperty("kind").GetString().ShouldEqual("none");
[Fact] void should_report_the_command_line_source() => _llm.GetProperty("source").GetString().ShouldEqual("--no-llm");
[Fact] void should_report_heuristics_only() => _notice.ShouldContain("Interpreting with heuristics only.");
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
// Copyright (c) Cratis. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

namespace Cratis.Cli.for_InterpretPrologueCommand.when_interpreting;

[Collection(CliSpecsCollection.Name)]
public class with_plain_output : given.an_interpret_command
{
void Establish() => _settings.Output = OutputFormats.Plain;
Task Because() => Interpret();

[Fact] void should_include_the_provider_in_the_result() => _output.ShouldContain("Language model: OpenAI; model: global-model; endpoint host: api.openai.com; source: global config");
[Fact] void should_announce_before_sending_evidence() => _noticeBeforeRequest.ShouldContain("source: global config");
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
// Copyright (c) Cratis. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

namespace Cratis.Cli.for_InterpretPrologueCommand.when_interpreting;

[Collection(CliSpecsCollection.Name)]
public class with_quiet_output : given.an_interpret_command
{
void Establish()
{
_settings.Quiet = true;
_settings.Output = OutputFormats.Plain;
}
Task Because() => Interpret();

[Fact] void should_announce_before_sending_evidence() => _noticeBeforeRequest.ShouldContain("source: global config");
[Fact] void should_keep_stdout_to_the_output_path() => _output.Trim().ShouldEqual(_settings.File);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
// Copyright (c) Cratis. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

namespace Cratis.Cli.for_InterpretPrologueCommand.when_interpreting;

[Collection(CliSpecsCollection.Name)]
public class with_table_output_and_no_llm : given.an_interpret_command
{
void Establish()
{
_settings.Output = OutputFormats.Table;
_settings.NoLlm = true;
}

Task Because() => Interpret();

[Fact] void should_include_none_and_its_source_in_the_result() => _output.ShouldContain("Language model: none; source: --no-llm. Interpreting with heuristics only.");
}
Loading
Loading