Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,58 @@ public class Milks
}
```

### Accessing failed output

Failed validator results expose optional `FailureInfo` through `Result<T>.Failure`.
`FailureInfo.Value` is an unvalidated `object`, not a value guaranteed to conform to `T`.
Check `HasValue` to distinguish unavailable data from explicit `null`, `false`, `0`, or empty strings.
JSON validators retain a cloned `JsonElement` when parsing succeeds, plus the original `InputText`
even when parsing fails. Their comment, trailing-comma, and depth-limit options are respected when
collecting failure data. Constraint and program validators retain the rejected instance.

Translation still returns a typed value on success and throws `TypeChatException` on final validation
failure. The exception now exposes the same context through `Failure`, alongside its existing
`Response`:

```cs
try
{
CalendarActions actions = await translator.TranslateAsync(requestText);
}
catch (TypeChatException error)
{
FailureInfo? failure = error.Failure;
if (failure?.HasValue == true)
{
object? rejected = failure.Value;
// Validate or repair the rejected value before treating it as CalendarActions.
}
string? originalResponse = failure?.ResponseText;
}
```

To manage repairs yourself, set `translator.MaxRepairAttempts = 0`.
On final failure, the context retains the latest rejected candidate and available model response text,
not a complete attempt history. These can come from different attempts if the latest repair response
cannot be parsed. Existing repair limits, diagnostics, events, typed success values, and cancellation
behavior are unchanged.

HTTP, I/O, JSON response-decoding, and nested TypeChat exceptions during a repair are rethrown unchanged,
with context available as `error.Data[FailureInfo.ExceptionDataKey]`.
An initial transport failure has no invented candidate, and cancellation and programming exceptions
are not intercepted. Custom validators can return
`Result<T>.Error(message).WithFailure(new FailureInfo { Value = rejected, ResponseText = text })`;
existing failure values and metadata take precedence over fallback context, without mutating the
original result. The existing `Error(value, message)` overload still retains its typed `Value` and
also exposes it as unvalidated failure context.

When a malformed repair reuses an earlier candidate, its custom failure metadata is retained alongside
the latest available input and response text. A new rejected candidate does not inherit the previous
candidate's custom metadata.

Use `Success` to distinguish outcomes, not the presence of a value. Rejected data and response text
may contain sensitive output; do not log or serialize them automatically.

## Microsoft.TypeChat.Program ##
TypeChat.Program translates natural language requests into simple programs (***Plans***), represented as JSON.

Expand Down Expand Up @@ -99,6 +151,9 @@ so the projects use its SDK-provided implementation rather than overriding it wi
* Go to the root directory of the project
* dotnet build Typechat.sln

For [CVE-2026-62900](https://github.com/advisories/GHSA-23fw-v26w-5fgq), use a fixed .NET 8 SDK
(8.0.130 or 8.0.424, or later servicing releases), or a patched .NET 9/10 SDK.

## Nuget Packages

* Microsoft.Typechat [![](https://img.shields.io/nuget/dt/Microsoft.TypeChat.svg?label=downloads&color=007edf&logo=nuget)](https://www.nuget.org/packages/Microsoft.TypeChat)
Expand Down
8 changes: 5 additions & 3 deletions src/typechat.program/ProgramValidator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,8 @@ public Result<Program> Validate(string json)
if (!result.Value.HasNotTranslated)
{
// Now validate the full program
return ValidateProgram(result.Value);
result = ValidateProgram(result.Value);
return result.Success ? result : result.WithFailure(new FailureInfo { InputText = json });
}
}
return result;
Expand All @@ -50,7 +51,8 @@ public virtual Result<Program> ValidateProgram(Program program)
{
if (_programValidator is not null)
{
return _programValidator.ValidateProgram(program);
Result<Program> result = _programValidator.ValidateProgram(program);
return result.Success ? result : result.WithFailure(new FailureInfo { Value = program });
}
return program;
}
Expand Down Expand Up @@ -85,7 +87,7 @@ public override Result<Program> ValidateProgram(Program program)
}
catch (Exception ex)
{
return Result<Program>.Error(ex.Message);
return Result<Program>.Error(ex.Message).WithFailure(new FailureInfo { Value = program });
}
}
}
2 changes: 1 addition & 1 deletion src/typechat/ConstraintsValidator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ public Result<T> Validate(T value)
return new Result<T>(value);
}

return Result<T>.Error(result.ErrorMessage);
return Result<T>.Error(result.ErrorMessage).WithFailure(new FailureInfo { Value = value });
}

private string ToErrorString(List<ValidationResult> validationResults)
Expand Down
92 changes: 92 additions & 0 deletions src/typechat/FailureInfo.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
// Copyright (c) Microsoft. All rights reserved.

namespace Microsoft.TypeChat;

/// <summary>
/// Unvalidated output associated with a failure. Retains bounded context, not an attempt history.
/// </summary>
public class FailureInfo
{
/// <summary>
/// Key used to attach failure context to transport exceptions without changing their type.
/// </summary>
public const string ExceptionDataKey = "Microsoft.TypeChat.FailureInfo";

private object? _value;

/// <summary>
/// Rejected value, when available. Check HasValue before using it, and validate it before
/// treating it as the target type. Parsed JSON is retained as an independent JsonElement.
/// </summary>
public object? Value
{
get => _value;
set
{
_value = value;
HasValue = true;
}
}

/// <summary>
/// Distinguishes unavailable data from an explicitly supplied value, including null.
/// </summary>
public bool HasValue { get; private set; }

/// <summary>
/// JSON input supplied to a validator, when available. If supplied without a value,
/// the translator respects the validator's decision not to provide parsed data.
/// </summary>
public string? InputText { get; set; }

/// <summary>
/// Latest model response text, when available. This can come from a later repair attempt
/// than Value if that response couldn't be parsed.
/// </summary>
public string? ResponseText { get; set; }

internal FailureInfo WithFallback(FailureInfo? fallback)
{
// Preserve fallback metadata unless this context has its own value or custom metadata.
FailureInfo source = !HasValue && GetType() == typeof(FailureInfo) && fallback is not null ? fallback : this;
var result = (FailureInfo)source.MemberwiseClone();
if (!result.HasValue && fallback?.HasValue == true)
{
result.Value = fallback.Value;
}
result.InputText = InputText ?? fallback?.InputText;
result.ResponseText = ResponseText ?? fallback?.ResponseText;
return result;
}

internal void AttachTo(Exception exception)
{
exception.Data[ExceptionDataKey] = exception.Data[ExceptionDataKey] is FailureInfo existing ?
existing.WithFallback(this) :
WithFallback(null);
}

internal static FailureInfo FromJson(string? json, JsonSerializerOptions options)
{
var result = new FailureInfo { InputText = json };
if (json is null)
{
return result;
}
try
{
using JsonDocument document = JsonDocument.Parse(json, new JsonDocumentOptions
{
AllowTrailingCommas = options.AllowTrailingCommas,
CommentHandling = options.ReadCommentHandling,
MaxDepth = options.MaxDepth
});
result.Value = document.RootElement.Clone();
}
catch (JsonException)
{
// The original validation error reports malformed JSON; only input text is available.
}
return result;
}
}
2 changes: 1 addition & 1 deletion src/typechat/IConstraintsValidator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ namespace Microsoft.TypeChat;
public interface IConstraintsValidator<T>
{
/// <summary>
/// Validate the value T
/// Validate the value T. Failures can include FailureInfo retaining the rejected instance.
/// </summary>
/// <param name="value">check constraints for this value</param>
/// <returns>result of the validation</returns>
Expand Down
3 changes: 2 additions & 1 deletion src/typechat/IJsonTypeValidator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ public interface IJsonTypeValidator<T>
/// <summary>
/// Parses and validates the given JSON string according to the associated schema. Returns a
/// Result containing the JSON object if validation was successful.Otherwise, returns
/// an Result with a message property that contains validation diagnostics.
/// a Result with a message containing validation diagnostics and optional FailureInfo
/// retaining the rejected value and input text.
/// </summary>
/// <param name="json"></param>
/// <returns></returns>
Expand Down
9 changes: 7 additions & 2 deletions src/typechat/JsonSerializerTypeValidator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -57,14 +57,19 @@ public JsonSerializerTypeValidator(JsonSerializerOptions? options = null)
}
catch (JsonException jex)
{
return Result<object?>.Error(ToErrorString(json, jex));
return CreateError(json, ToErrorString(json, jex));
}
catch (Exception ex)
{
return Result<object?>.Error(ex.Message);
return CreateError(json, ex.Message);
}
}

private Result<object?> CreateError(string json, string message)
{
return Result<object?>.Error(message).WithFailure(FailureInfo.FromJson(json, _options));
}

private string ToErrorString(string json, JsonException error)
{
StringBuilder sb = new StringBuilder();
Expand Down
34 changes: 32 additions & 2 deletions src/typechat/JsonTranslator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ public async Task<object> TranslateToObjectAsync(string request, CancellationTok

/// <summary>
/// Translate a natural language request into an object of type 'T'
/// Final validation failures retain unvalidated output in TypeChatException.Failure.
/// </summary>
/// <param name="request">text request</param>
/// <param name="cancelToken">optional cancel token</param>
Expand Down Expand Up @@ -208,9 +209,20 @@ public async Task<T> TranslateAsync(
requestSettings ??= _translationSettings;
Prompt prompt = CreateRequestPrompt(request, preamble);
int repairAttempts = 0;
FailureInfo? lastFailure = null;
while (true)
{
string responseText = await GetResponseAsync(prompt, requestSettings, cancelToken).ConfigureAwait(false);
string responseText;
try
{
responseText = await GetResponseAsync(prompt, requestSettings, cancelToken).ConfigureAwait(false);
}
catch (Exception ex) when (lastFailure is not null &&
ex is HttpRequestException or IOException or JsonException or TypeChatException)
{
lastFailure.AttachTo(ex);
throw;
}

JsonResponse jsonResponse = JsonResponse.Parse(responseText);
Result<T> validationResult;
Expand All @@ -232,11 +244,18 @@ public async Task<T> TranslateAsync(
validationResult = Result<T>.Error(TypeChatException.NoJson(jsonResponse));
}

validationResult = validationResult.WithFailure(new FailureInfo { ResponseText = responseText });
if (lastFailure is not null)
{
validationResult = validationResult.WithFailure(lastFailure);
}
lastFailure = validationResult.Failure;

// Attempt to repair the Json that was returned
++repairAttempts;
if (repairAttempts > _maxRepairAttempts)
{
TypeChatException.ThrowJsonValidation(request, jsonResponse, validationResult.Message);
TypeChatException.ThrowJsonValidation(request, jsonResponse, validationResult.Message, lastFailure);
}
NotifyEvent(AttemptingRepair, validationResult.Message);

Expand Down Expand Up @@ -275,15 +294,26 @@ protected virtual PromptSection CreateRepairPrompt(string responseText, Result<T
private Result<T> ValidateJson(string json)
{
var result = Validator.Validate(json);
if (!result.Success)
{
result = result.WithFailure(result.Failure?.HasValue == true || result.Failure?.InputText is not null ?
new FailureInfo { InputText = json } :
FailureInfo.FromJson(json, JsonSerializerTypeValidator.DefaultOptions()));
}
if (!OnValidationComplete(result))
{
return result;
}
if (result.Success)
{
T instance = result.Value;
result = (_constraintsValidator is not null) ?
_constraintsValidator.Validate(result.Value) :
result;
if (!result.Success)
{
result = result.WithFailure(new FailureInfo { Value = instance, InputText = json });
}
}
return result;
}
Expand Down
29 changes: 28 additions & 1 deletion src/typechat/Result.cs
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,9 @@ public Result(T value, string? message = null)
public Result(Result<object?> value)
{
Success = value.Success;
Value = (T)value.Value;
Value = value.Success || value.Value is T ? (T)value.Value : default;
Message = value.Message;
Failure = value.Failure;
}

internal Result(bool success, string? message)
Expand All @@ -55,6 +56,31 @@ internal Result(bool success, string? message)
/// </summary>
public string? Message { get; set; }

/// <summary>
/// Optional unvalidated output associated with a failed result. Value remains the typed
/// result; use Failure.Value for rejected data that does not conform to T.
/// </summary>
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
public FailureInfo? Failure { get; private set; }

/// <summary>
/// Returns a shallow copy of this failed result, retaining existing failure fields and
/// filling unavailable fields from the supplied context. Custom result metadata is preserved.
/// </summary>
/// <param name="failure">Fallback failure context.</param>
/// <returns>A failed result with the combined context.</returns>
public Result<T> WithFailure(FailureInfo failure)
{
ArgumentVerify.ThrowIfNull(failure, nameof(failure));
if (Success)
{
throw new InvalidOperationException("Failure context cannot be added to a successful result.");
}
var result = (Result<T>)MemberwiseClone();
result.Failure = Failure is null ? failure.WithFallback(null) : Failure.WithFallback(failure);
return result;
}

public static implicit operator Result<T>(T value)
{
return new Result<T>(value);
Expand Down Expand Up @@ -84,6 +110,7 @@ public static Result<T> Error(T value, string message)
{
var result = new Result<T>(false, message);
result.Value = value;
result.Failure = new FailureInfo { Value = value };
return result;
}
}
Loading
Loading