Skip to content

feat(evaluators): add Galileo Decision Models evaluator - #293

Merged
namrataghadi-galileo merged 8 commits into
mainfrom
feature/SAO-18345-add-jev-evaluator
Oct 23, 2026
Merged

namrataghadi-galileo merged 8 commits into
mainfrom
feature/SAO-18345-add-jev-evaluator

Conversation

@namrataghadi-galileo

@namrataghadi-galileo namrataghadi-galileo commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add the galileo.decision_models evaluator for invoking a pinned Galileo Decision Models scorer version.
  • Send structured Galileo records and authenticated execution context to the scorer-invoke endpoint, then apply the configured threshold and operator to the returned score.
  • Keep Decision Models context and runtime-flag definitions within its own package for now; these can be consolidated with Luna later.

Scope

  • User-facing: Registers galileo.decision_models with required scorer_id and scorer_version_id, plus optional label, threshold, operator, payload field, and timeout settings.
  • Internal: Adds a Decision Models client, request/response models, record and selector handling, and tests for invocation, serialization, comparison, and error cases.
  • Out of scope: The separate TypeSafe Jev example remains unchanged.

Risk and Rollout

  • Risk level: Medium. This adds a new external evaluator integration and requires the Galileo scorer-invoke runtime, API secret, and invoke URL to be configured.
  • Rollback: Remove the galileo.decision_models entry point and evaluator package implementation.

Testing

  • Added or updated automated tests.
  • Galileo package Ruff lint passed.
  • Galileo package mypy typecheck passed.
  • All PR CI checks passed, including ci, codecov/patch, sdk-ts-ci, ui-ci, validate, and build-and-push-server.

@namrataghadi-galileo namrataghadi-galileo changed the title add jev evaluator feat(evaluators): add jev evaluator Oct 19, 2026
@codecov

codecov Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.04918% with 9 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ontrol_evaluator_galileo/decision_models/client.py 94.23% 9 Missing ⚠️

📢 Thoughts on this report? Let us know!

@namrataghadi-galileo namrataghadi-galileo changed the title feat(evaluators): add jev evaluator feat(evaluators): add Galileo Decision Models evaluator Oct 20, 2026
@namrataghadi-galileo
namrataghadi-galileo enabled auto-merge (squash) October 23, 2026 16:29
@josjeon
josjeon self-requested a review October 23, 2026 16:31
@namrataghadi-galileo
namrataghadi-galileo merged commit c7cd862 into main Oct 23, 2026
6 checks passed
@namrataghadi-galileo
namrataghadi-galileo deleted the feature/SAO-18345-add-jev-evaluator branch October 23, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants