Hi — I maintain EvalPort, an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.
Opening this as an issue rather than a PR, per CONTRIBUTING.md's note that this repo is generated code and direct PRs aren't accepted — happy for this to just be triaged/considered however you'd normally handle a feature request like this.
Concrete precedent first: OpenAI's own SDK team is currently reviewing a community PR that adds native to_openeval()/from_openeval() dataset conversion helpers directly into openai-python, in src/openai/types/evals/openeval.py:
openai/openai-python#3619 (fixes openai/openai-python#3549)
To be accurate about status: it is open, not merged — 6 commits in, review requested, genuine back-and-forth with an OpenAI maintainer. But it shows a major provider SDK is willing to carry this conversion logic in-tree.
I'd like to propose the same for client-python, grounded in your actual generated types. I read src/mistralai/client/models/usermessage.py and assistantmessage.py: UserMessage.content and AssistantMessage.content are both Nullable[Union[str, List[ContentChunk]]], with role: Literal["user"|"assistant"] and AssistantMessage.tool_calls: List[ToolCall]. A rough sketch:
# Since this SDK is Speakeasy-generated, this would need to live in a hand-maintained
# path preserved via .genignore (already present at repo root) rather than in
# src/mistralai/client/models/, which regenerates.
from mistralai.client.models import UserMessage, AssistantMessage
def to_openeval(messages: list[UserMessage | AssistantMessage], *, id: str) -> dict:
"""content is already str | List[ContentChunk] on both message types —
maps close to 1:1 onto an EvalPort test_case's turns."""
...
def from_openeval(item: dict) -> list[UserMessage | AssistantMessage]:
...Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure at all if this isn't a priority right now — just wanted the precedent and a concrete, type-grounded sketch on record. Happy to answer questions.
Hi — I maintain EvalPort, an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.
Opening this as an issue rather than a PR, per CONTRIBUTING.md's note that this repo is generated code and direct PRs aren't accepted — happy for this to just be triaged/considered however you'd normally handle a feature request like this.
Concrete precedent first: OpenAI's own SDK team is currently reviewing a community PR that adds native
to_openeval()/from_openeval()dataset conversion helpers directly intoopenai-python, insrc/openai/types/evals/openeval.py:openai/openai-python#3619 (fixes openai/openai-python#3549)
To be accurate about status: it is open, not merged — 6 commits in, review requested, genuine back-and-forth with an OpenAI maintainer. But it shows a major provider SDK is willing to carry this conversion logic in-tree.
I'd like to propose the same for
client-python, grounded in your actual generated types. I readsrc/mistralai/client/models/usermessage.pyandassistantmessage.py:UserMessage.contentandAssistantMessage.contentare bothNullable[Union[str, List[ContentChunk]]], withrole: Literal["user"|"assistant"]andAssistantMessage.tool_calls: List[ToolCall]. A rough sketch:Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure at all if this isn't a priority right now — just wanted the precedent and a concrete, type-grounded sketch on record. Happy to answer questions.