Relia Prompt

Test and benchmark prompts accross LLM providers and models

This tool is aimed at agentic use-cases for large production applications that require fast and reliable llm calls. For example, extracting sentiment from social media posts, converting a sentence into structured JSON, etc.

Features

• Multi-Provider Testing – OpenAI, Bedrock, DeepSeek, Gemini, Groq, OpenRouter
• Parallel Execution – Run tests concurrently across all configured LLMs
• Repeatability – Each test runs N times per model to measure consistency
• Code-first – Define prompts and tests in code
• Fair prompt comparisons – Judge prompt versions against the same standard, average several judge samples, and cache judgements for identical outputs
• Suite metrics – Compute checks across all of a model's outputs, such as consistency between test cases

Quick Start

Prompts and tests live in your code. Use the example project pattern:

# From a project that has reliaprompt.definitions.ts (see example)
cd example
bun install
bun run reliaprompt:ui   # or: from your app, add "reliaprompt:ui" and run from project root
# Open http://localhost:3000

Set credentials via the RELIA_PROMPT_LLM_CONFIG_JSON environment variable (see Configuration). At least one provider is required.

Usage

Code-first (only mode)

Use ReliaPrompt inside your service for LLM benchmarking and testing from unit tests.

1. Install – Add relia-prompt as a dependency.

2. Initialize – Pass credentials at startup (or load from RELIA_PROMPT_LLM_CONFIG_JSON when using the UI):

   import {
       initializeReliaPrompt,
       runPromptTestsFromSuite,
       definePrompt,
       defineTestCase,
       defineSuite,
   } from "relia-prompt";
   
   initializeReliaPrompt({
       providers: {
           // Canonical keys can be provided directly in library mode.
           // For UI/server mode prefer RELIA_PROMPT_LLM_CONFIG_JSON in .env.
       },
   });

3. Define prompts and tests in code – Use the builder API and export suites for the UI:

   const prompt = definePrompt({ name: "my-prompt", content: "..." });
   const testCases = [
       defineTestCase({ input: "...", expectedOutput: "[...]", expectedOutputType: "array" }),
   ];
   export const suites = [defineSuite({ prompt, testCases })];

4. Run tests – Require testModels (and evaluationModel when using LLM evaluation) per run:

   const { score, results } = await runPromptTestsFromSuite(suite, {
     testModels: [{ provider: "provider-id", modelId: "model-id" }],
     evaluationModel: ..., // required when prompt.evaluationMode === "llm"
     runsPerTest: 1,
   });

5. Optional UI – From your project root (where your definitions live), run:

   yarn reliaprompt:ui

   The UI shows prompts and tests from your code (read-only tests; prompt edits in the browser are drafts only). Configure RELIA_PROMPT_LLM_CONFIG_JSON in .env and choose test/evaluation models on each run.

Comparing prompt versions with LLM evaluation

By default the LLM judge sees the prompt under test and each test case input. That works for checking one prompt, but when you compare two prompt versions, each one is judged against its own instructions. Set an evaluation task and input to judge every prompt version against the same standard, and judge each output several times to smooth out judge noise:

const prompt = definePrompt({
    name: "translate",
    content: translationPrompt, // the prompt under test
    evaluationMode: "llm",
    evaluationCriteria: "...",
    // Shown to the judge instead of `content`.
    evaluationTask: "Review a translation of the source text for an end user.",
    // The judge evaluates each output this many times and the scores are averaged.
    evaluationSamples: 3,
});

const testCase = defineTestCase({
    input: generationInput, // what the prompt under test receives
    // Shown to the judge instead of `input`.
    evaluationInput: "Source text: ...\nContext: ...",
    expectedOutput: "{}",
});

export const suites = [
    defineSuite({
        prompt,
        testCases: [testCase],
        // Computed over all of a model's results, for checks that span test cases.
        metrics: [
            {
                name: "Outputs",
                compute: (testCaseResults) =>
                    testCaseResults.flatMap((testCase) => testCase.runs).length,
            },
        ],
    }),
];

The judge is not deterministic, so the same output can score differently between runs. An evaluation cache stores judge results by judge model, judge prompt, and output, so identical outputs reuse the same judgements across runs and prompt versions:

const evaluationCache = createEvaluationCache("reports/evaluation-cache.json");
await runPromptTestsFromSuite(suite, { testModels, evaluationModel, evaluationCache });

The cache is saved at the end of each run. The UI shares an in-memory cache across the runs in a session; pass evaluationCachePath to startServer to keep it in a file between sessions. Changing the judge model, criteria, task, or input starts new cache entries.

Results include every judge sample on each run (evaluations) and each suite metric on each model's result (metrics). The UI shows both.

Configuration

Configuration is JSON-only via RELIA_PROMPT_LLM_CONFIG_JSON. Use .env.example as the canonical template for the full JSON object.

Bedrock

Relia Prompt discovers active global inference profiles from AWS using ListInferenceProfiles in the configured source Region. Only active system-defined profiles whose IDs start with global. are shown.

Configure AWS credentials in the bedrock provider block:

{
    "bedrock": {
        "accessKeyId": "YOUR_AWS_ACCESS_KEY_ID",
        "secretAccessKey": "YOUR_AWS_SECRET_ACCESS_KEY",
        "sessionToken": "OPTIONAL_AWS_SESSION_TOKEN",
        "region": "us-east-1"
    }
}

The region selects the Bedrock control-plane and runtime endpoints used for request signing and transport. Each global. profile ID routes inference through AWS global inference profiles. Listing a profile does not guarantee that your account can invoke it: IAM, SCPs, model access, quotas, and destination-Region policies still apply at request time.

Required IAM actions for Bedrock in Relia Prompt:

• bedrock:ListInferenceProfiles to populate the model selector
• bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream to run prompt tests

If your organization restricts global routing, ensure your SCPs allow the destination Regions used by global inference profiles.

When a prompt suite defines a valid expected schema, Bedrock requests include native structured output via outputConfig.textFormat.type: "json_schema". Providers without native schema support continue to receive the schema in the system prompt.

See example for a full example and smoke test.

Development

bun dev              # Backend + dashboard with hot reload
bun run dev:backend  # Backend only with hot reload
bun dev:dashboard    # Dashboard dev server
bun run build        # Build dashboard + backend
bun run lint         # Lint backend
bun run test         # Unit tests
bun run test:e2e     # E2E tests (Playwright)
bun run format       # Format code

Project Structure

├── src/                    # Backend (Express + Bun)
│   ├── server.ts           # API routes
│   ├── llm-clients/        # Provider clients
│   └── services/           # Test runner
├── dashboard/              # SvelteKit app
│   └── src/
│       ├── lib/            # Components & stores
│       └── routes/         # Pages
└── example/                # Example project

License

MIT