Evaluations

Run an evaluation suite

Evaluations run the same test cases through one to three models and show each output with term coverage, deterministic assertion scores, an optional judge score, latency, and token count. For an overview of the feature, see prompt evaluation.

Written by the HyperPrompt Team · Last updated November 5, 2026

Plan: Pro or Team to run comparisons. On Free, the run button is replaced by Upgrade to Pro.
Where: App navigation → Evals (/app/evals)
Per suite: Up to 20 test cases.
Per run: Up to 3 models; at most 10 model calls, including judge calls.

How a suite is structured

Suite: a name, description, Prompt template, and Judge rubric. Suites are personal; they are not shared in team workspaces.
Prompt template: the text sent to each model. {{input}} is replaced with the test case input.
Test case: an input plus the checks to apply to the output.
Run: every case sent to every selected model, one request at a time, with results saved to the suite.

1. Create a suite

Open Evals. In the New suite card, enter a Name (up to 120 characters) and an optional Description.
Edit the Prompt template. It starts as Answer this request: {{input}} (on two lines). Keep {{input}} wherever the case input should go.
Edit the Judge rubric (up to 4,000 characters). It is only used if you set a judge model when running.
Select Create suite.

Expected result: Evaluation suite created. The suite appears under Suites with “0 cases · 0 runs” and opens on the right.

2. Add test cases

In Test cases, enter a Case name and the Input (up to 10,000 characters).
Optionally fill Expected output or key terms. This drives the term coverage score.
Optionally add deterministic checks: Must include (comma separated), Must not include, Minimum characters, Maximum characters.
Select Add case.

Expected result: the case appears in the list with “Assertions configured” or “No deterministic assertions”. Delete a case with its trash icon; this also deletes its stored results.

Supported checks
Check	How it is scored
Must include	Pass if the output contains the term. Case-insensitive substring match. Up to 30 terms.
Must not include	Pass if the output does not contain the term. Case-insensitive. Up to 30 terms.
Minimum / Maximum characters	Pass if the output length in characters is within the bound (0 to 100,000). Minimum cannot exceed maximum.
Expected output or key terms	Not pass/fail. Term coverage = share of distinct words and numbers in your expected text that also appear in the output.

Through the API, a case can also include a variables object; each {{name}} in the template is replaced with the matching value. The web form does not expose variables.

3. Choose models and run

In the suite panel, enter models in OpenRouter model IDs, comma separated. The field starts with openrouter/free. IDs use the provider/model format listed in OpenRouter’s model catalog (openrouter.ai/models, checked 2026-11-05). Duplicates are removed; up to 3 models are allowed.
Optionally enter an Optional judge model ID. A judge adds a separate 0–100 rubric score. Leave it blank to run deterministic checks only.
Select Run comparison. The button reads Running models... until every call finishes. Keep the tab open; the run happens within that single request.

Expected result: Completed N model checks. (N = cases × models), and a Latest run card with one result per case and model.

Run limits

Cases × models must be 10 or fewer. A judge doubles the calls.
All calls in a run also count against your account’s 10 AI requests per minute, which enhancements share. In practice, a run without a judge can include up to 10 case–model pairs, and a judged run up to 5.
Each model call has a 2,000-token output cap and a 30-second timeout. Calls use temperature 0.7, so outputs and judge scores can differ between runs.
You do not need your own OpenRouter key. Calls go through HyperPrompt’s OpenRouter connection, and a failing evaluation model is not swapped for a different one.

4. Read the results

Result fields
Badge or field	Meaning
% term coverage	Lexical overlap with your expected text. Shown only when the case has expected text. It does not measure meaning or correctness.
% assertions	Passed checks ÷ total checks for that case. Shown only when the case has checks.
% judge	Score from 0 to 100 returned by the judge model using your rubric, the expected output, and the candidate output. Its reason appears under Judge:.
ms · tokens	Wall-clock latency of the model call and the total tokens reported by the provider.
Red error text	The model call failed. The provider’s error message is shown in place of the output.

Interpreting scores

Use assertions for hard requirements (must mention X, must stay under N characters). Treat term coverage and judge scores as signals to inspect, not verdicts: judge models can be inconsistent, and the same model can produce different output on a re-run. Compare outputs side by side before changing a prompt.

The page shows the latest run. The API returns the 10 most recent runs for a suite. Suites, cases, runs, and results are included in the JSON account export.

Troubleshooting

Upgrade to Pro instead of Run comparison, or “Multi-model evaluations require a paid plan”: running needs Pro or Team.
Run comparison is disabled, or “Add at least one test case”: add a case first.
“Each suite can have up to 20 cases”: delete a case or create another suite.
“A run can make at most 10 output and 10 judge requests”: use fewer models or cases.
“Too many AI requests; try again in a minute”: the run needs more calls than your remaining per-minute allowance. Wait a minute, or reduce cases, models, or drop the judge.
“Invalid request” when running: a model ID does not match provider/model, or more than 3 models were entered.
“Judge returned an invalid scoring payload.”: the judge did not return JSON with a 0–100 score and a reason. Try a different judge model.
“Evaluation run failed”: the run stopped unexpectedly. Retry.

Limitations

Suites cannot be renamed, edited, or deleted from the page. Use PATCH or DELETE /api/evaluations/{id} from the API.
Each case sends a single user message to each model. System prompts and multi-turn conversations are not supported.
Runs are not scheduled or triggered automatically.