Prompt evaluation: test one prompt across models
Write a prompt template once, give it a fixed set of test cases, and run it against up to three models. Each output is stored with deterministic checks, an optional judge score, latency, and tokens, so you can compare models on the same inputs.
Who it is for: people on Pro or Team who want to check a prompt against known inputs before they rely on it, or who want to see how different models handle the same task.
How an evaluation run works
- 1
Create a suite
Give the suite a name and an optional description. Write a prompt template of up to 40,000 characters and put {{input}} where each test case's input should go. You can also add a judge rubric of up to 4,000 characters.
- 2
Add test cases
Each suite holds up to 20 cases. A case has a name, an input, and an optional expected output or list of key terms. It can also have deterministic checks: terms that must appear (up to 30), terms that must not appear (up to 30), and a minimum or maximum length in characters.
- 3
Choose models
Enter one to three OpenRouter model IDs in provider/model format. You can also enter a separate judge model ID. If you leave the judge blank, the run uses the deterministic checks only.
- 4
Run the comparison
For each case, HyperPrompt puts the case input into your template and sends the result to each model as a single user message. Calls run one after another. A failed call is saved as an error on that case and model, and the rest of the run continues.
- 5
Read the results
The suite shows the latest run, one card per case and model. Each card has the model's output, the scores that apply, a judge reason if you used a judge, the latency in milliseconds, and the token count. Results are stored with the run.
What each score means
Assertion score
The percentage of a case's checks that passed. The checks are: includes each required term (case-insensitive), excludes each forbidden term, meets the minimum length, and stays under the maximum length. The same output always gets the same score.
Expected-term coverage
Only calculated when the case has an expected output. It is the share of distinct words and numbers in the expected output that also appear in the model's output. Word order and meaning are ignored.
Judge score (optional)
A separate call to your judge model, which receives the rubric, the expected output, and the candidate output. It must return a score from 0 to 100 and a short reason. If its reply can't be parsed, no score is recorded and the result says so. Judge scores can differ between runs.
Latency and tokens
Latency is the time HyperPrompt's server waited for the model's output. The token count is the usage OpenRouter reported for that output request.
Example: one case, two models
Illustrative example, not a measured result. The model IDs are placeholders and no model produced the outputs below. The example shows how a case is set up and how the assertion score is calculated.
Suite template:
Write release notes for customers from this changelog. Be factual.
{{input}}Case “Bug-fix release”: input is a three-line changelog. Must include fix. Must not include revolutionary. At most 400 characters.
Models: vendor-a/model-one, vendor-b/model-two. No judge model.
- Includes “fix”
- Excludes “revolutionary”
- At most 400 characters
Output text, latency (ms), and tokens are shown here in the real result view.
- Includes “fix”
- Excludes “revolutionary”
- At most 400 characters
Output text, latency (ms), and tokens are shown here in the real result view.
Run limits
- Up to 20 test cases per suite, and one to three models per run.
- A run can make at most 10 output requests (cases × models). For example, that allows 5 cases × 2 models or 3 cases × 3 models.
- Each account can make 10 AI requests per minute, and judge calls count toward that limit. A run that uses a judge model therefore has to fit within 10 calls in total, which means at most 5 cases × models.
- Runs use HyperPrompt's OpenRouter connection. You don't add your own provider key.
Plan requirements
Any signed-in account can create suites and test cases. Running them requires a paid plan: Pro ($9/month, or $79 billed yearly) or Team ($29/month with 3 seats, or $290 billed yearly). Free accounts see an upgrade button where the Run button would be. Details are on the pricing page.
What evaluations do not do
- There is no scheduling, CI hook, or webhook, and runs do not start automatically when a prompt changes. You start each run yourself.
- Suites are separate from your prompt library. You paste the template into the suite, and results are not attached to a library prompt or to one of its versions.
- Suites are personal. They are not shared in Team workspaces.
- Each test is one user message. There is no system-prompt field, no multi-turn conversation, no tool calling, no file or image input, and no temperature or other sampling setting.
- Each run produces one output per case and model. HyperPrompt does not average scores across repeated runs or report statistical confidence. It reports tokens, not cost in dollars.
- Term coverage and assertions only check the text itself. They cannot tell you whether an answer is factually correct.
- Which models are available depends on OpenRouter. This page does not keep a list of supported models.
- The web form fills in {{input}} only. The cases API also accepts other named {{variable}} values for each case.
- The suite view shows the latest run. The suite API returns the 10 most recent runs.
Related
- Evaluations documentation
- Prompt versioning and rollback
- Private prompt library
- Free prompt grader (checks one prompt's structure in your browser without calling a model)
Set up your first suite
Create an account to build suites and cases. Upgrade to Pro or Team when you are ready to run them across models.