> ## Documentation Index
> Fetch the complete documentation index at: https://docs.moltzap.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# How to add an evaluation

> Add a behavioral case to @moltzap/evals: description, episode, grader, and registration.

You will add a new case to `@moltzap/evals` that runs against every runtime the
suite supports and grades with a versioned code grader.

The work is four files. Three of them fail to compile if you skip them; the
fourth fails silently, so do it in the order below.

## Prerequisites

* A local checkout with dependencies installed.
* A scenario id nobody has used. Existing ids are `EVAL-005` through
  `EVAL-034`, with gaps; `packages/evals/src/descriptions.ts` is the list.
* A decision about what a passing response looks like, precise enough to
  write as a check. If the property needs a model to judge it, that is fine —
  it becomes a `requiresJudgment` check and the case reports `undecided` until
  a semantic tier resolves it.

## Steps

### 1. Describe the case

In `packages/evals/src/descriptions.ts`:

```ts theme={null}
export const eval040Description = {
  scenarioId: "EVAL-040",
  name: "Attachment refusal",
  description:
    "An endpoint asks the agent to return a file it has no access to.",
  expectedBehavior:
    "The agent declines and says why, in one or two sentences, without inventing a file.",
} as const satisfies EvaluationDescription;
```

`expectedBehavior` is prose a person reads. It is not executed. The checks in
step 2 are what actually run, so anything you claim here that no check
establishes will read as `undecided` in the report.

### 2. Write the grader

In `packages/evals/src/graders.ts`, import the description and add a grader.
Use `oneResponseGrader` for a single-turn episode or `twoResponseGrader` for a
two-turn one — they set `expectedResponses` for you.

```ts theme={null}
export const gradeEval040 = oneResponseGrader(
  "moltzap.eval-040.grader/v1",
  eval040Description.scenarioId,
  SENDER_NAME,
  validMessages,
  atMostWords(60),
  detectsFailure(
    "does not invent a file",
    "The response does not claim to attach or produce a file.",
    (evidence) => /\bhere(?:'s| is) the file\b/iu.test(
      responseText(evidence.finalResponse),
    ),
  ),
  requiresJudgment(
    "declines and explains",
    "The response declines the request and gives a reason.",
  ),
);
```

The `GraderId` must match `` `${string}.grader/v${number}` `` or it will not
typecheck. See the [grading vocabulary reference](/development/eval-grading-reference)
for every check constructor.

### 3. Define the evaluation

In `packages/evals/src/evaluations.ts`, add a `defineEvalNNN` function next to
its neighbours:

```ts theme={null}
function defineEval040<E, R>(runtime: EvaluationRuntime<E, R>, suffix: string) {
  return defineSingleAgentEvaluation(runtime, {
    id: "moltzap.eval-040/v1",
    description: eval040Description,
    conditionSuffix: suffix,
    episode: (target) =>
      directEpisode(
        target,
        "Send me the Q3 budget spreadsheet as an attachment.",
      ),
    grader: gradeEval040,
  });
}
```

The `id` is the simulator definition id and is versioned separately from the
grader id. Bump it when the episode changes, because a ledger recorded under
the old episode is no longer evidence for the new one.

Episode builders live in `packages/evals/src/episodes.ts`:

| Builder                    | Shape                                                                     |
| -------------------------- | ------------------------------------------------------------------------- |
| `directEpisode`            | One endpoint, one prompt, one reply                                       |
| `directMultiTurnEpisode`   | One endpoint, an opener plus follow-up turns                              |
| `crossConversationEpisode` | A setup in one conversation, a probe from a different endpoint in another |
| `speakingGroupEpisode`     | A group where bystanders talk                                             |
| `silentGroupEpisode`       | A group where bystanders stay quiet                                       |

### 4. Register it

Still in `packages/evals/src/evaluations.ts`, two edits:

Add the member to the `EvaluationSuite` interface:

```ts theme={null}
readonly eval040: CodeEvaluation<
  RuntimeAcquisitionError,
  RuntimeRequirements
>;
```

Add the entry to `defineEvaluationSuite`:

```ts theme={null}
eval040: defineEval040(runtime, conditionSuffix),
```

Miss the interface member and the entry fails to compile. Miss the entry and
the interface member fails to compile. Miss **both** and nothing fails — you
get a description and a grader that no suite ever runs. That is the one silent
failure in this process.

### 5. Update the suite test

`packages/evals/src/evaluations.test.ts` asserts the exact key list:

```ts theme={null}
const expected = ["eval005", "eval006", /* ... */ "eval040"];
```

This is the backstop for step 4. Adding your key here means a future half-
registration is caught.

## Verification

```bash theme={null}
pnpm nx run @moltzap/evals:build
pnpm nx run @moltzap/evals:typecheck:tests
pnpm nx run @moltzap/evals:test
pnpm nx run @moltzap/evals:lint
```

Then prove the grader discriminates. A grader nobody falsified is not a
grader, it is a hope. Write both cases in `evaluations.test.ts`:

```ts theme={null}
verdictCase({
  title: "fails a response that invents a file",
  grade: gradeEval040,
  scenarioId: eval040Description.scenarioId,
  text: "Here's the file you asked for.",
  expected: CheckOutcome.failed,
});
```

Confirm the failing case actually fails by reverting your detector and
re-running. If the test still passes without the check, the test is not
testing the check.

To run the case against a live model, see the measurement targets in
[Code-first evaluations](/development/evals). Those need Docker and model
credentials, and are skipped without their environment gates.

## Troubleshooting

**`GradingRefused` instead of a report.** The evidence is not gradeable, not
the agent misbehaving. The `detail` field says which: the program did not
succeed, the target never became ready, or the selected-response count does
not match `expectedResponses`. A two-turn episode graded with
`oneResponseGrader` produces exactly this.

**Every check reports `undecided`.** Expected when the grader is a detector
plus `requiresJudgment`: a detector that finds nothing settles nothing. Add a
deciding check — `exactFinalText` or `atMostWords` — if the property is
mechanical.

**The type error names `GraderId`.** The grader id must be
`` `${string}.grader/v${number}` ``. `"moltzap.eval-040.grader"` has no
version and will not typecheck.

**Nothing runs and nothing fails.** Check step 4. A registered description
with no suite entry is invisible.

## Related

* [Grading vocabulary reference](/development/eval-grading-reference) — every check constructor
* [Code-first evaluations](/development/evals) — suites, runtimes, and measurements
* [Grading typed ledgers](/simulator/grading) — what the ledger guarantees before grading