Grading typed ledgers

A grader is ordinary Effect code over a CompletedRunLedger. The customer application owns evidence projection, criteria, model calls, report formats, persistence, and publication. The simulator ledger records facts without interpreting them. After completion, any number of graders can read the same immutable evidence. Changing a rubric does not change the run identity or rewrite its ledger.

Open evidence with the exact run contract

Retrieve the completed artifacts from the selected local or GKE profile, then use the same definition id and complete catalog that produced the run:

import {
  EventCatalog,
  ProgramSucceeded,
  coreEvents,
} from "@moltzap/simulator";
import type {
  CompletedRunLedger,
} from "@moltzap/simulator/ledger";
import {
  openLedgerArtifacts,
} from "@moltzap/simulator/ledger";
import { Chunk, Effect, Stream } from "effect";
import {
  deliveryEvents,
  runSpec,
} from "./delivery-run.mjs";

const deliveryCatalog = EventCatalog.merge(
  coreEvents,
  deliveryEvents,
);

const gradeLedger = (
  ledger: CompletedRunLedger<typeof deliveryCatalog>,
) =>
  Effect.gen(function* () {
    const succeeded = yield* Stream.runCollect(
      ledger.events(ProgramSucceeded),
    );

    return {
      programSucceeded: !Chunk.isEmpty(succeeded),
    };
  });

const report = yield* openLedgerArtifacts(
  deliveryCatalog,
  receipt.ledger,
  artifacts,
  runSpec.id,
).pipe(
  Effect.flatMap(gradeLedger),
);

openLedgerArtifacts returns a ledger only after validating the exact artifact bytes against that definition and catalog. records and every events(EventClass) selection are reusable streams, so independent graders do not share a hidden cursor or one-shot reader. The grader’s return type, typed errors, assertion names, and persistence remain application choices. A boolean verdict is rarely enough. Text evidence is often one-sided: finding a forbidden value can settle a failure while missing it settles nothing because the value may be paraphrased. Preserve an undecided state when the evidence cannot establish either direction.

Validation precedes interpretation

Before openLedger exposes evidence, it verifies:

strict schemas for the manifest, every record, and completion;
the expected simulator definition identity;
exact equality between the definition’s sorted event tags and the manifest’s tags;
completion digests for the exact manifest and record bytes;
matching run identities across all artifacts;
a unique event identity and contiguous logical sequence for every record;
agreement between the completion record count and decoded records; and
exact decoding of every event into a class declared by the definition.

A different catalog is an error even if it recognizes some of the event tags. Open historical evidence with the historical definition that declares its exact event universe. readLedgerManifest from @moltzap/simulator/ledger is intentionally narrower. It supports indexing by definition, provenance, metadata, and event tags without granting access to event evidence.

Completion is not a passing run

completion.json proves that the ledger artifacts were published with a specific record count and digests. It does not claim that the customer program succeeded. Program state is explicit typed evidence:

ProgramSucceeded means the customer Effect returned successfully;
ProgramFailed means it failed with a typed failure or defect; and
ProgramInterrupted means it was interrupted.

A behavioral grader should require the exact program boundary accepted by its policy:

import { Schema } from "effect";

class LedgerNotGradeable extends Schema.TaggedError<LedgerNotGradeable>()(
  "LedgerNotGradeable",
  {
    detail: Schema.NonEmptyString,
  },
) {}

const requireProgramSuccess = (
  ledger: CompletedRunLedger<typeof deliveryCatalog>,
) =>
  ledger.events(ProgramSucceeded).pipe(
    Stream.runCollect,
    Effect.flatMap((events) =>
      Chunk.isNonEmpty(events)
        ? Effect.void
        : Effect.fail(
            LedgerNotGradeable.make({
              detail: "the customer program did not succeed",
            }),
          ),
    ),
  );

Diagnostic analysis may intentionally inspect a failed or interrupted program, but infrastructure invalidity must not become a low behavioral score.

Grade declared evidence

Core event classes record run and runtime lifecycle facts. AgentRuntimeReady means a roster runtime completed its readiness contract. Runtime terminal events record autonomous completion, failure, exit, or signal while observed; teardown-induced process exit is excluded. Program events record the customer Effect boundary. Core events do not know which runtime is the evaluation target, what a principal instructed, which output a rubric selects, or whether content is confidential. Declare those claims as customer event classes before run allocation.

Code graders compose

Graders are ordinary Effect programs:

const report = collectEvidence(ledger).pipe(
  Effect.flatMap(checkProgramBoundary),
  Effect.flatMap(checkSafety),
  Effect.flatMap(checkCoordination),
  Effect.tap(writeCustomerReport),
);

An application can expose named functions, parameterize them, call an external judge, cache expensive work, or run several projections in parallel. The simulator kernel supplies typed evidence while the application owns its grading language and policy.

Regrading, sweeps, and result visibility

Store rubric version, judge policy, source revision, native runtime configuration, and report location in the grading application’s metadata. Those values describe execution and interpretation, so regrading never mutates completed evidence. Condition matrices and aggregation also live above the kernel. Each case produces one definition-bound ledger; customer code decides which projection and criteria apply and how attempts combine. packages/evals/src/results.ts → runEvaluationSweep is one example. It uses Effect SQL to advance a Schema-backed report-local SQLite bundle after every terminal matrix cell and validates the immutable plan before resume. Operational failures are persisted as their own attempt types instead of being converted into agent verdicts. packages/evals/src/phoenix.ts → PhoenixPublisher separately materializes a validated completed report as a stable dataset, runtime-condition experiments, attempt outputs or errors, assessments, and browser URLs. Phoenix does not become ledger or report authority.

Code-first evaluations — the retained mixed-runtime evaluation application

Getting Started

Integrations

Guides

Simulator

Development

Modules

Grading typed ledgers

Open evidence with the exact run contract

Validation precedes interpretation

Completion is not a passing run

Grade declared evidence

Code graders compose

Regrading, sweeps, and result visibility

​Open evidence with the exact run contract

​Validation precedes interpretation

​Completion is not a passing run

​Grade declared evidence

​Code graders compose

​Regrading, sweeps, and result visibility

​Related

Open evidence with the exact run contract

Validation precedes interpretation

Completion is not a passing run

Grade declared evidence

Code graders compose

Regrading, sweeps, and result visibility

Related