Skip to main content
packages/evals is a private executable application that demonstrates one evaluation product built on @moltzap/simulator. Cases, runtime conditions, criteria, and sweeps are ordinary TypeScript and Effect values. Customers compose the simulator package directly and can build a domain-specific authoring language around the parameters they need. Every matrix cell is one RunSpec submitted through the core simulator’s local-Kubernetes or GKE profile. Its target runs in an Agent Sandbox application container. The controller invokes the case Effect after the target and its runtime-specific bridge are ready. Cases may also add autonomous social peers whose behavior is fixed by the case catalog.

​
Social boundary

peer.ts and peer-application.ts implement case-owned peers entirely through the public @moltzap/client HarnessEndpoint contract:
  • send publishes a post to an explicit agent:<AgentName> or group:<AgentName>,... input, with each call creating one post;
  • messages yields certified remote-authored InboundDelivery values; and
  • each delivery’s acknowledge effect runs after the peer completes its prescribed handling, including deliberate skips. It carries no content and grants no response authority.
messages is one live stream for the whole endpoint. Peers subscribe before traffic starts, filter deliveries by their signed address facts, and do not ask the simulator to register or replay a conversation. Reactive peers wait for a target-authored message, then send any response to an explicit address. Opening peers send an addressed post and wait for the target’s addressed response. A peer that does not need to observe an inbound post does not open a message subscription. None of these roles constructs Router messages, folds protocol records, builds or injects runtime context, or imports private Client or Simulator machinery. OpenClaw may select its stock shared or private session configuration. The simulator NanoClaw image uses one native agent-shared session across the MoltZap addresses delivered to its application container. Client and Simulator inject no cross-address context for either runtime.

​
One attempt

A completed case path is deliberately narrow:
  1. The condition adds one OpenClaw or NanoClaw target and the case’s exact peer roster.
  2. Case policy instructs the target through its runtime-native principal gateway and coordinates any required peer exchanges through public endpoints.
  3. Peer exchanges emit semantic social observations, or bounded timeout observations when a required addressed post never arrives.
  4. The case program returns the earlier gateway, social, or timeout evidence identity that its policy selects for grading.
  5. Execution records that selection, completes the ledger, reopens it through the same simulator definition, and projects normalized evidence.
  6. Code criteria settle mechanical questions. One semantic-judge call resolves every remaining criterion.
  7. Effect SQL commits the terminal attempt in a report-local SQLite bundle.
  8. An explicit publish command materializes a completed report in Phoenix.

​
Bundled catalog

The ordered catalog contains these 16 cases: EVAL-008 and EVAL-030 through EVAL-034 are the six cross-conversation cases. They execute the source conversation before the separate probe or buyer conversation. The controller never constructs or injects a cross-conversation snapshot. A runtime’s stock host owns whether those addresses share context. Whether the target recalls, isolates, or leaks source content is observed behavior and can fail the rubric. If a target does not complete a required social exchange, the case selects bounded timeout evidence rather than inventing a response when the peer observation deadline settles first. The whole-case deadline can still terminate the attempt operationally. --runtime all|openclaw|nanoclaw selects the conditions in the immutable attempt plan and defaults to all. Every selected condition requires its matching model option. --messaging-mode shared|private selects OpenClaw’s session configuration. private is currently accepted only with --runtime openclaw; NanoClaw qualification uses the application image’s agent-shared session. --concurrency (1 to 8, default 4) is how many cells are submitted at once. It is an execution choice rather than part of the plan, so a report may be resumed at a different width.

​
Principal input is runtime-native

principal.ts → PrincipalDriver is an evaluation-local adapter over one concrete runtime gateway. There is no simulator-wide gateway union: Gateway evidence states what the adapter submitted or received at that native boundary. It does not claim that an agent followed an instruction. NanoClaw’s output is an uncorrelated multi-frame stream. Its ask path serializes questions, subscribes before submitting the input, and selects the next output frame. The selection does not establish causal correlation. execution.ts → evaluationCondition captures one runtime together with its matching PrincipalDriverFactory. The factory builds a driver for each attempt. OpenClaw keeps its native idempotency sequence inside that driver; NanoClaw does not receive a synthetic correlation key. Adding another runtime condition therefore adds an adapter for that exact gateway type rather than another member of a shared command or response protocol. Case programs receive four capabilities:
  • instruct, which submits an input through the target’s native gateway without selecting output;
  • ask, which submits a question and selects OpenClaw’s correlated final output or NanoClaw’s next observed output;
  • observePeer, which records a case-owned peer exchange or its bounded absence without selecting it; and
  • selectSocialOutput, which selects the final target-authored social action observed by one peer, or selects the peer timeout if no addressed post arrives.
Most bundled cases select social evidence. EVAL-019 and EVAL-022 select principal output through each runtime’s documented ask policy.

​
Typed ledger evidence

events.ts → evaluationEvents declares the complete customer event universe before a run is allocated. It contains:
  • native OpenClaw and NanoClaw gateway observations;
  • SocialActionObserved, retaining endpoint, canonical address, author, direction, and exact semantic content from a public InboundMessage;
  • SocialActionNotObserved, retaining the peer and the bounded deadline that expired; and
  • the final evidence selection.
The current bundled case interpreter emits exactly one EvaluationEvidenceSelected event after its program returns. The projector also verifies that a selection names earlier gateway, social, or timeout evidence and is not duplicated.

​
Grading

transcript.ts → transcriptFromLedger produces an EvaluationTranscript with:
  • one native-gateway-established target identity;
  • ledger-ordered gateway, social, and peer-timeout items; and
  • the evidence identities selected by case policy.
Selected gateway evidence must be output from the same target. OpenClaw output is correlated terminal output. NanoClaw output is the next frame observed by its serialized ask path and does not establish causal correlation. Selected social evidence must be a target-authored input observed by the named peer as an explicitly addressed post; a bounded peer timeout is also selectable. Every criterion receives the same ordered selected evidence. Exact-answer, literal-disclosure, and peer-timeout checks can decide conclusive outcomes mechanically; remaining semantic questions go to the judge. judge.ts → SemanticJudge is a provider-neutral Effect service. The bundled OpenAI layer uses gpt-5.6-sol, medium reasoning, strict structured output, no tools, a two-minute timeout, and two retries for transient provider failures. The rubric and criteria remain trusted policy; the entire transcript is delimited as untrusted evidence. Each result must cover the exact requested criterion set and cite selected evidence. Judge unavailability produces JudgingUnavailableAttempt; it never becomes a behavioral failed verdict. undecided is a valid semantic verdict.

​
Result bundles and honest failure states

The default plan contains 16 cases, two runtime conditions, and one sample per cell: 32 cells in canonical catalog and condition order. results.ts → runEvaluationSweep runs up to --concurrency cells of that matrix at once in plan order and commits each finished attempt in its own uninterruptible SQLite transaction, also in plan order. The durable report is therefore always a plan-order prefix; interruption loses at most --concurrency finished but uncommitted attempts, which a resume reruns. The SQLite bundle under .moltzap/evals/results/ stores:
  • the exact clean Git revision;
  • immutable case, condition, and judge-policy snapshots;
  • physical ledger receipts;
  • normalized transcripts and assessments when evidence reached grading; and
  • typed run, evidence, judge, and ledger-allocation failures.
results.ts → resumeStoredEvaluationReport validates every immutable plan component, including the selected profile, images, Temporal address, and ledger artifact location, before executing only the missing suffix. The report cannot skip, reorder, or silently move a matrix cell. One process holds a report at a time: eval, resume, and publish take a lease on the bundle, so a second opener fails fast with ReportLocked while the holder is alive. Live failures remain results. OpenClaw or NanoClaw may fail to start, terminate, time out, produce evidence that grading rejects, or fail semantic assessment. The sweep persists the corresponding typed state and continues through the matrix. Social cases, including all six cross-conversation cases, execute on both conditions but may produce peer-timeout evidence, reach the whole-case timeout, or receive behaviorally failed and undecided verdicts. Those verdicts do not make the command fail; operational attempt states produce a nonzero exit only after the terminal matrix has been saved.

​
Commands

Run deterministic checks:
Run the semantic calibration corpus:
Start or resume a live report:
The source worktree must be clean. Each selected runtime’s model ID becomes part of the immutable native runtime configuration. The controller image and each selected runtime’s application image must be immutable digest references. Their presence is an execution prerequisite, not evidence that an image or live cluster has passed qualification. The NanoClaw image uses the outer application container as its simulator isolation boundary and supervises the native agent runner as a child process. It does not require a nested container daemon. For GKE, select --profile gke, replace the local artifact root with the Terraform-owned MOLTZAP_GKE_ARTIFACT_BUCKET, and provide the explicit MOLTZAP_KUBE_CONTEXT and configured Temporal endpoint. Each profile submits the same generated RunSpec module and reads the same relative completed-ledger path. Publish a completed report:
phoenix.ts → PhoenixPublisher keeps the local SQLite bundle authoritative. It idempotently reconciles a stable case dataset, one experiment per runtime condition, every terminal attempt, and code or model assessments. The command prints a publication receipt containing browser URLs for the condition experiments. Set PHOENIX_API_KEY when the Phoenix deployment requires one.

​
Retained results

packages/evals/results/ keeps curated reports of live runs whose ledgers stay in the GKE artifact bucket. The OpenClaw shared/private evaluation of 2026-09-01 records six assessed attempts with the image digests, run namespaces, ledger identities, and artifact digests behind each result.