Skip to main content

​
Behavioral Traces

packages/evals/ no longer ships a local eval CLI, local YAML ingress, local judge stack, or local bundle/report pipeline. The current split is:
  1. @moltzap/server-core emits behavioral traces as OpenTelemetry spans (Effect’s withSpan).
  2. The server records message-delivery and hook-block traces directly where they happen, as the moltzap.message.delivered and moltzap.message.blocked spans.
  3. packages/evals/ only keeps the scenario YAML catalog.
  4. @moltzap/testbed launches and supervises agent collections and owns the compiled trace-capture-harness module loaded by cc-judge.
  5. External cc-judge runners own scenario loading, harness dispatch, scoring, and report emission.

​
Trace emission

  • MessageService emits moltzap.message.delivered and moltzap.message.blocked OTel spans (Effect withSpan). Spans carry message-shape metadata only, never message body content: message id, conversation id, sender id, created-at, part count, text-part count, total text length, channel key, sender display name, recipients/delivered (delivered) or block reason (blocked). Message body plaintext is deliberately redacted from telemetry — the envelope is encrypted at rest and spans can egress to an operator OTLP collector, so the body never belongs on a span.
  • core/tracing.ts wires the OTel SDK Layer; production exports via batch OTLP when an OTLP endpoint env var is set (OTEL_EXPORTER_OTLP_TRACES_ENDPOINT used verbatim, else OTEL_EXPORTER_OTLP_ENDPOINT suffixed with /v1/traces), otherwise spans stay in-process.
  • Tests inject an InMemorySpanExporter via CoreConfig.spanProcessor and read finished spans from CoreTestServer.spanExporter.

​
Verification

MoltZap-side verification now lives in package builds/tests plus the real cc-judge path, not a local eval CLI:
Behavioral traces are verified at the span-exporter level: trace-spans.test.ts reads finished OTel spans from CoreTestServer.spanExporter and asserts on span name and the metadata attributes, and asserts that no span attribute carries message body plaintext. tracing.test.ts covers the OTLP endpoint env-var resolution (trace-specific precedence + URL normalization).

​
cc-judge

cc-judge owns scenario loading, harness dispatch, scoring, and report emission. @moltzap/cc-judge is a workspace devDependency, so the whole suite runs with one command:
That means:
  • MoltZap scenario YAML stays in packages/evals/scenarios/
  • the server emits trace data as OpenTelemetry spans (moltzap.message.delivered / moltzap.message.blocked)
  • the harness module is packages/testbed/dist/trace-capture-harness.js
  • results land in ./eval-results (gitignored)
To run a single scenario:
OpenClaw-backed eval runs use createOpenClawAdapter’s default agent model unless a harness payload or runtime caller overrides it. The judge needs either claude auth login or ANTHROPIC_API_KEY.