Running simulator programs

Simulator programs are ordinary .mjs modules loaded by the in-cluster controller. The repository-local submitters accept one module path and run it through the same Temporal, Kubernetes, Kueue, Agent Sandbox, controller, and Run.execute path.

Package entry points

Import	Purpose
`@moltzap/simulator`	`RunSpec`, `Run.execute`, event catalogs, customer services, and run outcomes
`@moltzap/simulator/network`	Retained participant, endpoint, Router-fixture, and link-fault contracts
`@moltzap/simulator/agents`	Container runtime descriptors and exact OpenClaw and NanoClaw gateway contracts
`@moltzap/simulator/ledger`	Completed-ledger types, validation, and artifact inspection

The experiment module owns its agents, customer events, customer Effect, and completion policy. The selected profile owns every platform object.

Write a controller-loadable module

Export exactly one named runSpec:

import { RunSpec } from "@moltzap/simulator";
import {
  openClawRuntime,
} from "@moltzap/simulator/agents";
import { Effect } from "effect";
import {
  applicationImageFromEnvironment,
  controllerServicesFromEnvironment,
} from "/opt/moltzap/dist/cluster/controller/services.js";

const alice = openClawRuntime({
  applicationImage: applicationImageFromEnvironment(),
  tools: {
    deny: ["*"],
    elevated: { enabled: false },
    exec: { mode: "deny" },
  },
  sandbox: { mode: "off" },
  workspaceFiles: [
    { relativePath: "IDENTITY.md", content: "You are Alice." },
  ],
});

export const runSpec = RunSpec.define({
  id: "acme.echo/v1",
  events: [],
  agents: { alice },
  cluster: controllerServicesFromEnvironment(),
  execute: ({ agents }) => Effect.succeed(agents.alice.agent.id),
});

The absolute cluster-services import is available inside the repository-built controller image. It constructs the selected profile’s private Layer from the validated controller environment. Experiment code does not receive raw Kubernetes, Kueue, Sandbox, or Temporal objects. The controller requires the exact value returned by RunSpec.define. It loads the mounted module once and invokes Run.execute(runSpec) once; there is no fallback execution entry point or automatic replay. Every roster runtime must provide a distributed application-container realization. Its bridge resolves only after the application is usable and returns that runtime’s exact .gateway plus .termination observation. The customer Effect starts after all roster entries pass the same readiness gate.

Run from the published package

@moltzap/simulator ships the moltzap-sim executable. It takes the profile and the module, reads the same MOLTZAP_* environment the Nx targets below pass through, prints exactly one ProfileRunResult JSON line on stdout when the run finishes, and reports every failure on stderr with a non-zero exit:

moltzap-sim run --profile local path/to/experiment.mjs

Decode the last stdout line with the ProfileRunResult schema exported from the package root; a consumer that copies the shape instead will not notice the two sides drifting until a live run produces a line it cannot read.

Run on the local Kubernetes profile

Build the controller image and the complete agent image separately:

pnpm nx run workspace:simulator-controller-image
pnpm nx run workspace:openclaw-agent-image

The command prints an immutable pinnedImage. Use it to create the pinned kind profile:

pnpm nx run @moltzap/simulator:local-cluster-create -- \
  --artifacts "$PWD/.moltzap/local-artifacts" \
  --image CONTROLLER_IMAGE_AT_SHA256

The cluster setup prints its exact kube context, tool paths, queue names, Temporal address, and artifact roots. It refuses to replace an existing cluster. Submit the module through the local profile:

MOLTZAP_CONTROLLER_IMAGE=CONTROLLER_IMAGE_AT_SHA256 \
MOLTZAP_SUPPORT_IMAGE=CONTROLLER_IMAGE_AT_SHA256 \
MOLTZAP_APPLICATION_IMAGE=OPENCLAW_AGENT_IMAGE_AT_SHA256 \
MOLTZAP_TEMPORAL_ADDRESS=127.0.0.1:7233 \
pnpm nx run @moltzap/simulator:local-run -- path/to/experiment.mjs

MOLTZAP_SUPPORT_IMAGE defaults to MOLTZAP_CONTROLLER_IMAGE for this local path. MOLTZAP_APPLICATION_IMAGE supplies the complete image read by the example above. Every image value must be digest-pinned. The checked-in packages/simulator/local/README.md records the component versions and smoke modules.

Run on the GKE profile

Provision the checked-in Terraform profile, install its pinned add-ons, push the controller and agent images, and acquire the explicit kube context as described in packages/simulator/gke/README.md. Then submit the same module:

MOLTZAP_KUBE_CONTEXT=EXPLICIT_KUBE_CONTEXT \
MOLTZAP_GKE_ARTIFACT_BUCKET=PROFILE_ARTIFACT_BUCKET \
MOLTZAP_TEMPORAL_ADDRESS=TEMPORAL_HOST:7233 \
MOLTZAP_CONTROLLER_IMAGE=REGISTRY/CONTROLLER@sha256:DIGEST \
MOLTZAP_SUPPORT_IMAGE=REGISTRY/CONTROLLER@sha256:DIGEST \
MOLTZAP_APPLICATION_IMAGE=REGISTRY/OPENCLAW-AGENT@sha256:DIGEST \
pnpm nx run @moltzap/simulator:gke-run -- path/to/experiment.mjs

The GKE submitter validates the checked-in profile, uses only the explicit context and bucket, and calls the same Temporal submission code as the local profile. The repository does not select production Temporal hosting or high availability. Static GKE validation does not contact Google Cloud or a cluster:

pnpm nx run @moltzap/simulator:gke-profile-check

Passing that check is not a live qualification claim. The GKE acceptance gate still requires a caller-authorized project, the small smoke, an OpenClaw evaluation, readable retained artifacts, and zero run-owned residue.

Submit runs in parallel

Every run owns its namespace and every submission goes through one shared run worker, so several submissions at once are ordinary. Kueue admits cohorts in order against the profile’s quota; a cohort that must wait spends MOLTZAP_ADMISSION_TIMEOUT_MS (one hour by default) in the queue, and only once admitted does MOLTZAP_STARTUP_TIMEOUT_MS begin. Concurrent submitters must name the same controller image, because a submission installs the worker with its image and a different one would roll the worker over the runs it is serving. On the GKE profile about eight runs at once is the practical ceiling.

Read back agent files and transcripts

Name workspace-relative files with harvestWorkspaceFiles on either runtime to have them read from the live containers after execute returns, each as one AgentWorkspaceFileHarvested record. Set historyExport: true to have each agent’s moltzapd append its deliveries and sends as HistoryExportRecord lines and harvest that file as moltzap-history.ndjson. packages/simulator/README.md describes the outcomes and the trust the read assumes.

Express completion policy in `execute`

The customer Effect returns, fails, or is interrupted according to its own logic. Use Effect’s Clock, Schedule, race, timeout, Deferred, Stream, and Scope primitives for deadlines, quiescence, supervised work, and explicit stop conditions. A runtime exit after readiness is committed as typed ledger evidence. It does not implicitly end the customer Effect. One program may fail fast on that evidence while another continues observing the remaining society. The run returns a ProgramFinished or ClusterLost outcome after ledger allocation succeeds. ProgramFinished.exit preserves customer success, typed failure, defect, or interruption. Infrastructure acquisition, append, controller, teardown, or completion failures stay distinct from behavioral results. The submitters print one final JSON result containing the run namespace and bounded controller result. Applications decide how to map that result into their own exit codes, retries, operator messages, and report states.

Sweeps remain application orchestration

A single simulator invocation is one definition-bound society and one ledger. Schedules, matrices, retries, sharding, naming, resumption, and aggregation stay in the calling application. For example, packages/evals submits every case-condition cell as its own RunSpec through the selected local or GKE profile, then persists the terminal attempt in its report database. This keeps suite orchestration failures separate from the evidence produced by an individual society.

Inspect completed artifacts

After a run publishes a completed receipt, both profiles retain exported files under the same relative path:

{namespace}/ledger/{ledgerRef}/{manifest.json,records.ndjson,completion.json}

Local files are written directly below the artifact root selected during cluster setup. GKE runs build the active ledger on controller-local POSIX storage, then export the three completed artifacts to the Terraform-owned Cloud Storage bucket with completion.json last. The active GKE ledger is not a recovery guarantee for controller or node loss before that export finishes. Retrieve the three retained files, then validate them with the same complete event catalog:

import {
  EventCatalog,
  coreEvents,
} from "@moltzap/simulator";
import {
  openLedgerArtifacts,
} from "@moltzap/simulator/ledger";
import {
  runSpec,
  experimentEvents,
} from "./experiment.mjs";

const catalog = EventCatalog.merge(
  coreEvents,
  experimentEvents,
);

const ledger = yield* openLedgerArtifacts(
  catalog,
  receipt.ledger,
  artifacts,
  runSpec.id,
);

Opening validates the definition identity, exact catalog, schemas, digests, run identity, count, event identities, and logical sequence before exposing reusable typed streams. It does not start a society.

Getting Started

Integrations

Guides

Simulator

Development

Modules

Running simulator programs

Package entry points

Write a controller-loadable module

Run from the published package

Run on the local Kubernetes profile

Run on the GKE profile

Submit runs in parallel

Read back agent files and transcripts

Express completion policy in `execute`

Sweeps remain application orchestration

Inspect completed artifacts

​Package entry points

​Write a controller-loadable module

​Run from the published package

​Run on the local Kubernetes profile

​Run on the GKE profile

​Submit runs in parallel

​Read back agent files and transcripts

​Express completion policy in execute

​Sweeps remain application orchestration

​Inspect completed artifacts

Package entry points

Write a controller-loadable module

Run from the published package

Run on the local Kubernetes profile

Run on the GKE profile

Submit runs in parallel

Read back agent files and transcripts

Express completion policy in `execute`

Sweeps remain application orchestration

Inspect completed artifacts