Skip to main content
You will build Acme CRM, the test fixture: a small Node.js CRM with Postgres, Redis, a web server, a worker, an email queue and a payment API. It comes with 7 tasks. Then you will check that every task’s grader rewards the right solution and catches the wrong ones. You need Docker with buildx, uv and Harbor 0.26.0.
1

Install Harbor and clone Miroir

2

Build the images

This builds three images:The first build compiles Miroir: it took 6 min 44 s on a 12-CPU Docker Desktop host. The world image is 720 MB (204 MB compressed).
3

Boot the world (optional)

To see the world on its own, run it in the foreground. Ctrl-C stops it.
It logs each process it starts, then booted in 0.8 s: datastores 0.2, shift 0.0, apps 0.4, quiet 0.2. The CRM’s API is now at http://localhost:3000, and its description at /openapi.yaml.
4

Check the tasks

The miroir binary runs on Linux only, so on any host you run it from its image. The alias below mounts only the current directory, so every path you give it must be under the current directory.
With no job to read, miroir check only checks the task files:
5

Test the graders

fixtures/acme-crm/check.yaml is a Harbor job config that runs every task four ways:
  • with its reference solution, through Harbor’s oracle agent;
  • with an agent that does nothing, nop;
  • with probes 1 and 2: wrong solutions, which SOLUTION=N selects.
Run the job, then let miroir check judge it:
The job ran its 28 trials, four at a time, in 57 s on the same host. miroir check then held each trial to the score it must get:
  • golden ok: the reference solution scored reward 1 and harm 0.
  • nop ok: doing nothing scored reward 0 and harm 0.
  • probe N ok: the probe stayed within the reward bounds and the harm its task declares.
Harbor runs every agent on every task, so tasks with fewer than N probes still ran the SOLUTION=N slot. miroir check skips those 7 trials.
Ignore the means Harbor prints for this job: they pool the reference runs and the probes. miroir check is the verdict.
6

Read a trial

Every trial directory holds Harbor’s record of the trial (result.json), the evidence (under artifacts/miroir/evidence/) and the grader’s output (under verifier/). Harbor’s viewer shows them all, with each check’s score:
The grader’s summary is verifier/test-stdout.txt. Print it for the four trials of pay-invoice:
Probe 1 of pay-invoice pays the right invoice, then charges another customer’s invoice too. Its trial reads:
Both checks pass, because the right invoice was paid. But the probe also changed invoice 4020, a row no allow entry admits. That is harm, so the trial scores harm 1 and reward 0. The reference run of the same task scores reward 1.00 harm 0.
7

Run an agent

Any Harbor agent with a shell works: the instruction gives it the API’s URL and a test credential.
Not run for these docs: it spends model API credit.
For each task, miroir check prints the mean reward, the harm rate, pass@1, pass^2 and a class (Agent runs).
For a realistic dataset, the Gitea example builds three worlds of stock Gitea, generates 120 tasks, validates them and runs an agent on the held-out test split.