Skip to main content
A grader you have not tested can reward the wrong thing. So before any agent runs on a task, Miroir holds its grader to three kinds of runs: the reference solution must score reward 1 and harm 0, doing nothing must score 0 and 0, and every probe must stay within the reward bounds and the harm it declares. One Harbor job runs all of them, and miroir check asserts the results. The same command later summarizes agent runs, to pick the tasks worth training on.

Run the check job

Each dataset ships a Harbor job config that runs every task with the reference solution, with nothing, and with each probe number. The fixture’s:
Run it from the repository root, then check it. Before a release, run two attempts of everything with -k 2:
With two or more attempts, the reference solution, doing nothing and each probe must also score the same in every attempt: a probe whose attempts differ fails, even inside its bounds. A Harbor job runs every agent on every task, so each SOLUTION=N slot also runs on tasks with fewer than N probes. That is 7 of the fixture’s 28 trials, and miroir check skips them.
Rerun a check job in full after any change, never with harbor run --diff: a change to a solution or a probe is a patch bump, which --diff would reuse.
Ignore Harbor’s means for a check job. Harbor groups a job’s metrics by agent name, so its summary pools the reference runs and every probe slot into one oracle mean (reward 0.333 and harm 0.333 for the fixture). miroir check is the verdict.

miroir check

It runs on the host from the Miroir image, through the alias in the quick start. So every --tasks and job path must be under the current directory, as Harbor’s ./jobs is. Exit status: 0 with no violation, 1 with violations, 2 on a usage or I/O error (an unreadable job directory, a --tasks directory with no task).

Static task checks

miroir check checks every directory below each --tasks DIR that holds a task.toml. The search is recursive, so a directory of split directories works.
  • task.toml parses, [task].name is set, and [task].version is MAJOR.MINOR.PATCH. No two tasks share a name.
  • MIROIR_SPEC is set, and is a valid spec of at most 32 KiB.
  • [verifier.environment].docker_image is set, [verifier].user is absent, and environment_mode is absent or separate.
  • A [[verifier.collect]] hook runs miroir collect in a service other than main, and no hook has a user.
  • artifacts holds { source = "/miroir/evidence", service = <that service> } with no destination, and environment/docker-compose.yaml defines the service.
  • network_mode in [environment], [agent] and [verifier] is absent or public, and allow_internet is absent or true.
  • solution/solve.sh exists, and solution/probe-N.sh exists for exactly N = 1 to the number of [[metadata.miroir.probes]] entries, each with harm of 0 or 1 and a reward bound in [0, 1].
miroir check parses task.toml as TOML 1.1, which accepts a little more than Harbor’s parser. harbor lint and harbor run decide whether a task.toml parses.

Trial checks

Every immediate subdirectory of a job directory whose result.json has a trial_name is a trial. miroir check reads its task, agent, model and SOLUTION; its rewards and whether Harbor holds them valid; the error Harbor caught; agent/exit-code.txt; the errors in verifier/reward-details.json; and, for a void, the first line of verifier/test-stdout.txt that starts with void: . With --tasks and job directories together, coverage is checked too: every task needs a golden and a nop trial, and every declared probe a trial. Probe trials need --tasks, where their bounds live. Trials match tasks by [task].name. A violation names the trial and the file that shows why; a missing trial, or attempts that differ, name no file. Here the check job was regraded after pay-invoice’s paid check (weight 0.6) was changed to SELECT 0, and after one golden trial’s evidence was deleted:

Agent runs

Runs of any other agent are summarized by agent and model, then by task:
  • trials, valid trials and voids;
  • the mean reward and the harm rate, over valid trials;
  • pass@1, the share of valid trials with reward 1;
  • pass@n, 1 if any attempt scored 1;
  • pass^n, 1 if every valid attempt scored 1;
  • a class over the valid attempts: solved when every attempt scored 1, learnable when the rewards differ, unsolved when every attempt scored the same value below 1 (such as 0.6 every time), and none with fewer than 2 valid attempts.
Here, Harbor’s nop agent run with -k 2 under its import path (-a harbor.agents.nop:NopAgent), which miroir check summarizes as an agent instead of holding it to the null run’s expectations:
Train uses the classes to pick tasks.

JSON output

--json prints the same report as one object with three lists:
  • violations: {task, check, verdict, detail, path}. check is static, golden, nop or probe N; verdict is FAIL or VOID; path is null for a missing trial or attempts that differ.
  • tasks: {task, dir, trials, skipped, checked, violations}. checked lists the trial kinds seen, such as ["golden", "nop", "probe 1"].
  • agents: {agent, model, trials, valid, voids, reward, harm, pass@1, pass^n, solved, learnable, unsolved, tasks}. Each of tasks is {task, trials, valid, voids, reward, harm, pass@1, pass@n, pass^n, class}. The rates are null without a valid trial, and so is the class with fewer than two valid attempts.

Regrade after a spec fix

A trial’s evidence stays in its job directory, so fixing a spec needs no agent run. Edit MIROIR_SPEC, bump the task’s minor version, and regrade the check job, which runs only the verifiers:
For an evaluation, harbor run --diff reruns, regrades or reuses each task by its version bump.
Not run for these docs: it spends model API credit.
Both are Harbor features: see Regrade.