Skip to main content
The spec says how to grade a trial’s evidence. It is YAML in the task’s task.toml, under [verifier.env] MIROIR_SPEC, and has three parts:
  • checks measure progress. Each scores from 0 to 1, and their weighted mean is the reward.
  • gates are conditions that must hold. A gate that fails is harm.
  • allow lists the changes the task may make. Any other change is harm.
Checks and gates are SQLite queries over the evidence, which also has recipes for common checks. For a first spec, draft one from reference runs.

Format

MIROIR_SPEC holds at most 32 KiB. Write it in task.toml as a TOML multi-line literal string ('''), in which nothing is escaped. SQL never needs ''': miroir draft writes char(39) where a literal would hold three quotes in a row. Never write the whole value as ${…}, which Harbor would template. A spec is refused, and every trial of the task voids, for:
  • an unknown key;
  • a missing checks or allow, or an empty checks;
  • a bad or duplicate name;
  • an empty sql or where;
  • a weight that is not a finite number above 0;
  • an op other than insert, update or delete;
  • columns on an insert or a delete, or an empty columns;
  • more than 32 KiB.
Each error names its path, such as checks[1].weight: must be > 0. miroir check --tasks reports them before any run.

How a check or a gate scores

Each one runs as SELECT * FROM (<sql>) LIMIT 2, so it must be one SELECT; a trailing ; is fine. A statement that does not compile, or that returns more than one column, voids the trial. Both depend only on the spec and the world’s tables, never on what the agent did.

Scope

A row of changes is in scope when some allow entry has its table and op, and:
  • for an update, the entry lists no columns, or lists every changed column;
  • the entry has no where, or the change is among the rows its where admits:
    • insert: SELECT _change FROM after."T" WHERE _op = 'insert' AND (<where>);
    • delete: SELECT _change FROM before."T" WHERE _op = 'delete' AND (<where>);
    • update: both images, intersected, so a row can neither enter nor leave the allowed region unnoticed.
truncate and schema changes are never in scope. A where that fails at run time admits nothing, and its error is recorded on the scope criterion.

Reward and harm

Outcomes

A trial voids only for Miroir’s own failures. Anything the agent can influence is scored. Void. The grader prints void: <reason>, writes no reward file and exits 75. Harbor records an invalid trial and leaves it out of its means.
  • MIROIR_SPEC is missing, invalid or over 32 KiB.
  • A check, gate or where does not compile, or a check returns more than one column.
  • An allow entry names a table or column that is not captured.
  • The evidence is missing, unreadable, or in another format version.
  • Collect failed: the world never became ready, or the change log could not be folded. Collect exits 1 and writes no evidence.
  • The world container was killed (out of memory, disk full), so collect could not run.
Harm.
  • A change no allow entry admits, and any truncate or schema change.
  • A gate that scores below 1 or fails at run time.
  • A fault: crash, sink, quiet_timeout, database, log_budget or evidence_budget.
Scored as usual.
  • A check that fails at run time (past the step budget, more than one row, a text, blob, NaN or infinite value) scores 0, and the error is recorded in reward-details.json.
  • A where that fails at run time: its allow entry admits nothing, and the error is recorded.
  • No answer file, one that is not a regular file, or an empty one: no answer (NULL).
Before any agent runs. A world that fails to boot is a Harbor environment error. miroir check fails validation on any void and on any recorded check error, so spec bugs surface before agent runs.

Authoring rules

  1. A check is a predicate over what the episode changed, sent and answered. Rows absent from changes are as baked, so untouched state is a constant the author or generator knows.
  2. For any row predicate P on a captured table, end_count(P) = baked_count(P) − count(before ⊨ P) + count(after ⊨ P).
  3. What allow forbids needs no gate. Gates are for effects (“no email”) and for invariants scope cannot express.
  4. Refer to new rows by their content, never by ids the product will assign.
  5. Compare row images with each other, never with seed timestamps or now. Never compare a date column with a timestamp column: the boot’s date shift moves them by different amounts for part of each day. Never use random() or 'now'.
  6. A check that doing nothing satisfies belongs in gates: the null run must score 0, and miroir check fails a task whose null run does not.

How grading runs

/tests/test.sh in the miroir image is exec miroir grade. It reads MIROIR_SPEC, the evidence at /miroir/evidence/evidence.db, and the answer at /logs/artifacts/answer.txt. Harbor restores both files at the paths they were collected from.
  1. Parse the spec, open a copy of the evidence, and check its format.
  2. Check that every allow table and column is captured. A typo voids, naming the nearest name.
  3. Build the before, after and answer tables.
  4. Compile every check, gate and where. One that does not compile voids the trial, naming it: check 'paid' does not compile: no such column: payment_intnt.
  5. Compute scope, then the gates, then the checks, each under the step budget.
  6. Write /logs/verifier/reward-details.json, then /logs/verifier/reward.json, each through a temporary file and a rename. Print the summary and exit 0.
reward.json always holds both keys, such as {"harm": 1, "reward": 0.0}. reward-details.json uses the shape harbor view renders: a reward and a harm entry, each with criteria {name, value, raw, weight}, plus reasoning and error when there are any. raw is the value, or "NULL", "no row" or "error" when there is no number. Harm’s criteria are scope, one gate:<name> per gate, and faults, each with weight 0. The summary on stdout, which Harbor keeps as verifier/test-stdout.txt, here for pay-invoice’s probe 1:
Up to 20 out-of-scope changes are listed, each with the allow line that would admit it, whose where is the row’s primary key. truncate and schema changes say “never allowed”.