miroir grade. This page covers what Miroir adds: a few fixed task.toml fields, the
instruction, the reference solution and probes, and the loop that turns reference runs into a
grading spec. What a spec can say is in Grading spec; what its SQL can read is in
Evidence.
The examples are the fixture’s
tasks, which pass
miroir check.
The task directory
tests/ directory, because the verifier image holds /tests/test.sh. There is no
environment/Dockerfile either: both images are prebuilt.
task.toml
fixtures/acme-crm/tasks/pay-invoice/task.toml:
Leave these out:
[agent] user: the agent runs as root, and nothing that grades lives inmain.network_modein[environment],[agent]and[verifier]: only Harbor’spublicmode works with a world service.allow_internetis neverfalse.[environment.env]: nothing per task reaches the world, and the credential is ininstruction.md.[verifier] userandenvironment_mode:[verifier.environment]already makes the verifier separate.
The compose file
environment/docker-compose.yaml puts the world next to main:
- Name the world service after the product, so the agent sees realistic URLs.
- Use the same bytes in every task of a world. Between worlds of one product, only the image differs.
- Keep to these keys.
depends_onwithservice_healthyis what makesmainwait for the boot on every provider, and verifiers’ compose runtime rejects others, such asdockerfile_inlineandinit.
docker compose up -d a fixed 120 s, which must cover pulling
the world and main images and the whole boot. Locally only the healthcheck’s 300 retries limit
the boot, so the local gate cannot show an overrun. No Miroir world has run on a cloud sandbox
yet.
instruction.md
Write the instruction in this order:- Who the agent acts as: one of the test accounts the world seeded.
- Where the product is and how to authenticate: the base URL, the API description’s URL, and the literal credential header.
- The product’s conventions, if any.
- The request, in the words a colleague or customer would use.
- For a question:
Write just the number to /logs/artifacts/answer.txt., or the format the task needs.
The reference solution and probes
solution/solve.sh is the reference solution. Harbor’s oracle agent uploads solution/ to
/solution in main and runs it.
A probe is a wrong way to do the task, in solution/probe-N.sh, with the reward bounds and harm
that task.toml declares. A job config runs probe N by setting SOLUTION=N in the agent’s
environment, and solve.sh hands over to it before doing anything else:
- Make every failed call fail the script. Use
curl -fsSfor every HTTP call, or assert the status you expect. The oracle records the exit code in the trial’sagent/exit-code.txt, andmiroir checkfails a non-zero one: a probe whose calls fail would score 0 and pass without testing anything. - End an answer task by writing the answer, as
count-overdue-invoicesdoes:crm GET '/invoices?contact_id=101&overdue=true' | jq length >/logs/artifacts/answer.txt. - Write one probe per plausible mistake: doing too much (another row changed), the wrong
record, half the work, an email sent twice. Declare
max_rewardfor a probe that must not earn reward, andmin_rewardfor one that must keep partial credit. - When a job’s
SOLUTIONnames a probe the task does not have,solve.shexits 127 andmiroir checkskips the trial.
Draft the spec
For any task with a reference solution, the fastest way to a first spec ismiroir draft. It
reads the evidence of reference runs and writes a draft: an allow entry and a check for each
row the runs changed, and a check for each request, email and answer they all sent.
A new task needs a placeholder spec before it can run, such as
checks: [{name: todo, sql: SELECT 0}] with allow: []. Then run the oracle twice, one attempt
after the other, and draft from both trials. Here on the fixture’s notify-new-owner, whose
hand-written spec is in its task.toml:
MIROIR_SPEC, review it, then regrade the same trials
instead of running them again.
How the draft is built
- Values that differ between the runs (
updated_atabove) are left out of checks and listed at the end. From one run every value looks stable, so a draft from a single run says to run two. - An update is allowed on its row, for every column a run changed.
- An insert is allowed where its integer and boolean values hold, so an agent that words a note differently is not charged with harm. Its check also compares text.
- Keys the product assigns are left out: an inserted row’s own id, a
*_idor*Idcolumn named after that row’s table (label_idfor a newlabelsrow), and an answer whose value is the key of a row the same run inserted. A comment names each, and the row is matched by its content instead (authoring rule 4). - Text over 256 bytes is not compared. Dates and times compared as constants are listed, since every boot moves them.
- A change only some runs made is allowed but not checked, and listed as
seen in k of N runs.
Review the draft
A draft checks what the reference did, not what the task asks. Before you keep it:- Drop checks the instruction does not ask for.
- Loosen exact text the agent may word its own way. For
reassign-stale-deals, the draft checks each note’s exact text; the hand-written spec counts the deals that got a new note,count(DISTINCT deal_id) / 3.0. - Turn invariants into gates.
- Review every
*_idvalue that equals a new row’s key. A reference named by its role, such as Gitea’sposter_idorrepo_id, is not recognized and stays a constant. - Add probes.
miroir draft prints parses as a spec, and the runs it was drafted from score reward
1 and harm 0 with it, unless a run did what no spec admits (a truncate, a schema change or a
fault), which a comment names. It exits 0 with a draft, 1 when there is nothing to draft (no run
changed a row, sent anything or answered, or no check holds in every run), and 2 on an input
error, such as a trial of another agent or of two different tasks. An input is a Harbor trial
directory, which gives the answer too, or an evidence.db file.
Iterate with regrade
A trial’s evidence stays in its job directory, so a spec change needs no new agent run. Regrade instead:allow: [], the grader prints a paste-ready allow line for every change: a quick way to
learn what a solution changes. Harbor’s regrade
matches trials to tasks by [task].name.
Version bumps
Bump[task].version with every change. harbor run --diff <job> compares each task’s version
with the job’s: a different major reruns the trial, a different minor regrades it, and a
different patch reuses it. An equal version with different task contents is an error, so a
forgotten bump fails loudly.
A grader fix is a new
miroir image digest in every task’s [verifier.environment]: a minor
bump. Check jobs always rerun in full, never with --diff
(Validate graders).
Generate tasks
Hundreds of tasks come from a script that writes task directories; Miroir has no task format of its own. The reference is the Gitea example’stasks.py with its
families.py:
- Read the ids the product gave the seeded rows from the world image, never from a guess. The
Gitea seed writes
/opt/gitea-env/ids.json, whichdocker run --rm IMAGE catreads. Task content never ships inside a world image. - Write the same bytes on every run, from one
VERSIONconstant for the whole dataset. - Write
environment/byte-identical across a world’s tasks, and pass the images in, so a publish run can give digests. - Assert that no spec contains
''', and write each split’sdataset.toml.
An MCP server the product ships
Miroir declares no MCP server, but a product that ships one can be evaluated with no Miroir code:- Run the server as a manifest process listening on an
exposeport, acting as the task’s account through its own configuration (a token in its processenv). The URL must need no credential: Harbor rejects literal credentials in MCP headers, and verifiers’ MCP bridge sends none. - To evaluate with Harbor, add the server per run in a job config, so the same tasks run with
and without it:
agents: [{name: claude-code, mcp_servers: [{name: product, transport: streamable-http, url: "http://<service>:<port>/mcp"}]}]. - For training consumers, declare
[[environment.mcp_servers]]with the same fields in a separate dataset. Several Harbor agents and verifiers’terminus_2andmini_swe_agentfail on tasks that declare MCP, so these never share a dataset with shell-only tasks.