Skip to main content
A Miroir task is a plain Harbor task directory (Harbor’s task format) whose verifier is miroir grade. This page covers what Miroir adds: a few fixed task.toml fields, the instruction, the reference solution and probes, and the loop that turns reference runs into a grading spec. What a spec can say is in Grading spec; what its SQL can read is in Evidence. The examples are the fixture’s tasks, which pass miroir check.

The task directory

There is no tests/ directory, because the verifier image holds /tests/test.sh. There is no environment/Dockerfile either: both images are prebuilt.

task.toml

fixtures/acme-crm/tasks/pay-invoice/task.toml:
Most of it is the same in every task. The fields Miroir relies on: Leave these out:
  • [agent] user: the agent runs as root, and nothing that grades lives in main.
  • network_mode in [environment], [agent] and [verifier]: only Harbor’s public mode works with a world service. allow_internet is never false.
  • [environment.env]: nothing per task reaches the world, and the credential is in instruction.md.
  • [verifier] user and environment_mode: [verifier.environment] already makes the verifier separate.
Published tasks pin all three images by digest (Publish).

The compose file

environment/docker-compose.yaml puts the world next to main:
  • Name the world service after the product, so the agent sees realistic URLs.
  • Use the same bytes in every task of a world. Between worlds of one product, only the image differs.
  • Keep to these keys. depends_on with service_healthy is what makes main wait for the boot on every provider, and verifiers’ compose runtime rejects others, such as dockerfile_inline and init.
On Daytona and GKE, Harbor gives docker compose up -d a fixed 120 s, which must cover pulling the world and main images and the whole boot. Locally only the healthcheck’s 300 retries limit the boot, so the local gate cannot show an overrun. No Miroir world has run on a cloud sandbox yet.

instruction.md

Write the instruction in this order:
  1. Who the agent acts as: one of the test accounts the world seeded.
  2. Where the product is and how to authenticate: the base URL, the API description’s URL, and the literal credential header.
  3. The product’s conventions, if any.
  4. The request, in the words a colleague or customer would use.
  5. For a question: Write just the number to /logs/artifacts/answer.txt., or the format the task needs.
Never mention Miroir, tools, grading or “submit”. Never write an absolute date: every boot moves the world’s dates to the trial’s day, so write “today”, “overdue” or “last week”.

The reference solution and probes

solution/solve.sh is the reference solution. Harbor’s oracle agent uploads solution/ to /solution in main and runs it. A probe is a wrong way to do the task, in solution/probe-N.sh, with the reward bounds and harm that task.toml declares. A job config runs probe N by setting SOLUTION=N in the agent’s environment, and solve.sh hands over to it before doing anything else:
  • Make every failed call fail the script. Use curl -fsS for every HTTP call, or assert the status you expect. The oracle records the exit code in the trial’s agent/exit-code.txt, and miroir check fails a non-zero one: a probe whose calls fail would score 0 and pass without testing anything.
  • End an answer task by writing the answer, as count-overdue-invoices does: crm GET '/invoices?contact_id=101&overdue=true' | jq length >/logs/artifacts/answer.txt.
  • Write one probe per plausible mistake: doing too much (another row changed), the wrong record, half the work, an email sent twice. Declare max_reward for a probe that must not earn reward, and min_reward for one that must keep partial credit.
  • When a job’s SOLUTION names a probe the task does not have, solve.sh exits 127 and miroir check skips the trial.

Draft the spec

For any task with a reference solution, the fastest way to a first spec is miroir draft. It reads the evidence of reference runs and writes a draft: an allow entry and a check for each row the runs changed, and a check for each request, email and answer they all sent. A new task needs a placeholder spec before it can run, such as checks: [{name: todo, sql: SELECT 0}] with allow: []. Then run the oracle twice, one attempt after the other, and draft from both trials. Here on the fixture’s notify-new-owner, whose hand-written spec is in its task.toml:
Paste it into MIROIR_SPEC, review it, then regrade the same trials instead of running them again.

How the draft is built

  • Values that differ between the runs (updated_at above) are left out of checks and listed at the end. From one run every value looks stable, so a draft from a single run says to run two.
  • An update is allowed on its row, for every column a run changed.
  • An insert is allowed where its integer and boolean values hold, so an agent that words a note differently is not charged with harm. Its check also compares text.
  • Keys the product assigns are left out: an inserted row’s own id, a *_id or *Id column named after that row’s table (label_id for a new labels row), and an answer whose value is the key of a row the same run inserted. A comment names each, and the row is matched by its content instead (authoring rule 4).
  • Text over 256 bytes is not compared. Dates and times compared as constants are listed, since every boot moves them.
  • A change only some runs made is allowed but not checked, and listed as seen in k of N runs.

Review the draft

A draft checks what the reference did, not what the task asks. Before you keep it:
  • Drop checks the instruction does not ask for.
  • Loosen exact text the agent may word its own way. For reassign-stale-deals, the draft checks each note’s exact text; the hand-written spec counts the deals that got a new note, count(DISTINCT deal_id) / 3.0.
  • Turn invariants into gates.
  • Review every *_id value that equals a new row’s key. A reference named by its role, such as Gitea’s poster_id or repo_id, is not recognized and stays a constant.
  • Add probes.
Every draft miroir draft prints parses as a spec, and the runs it was drafted from score reward 1 and harm 0 with it, unless a run did what no spec admits (a truncate, a schema change or a fault), which a comment names. It exits 0 with a draft, 1 when there is nothing to draft (no run changed a row, sent anything or answered, or no check holds in every run), and 2 on an input error, such as a trial of another agent or of two different tasks. An input is a Harbor trial directory, which gives the answer too, or an evidence.db file.

Iterate with regrade

A trial’s evidence stays in its job directory, so a spec change needs no new agent run. Regrade instead:
With allow: [], the grader prints a paste-ready allow line for every change: a quick way to learn what a solution changes. Harbor’s regrade matches trials to tasks by [task].name.

Version bumps

Bump [task].version with every change. harbor run --diff <job> compares each task’s version with the job’s: a different major reruns the trial, a different minor regrades it, and a different patch reuses it. An equal version with different task contents is an error, so a forgotten bump fails loudly. A grader fix is a new miroir image digest in every task’s [verifier.environment]: a minor bump. Check jobs always rerun in full, never with --diff (Validate graders).

Generate tasks

Hundreds of tasks come from a script that writes task directories; Miroir has no task format of its own. The reference is the Gitea example’s tasks.py with its families.py:
  • Read the ids the product gave the seeded rows from the world image, never from a guess. The Gitea seed writes /opt/gitea-env/ids.json, which docker run --rm IMAGE cat reads. Task content never ships inside a world image.
  • Write the same bytes on every run, from one VERSION constant for the whole dataset.
  • Write environment/ byte-identical across a world’s tasks, and pass the images in, so a publish run can give digests.
  • Assert that no spec contains ''', and write each split’s dataset.toml.

An MCP server the product ships

Miroir declares no MCP server, but a product that ships one can be evaluated with no Miroir code:
  1. Run the server as a manifest process listening on an expose port, acting as the task’s account through its own configuration (a token in its process env). The URL must need no credential: Harbor rejects literal credentials in MCP headers, and verifiers’ MCP bridge sends none.
  2. To evaluate with Harbor, add the server per run in a job config, so the same tasks run with and without it: agents: [{name: claude-code, mcp_servers: [{name: product, transport: streamable-http, url: "http://<service>:<port>/mcp"}]}].
  3. For training consumers, declare [[environment.mcp_servers]] with the same fields in a separate dataset. Several Harbor agents and verifiers’ terminus_2 and mini_swe_agent fail on tasks that declare MCP, so these never share a dataset with shell-only tasks.