Skip to main content
Miroir adds no training code: every trainer below runs Harbor task directories itself. What Miroir gives a trainer is a reward contract, voids it can recognize, and calibration with miroir check.

The reward contract

The grader writes /logs/verifier/reward.json with two keys, always both, finite numbers only:
  • reward, from 0 to 1, is the training signal: the weighted mean of the task’s check scores, or 0 when there is harm.
  • harm, 0 or 1, is a metric: a change no allow entry admits, a failed gate, or a fault. Its mean over a job is the harm rate.
  • A void writes no reward file and exits 75. Harbor records an invalid trial, and a trainer must leave it out of the batch, not count it as 0. A trial voids only for Miroir’s own failures, such as a broken spec or missing evidence (Outcomes).
  • reward-details.json, next to it, holds every check’s score and the out-of-scope changes.
Harm zeroes reward but never makes it negative. A trainer that wants a penalty below 0 composes reward − λ·harm where the consumer allows it (verifiers and OpenEnv do).

Calibrate

Before training, run the agent and model you will train on the train split, with at least two attempts per task. Then keep the tasks whose rewards differ between attempts:
Those are the learnable tasks (classes). A task that scores the same every time, even 0.6, has no reward variance, so a GRPO group gives it no advantage. Harbor runs a subset with -i <task name> (repeatable) or a job config’s datasets[].task_names (job configs).

The committed recipe

The recipe is TRL’s own example, examples/async_grpo_harbor/async_grpo_harbor.py: AsyncGRPO through OpenEnv’s harbor_env, which runs Harbor’s own trials, on a host with two GPUs and Docker, over the Gitea example’s train split.
Not yet run. Its exact commands, pinned to the TRL commit and OpenEnv release they ran with, go in the Gitea example’s Train section after the first run.
What matters to Miroir in it:
  • the tasks are read from local task directories, with no publishing;
  • --sandbox docker runs their compose services (the default, e2b, does not);
  • the harness is a shell, with no MCP;
  • --reward-key reward picks the training signal out of {reward, harm};
  • a rollout the verifier did not score comes back as None, which keeps voids out of the group’s baseline.
The example adds its own shaping on top: up to 0.3 for few tool calls on a fully solved task.

Consumers

Expected means it should work but has not been run yet. Unsupported comes from reading each consumer’s source; none of those was run. Whether training on the Gitea train split improves held-out work is the Gitea example’s open claim.

verifiers HarborTaskset

  • It loads Harbor Hub packages only, so publish the split first. A private split works with Hub credentials.
  • Set agent.deployment.trust_compose = true, use the bash harness, and use a Docker Compose with config --no-env-resolution (2.35.0 or later).
  • reward_key defaults to reward, and harm becomes a metric.
  • Collect hooks are strict, and miroir collect exits 0 whenever it writes evidence.
  • The artifact budget is 32 MiB per trial, shared with main’s /logs/artifacts. Evidence is capped at 16 MiB.