miroir check.
The reward contract
The grader writes/logs/verifier/reward.json with two keys, always both, finite numbers only:
reward, from 0 to 1, is the training signal: the weighted mean of the task’s check scores, or 0 when there is harm.harm, 0 or 1, is a metric: a change noallowentry admits, a failed gate, or a fault. Its mean over a job is the harm rate.- A void writes no reward file and exits 75. Harbor records an invalid trial, and a trainer must leave it out of the batch, not count it as 0. A trial voids only for Miroir’s own failures, such as a broken spec or missing evidence (Outcomes).
reward-details.json, next to it, holds every check’s score and the out-of-scope changes.
reward but never makes it negative. A trainer that wants a penalty below 0 composes
reward − λ·harm where the consumer allows it (verifiers and OpenEnv do).
Calibrate
Before training, run the agent and model you will train on the train split, with at least two attempts per task. Then keep the tasks whose rewards differ between attempts:learnable tasks (classes). A task that scores the same
every time, even 0.6, has no reward variance, so a GRPO group gives it no advantage. Harbor runs a
subset with -i <task name> (repeatable) or a job config’s datasets[].task_names
(job configs).
The committed recipe
The recipe is TRL’s own example,examples/async_grpo_harbor/async_grpo_harbor.py: AsyncGRPO
through OpenEnv’s harbor_env, which runs Harbor’s own trials, on a host with two GPUs and
Docker, over the Gitea example’s train split.
Not yet run. Its exact commands, pinned to the TRL commit and OpenEnv release they ran with, go
in the Gitea example’s
Train section
after the first run.
- the tasks are read from local task directories, with no publishing;
--sandbox dockerruns their compose services (the default, e2b, does not);- the harness is a shell, with no MCP;
--reward-key rewardpicks the training signal out of{reward, harm};- a rollout the verifier did not score comes back as
None, which keeps voids out of the group’s baseline.
Consumers
Expected means it should work but has not been run yet. Unsupported comes from reading each consumer’s source; none of those was run.
Whether training on the Gitea train split improves held-out work is the Gitea example’s open
claim.
verifiers HarborTaskset
- It loads Harbor Hub packages only, so publish the split first. A private split works with Hub credentials.
- Set
agent.deployment.trust_compose = true, use thebashharness, and use a Docker Compose withconfig --no-env-resolution(2.35.0 or later). reward_keydefaults toreward, andharmbecomes a metric.- Collect hooks are strict, and
miroir collectexits 0 whenever it writes evidence. - The artifact budget is 32 MiB per trial, shared with
main’s/logs/artifacts. Evidence is capped at 16 MiB.