MetaHarness Darwin 0.12.0 adds independent confirmation, and its own pilot says keep the baseline

What Darwin is

MetaHarness is a toolkit for building and improving agent harnesses. @metaharness/darwin is its improvement engine: it proposes edits to a harness's documents and measures whether they help. Releases 0.11.0 and 0.12.0 added an opt-in contrastive entry point, @metaharness/darwin/contrastive. Existing evolve behavior is unchanged.

What changed

Darwin 0.11.0 (October 16, 16:48:05Z) introduced bounded contrastive development. It covers invariant and domain edits, measured composition, deduplicated probes, admission budgets and scoped experimental memory. The release labels its benchmark figures as mechanism and integration results, not general model performance.

Darwin 0.12.0 (17:15:10Z) adds confirmContrastive. A candidate that looked better on development tasks is treated as a hypothesis. It is frozen together with its baseline, environment, execution envelope and decision policy. Both are then run on separate, unseen task pairs. The gate checks a minimum case count, minimum wins, a minimum accuracy gain, allowed regressions and an exact one-sided sign test. A tie, too little evidence or a forbidden regression cannot confirm a candidate. One deadline covers the whole confirmation, and costs are reserved conservatively. The result always reports promotion as not authorized.

The release reports its own small public pilot on CLINC150 intent classification. On development, accuracy went from 23/24 to 24/24. On 40 unseen confirmation requests, baseline, candidate and control each scored 38/40. Both superiority tests rejected with p = 1, and the candidate cost 15.68% more to run. The release's conclusion is to keep the baseline. It states that the pilot supports no general accuracy or agent-performance claim, and this newsroom makes none.

What the published package contains

• npm lists @metaharness/darwin 0.12.0 at 17:16:03Z on October 16, tagged latest, requiring Node 20 or later. Version 0.11.0 was published at 16:49:00Z.
• The tarball we downloaded matches the registry's SHA-512 integrity value and holds 317 files, including CONTRASTIVE.md. Its exports include ./contrastive, which exports confirmContrastive and pairedSignPValue.
• The release's source commit ee4ea2b is three commits ahead of main. PR #407, which carries this work, is still open and unmerged. The release says Rust CI is blocked by a Candle 0.8 dependency on a yanked zip 1.x crate (issue #408) and does not claim full Rust CI success.
• The release reports 830 passing tests. The newsroom did not rerun them.

Get started

Prerequisites: Node 20 or later and an empty folder with a package.json. The first command only asks the registry for the version and should print 0.12.0. The last calls the package's exported sign-test helper with no network, model or provider. CONTRASTIVE.md states that five wins and zero losses give p = 0.03125, so that is the expected output.

Documentation and artifact verified only. We read the packed guide and type declarations but did not install the package or run these commands.

npm view @metaharness/darwin@0.12.0 version

npm install @metaharness/darwin@0.12.0

node --input-type=module -e "import { pairedSignPValue } from '@metaharness/darwin/contrastive'; console.log(pairedSignPValue(5, 0));"

A real confirmation needs your own evaluatePair adapter. For each task it must run both documents under identical conditions and return the attested fields listed in CONTRASTIVE.md. The guide's own example disables network, shell and promotion in the envelope. MCP: this package documents no MCP server or configuration. Skills: no npx skills setup is documented.

Use it today

Input: a prompt or harness change that scored better on your development set, plus a separate set of tasks with IDs that never appeared in development. Workflow: freeze the candidate, baseline and policy, then call confirmContrastive with an evaluator that runs both versions and returns pass or fail per task. Output: a verdict with paired counts, digests, costs and receipts, plus promotion: not-authorized. The decision to ship stays with you.

Acceptance: on a confirmation set where the two versions behave the same, the verdict must not confirm the candidate. That is what the release's own pilot showed.

Experimental commentary — acceptance test

One idea is to put confirmation in front of every automated harness edit, so a development win never reaches users without an independent check. Limitation: the API cannot detect reused inputs hidden behind new IDs, or data already seen in training, and repeated tests on the same set need their own multiplicity control. Falsifiable test: feed the gate a candidate that is byte-identical to its baseline. If it ever returns a confirmed verdict, the setup is wrong.

Limits and verification

• Draft history: prepared for the unpublished October 19 candidate; intended for first publication with the October 27 candidate. Source event and verification dates are unchanged.
• The pilot numbers are the release's own, from one small public subset. They show the gate working, not a model or agent improvement.
• Digest and scope checks show identity within the adapter contract. They are not proof that an external evaluator ran honestly.
• Nothing was installed or run by the newsroom, and no provider was contacted.