ONNX Agent v2 checks whether a quantized model is still the same model

What it does

ONNX Agent answers one question carefully: does this optimization preserve the model's output while actually reducing deployment cost? It exports a deterministic linear model to ONNX, quantizes its weights to int8 storage with a reduced 7-bit range, runs both models through ONNX Runtime on CPU, and compares them against an independent NumPy reference.

It is a qualification foundation, not a training service. The historical DSPy and PyTorch experiment code stays in src/ but is excluded from the v2 wheel and the supported test suite.

What changed

Commit 70f112b, merged as pull request #2 on September 11, adds the full v2 pipeline: a seeded 64-by-32 linear graph at fixed batch 8, opset 13 and IR 10; dynamic int8 quantization compared with optimizations disabled versus enabled at one CPU thread; differential validation on 16 held-out batches with fp32 error at most 0.00001 and int8 at most 0.04; and a benchmark with 20 warmups, up to 1000 measured iterations, p50 and p95 latency, byte counts and SHA-256.

The gate is the interesting part. A 5% measured latency improvement is required alongside parity, so an optimization that preserves output but does not pay for itself does not pass.

Get started

Python 3.12 or later and Node 24:

git clone https://github.com/ruvnet/onnx-agent.git
cd onnx-agent
python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
npm ci
npm test

Then run the qualification against the built-in fixture:

python -m onnx_agent qualify --iterations 1000

Expected result: a qualification report with p50 and p95 latency for both models, the differential error against the NumPy reference, and a pass or fail against the parity and 5% latency gates.

{"mcpServers":{"onnx-agent":{"command":"node","args":["/absolute/path/onnx-agent/mcp/cli.mjs","mcp"]}}}

MCP tools are project_status, model_qualify, project_benchmark and project_test, with policy at ruv://onnx-agent/policy. No MCP tool accepts model paths, URLs, custom code, provider settings or credentials. Test execution requires RUV_ALLOW_VALIDATION=1.

Use it today

Practical case: a team about to ship a quantized model because a blog post said it would be faster. Input is the fixture, or your own reviewed local ONNX file through the validate command. Workflow is qualify, read the differential error, read the measured latency delta. Output is a decision backed by numbers from your own hardware.

Acceptance test: run qualify twice with the same iteration count. The parity result must be identical and p50 should be close. Latency that swings wildly between runs means the machine is too noisy for the 5% gate to mean anything.

Push it further

Experimental commentary. The same harness shape would extend to comparing runtimes rather than precisions, using the NumPy reference as the fixed point both are measured against.

Limitation: CPU is the only qualified provider and fallback is disabled. CUDA, TensorRT, training and real model accuracy are explicitly separate work, and local file validation checks structure only — it does not execute the file or certify it against native runtime vulnerabilities. Falsifiable test: qualify a model you know is degraded by quantization. It must fail the parity bound rather than pass on latency alone.