Benchmark · wolf 0.2.25
Held-out DOCX agent-edit benchmark
We ran 354 held-out DOCX tasks from the test split, including 37 image tasks. We repeated the same tasks across 3 iterations in all four editing workflows with gpt-6.1-sol. This run measured DOCX only. It did not measure PPTX or XLSX.
Pass rates by editing workflow
A pass required the requested change without the collateral damage checked by our frozen scorer. We report mean pass rates and 95% Student t intervals across complete iterations. These intervals describe variation across repeated runs on the same tasks. They do not estimate variation across all DOCX tasks.
| Workflow | Mean pass rate | 95% t interval |
|---|---|---|
wolf-lifecycleWolf document lifecycle tools | 95.9% | 87.1% to 100.0% |
wolf-3toolWolf with three tools | 92.5% | 86.8% to 98.1% |
docx-skillDOCX skill with shell and OOXML editing | 86.8% | 80.5% to 93.2% |
python-docxPython with the python-docx library | 84.1% | 80.0% to 88.1% |
Route failures. In iteration 2, we retried 468 route failures once on the same route. The 70 episodes that still failed count as failures in the pass rates.
Paired differences against the DOCX skill
We subtract the frontier docx-skill pass rate from the wolf-lifecycle pass rate, pairing the same tasks within each iteration. Differences are in percentage points, with 95% Student t intervals in parentheses.
| Wolf lifecycle route | All 354 tasks | 317 tasks without images |
|---|---|---|
| Frontier (gpt-6.1-sol) | +9.0 (+6.5 to +11.6) | +4.1 (+2.0 to +6.2) |
| Open weights (morph-dsv41flash-1017) | +9.4 (+3.3 to +15.5) | +4.5 (-1.4 to +10.5) |
The open-weights lifecycle arm averaged 96.2% across all tasks (95% t interval 95.4% to 97.0%). That comparison pairs tasks by iteration number across separate runs on different model routes. It does not isolate the effect of the editing workflow. Without the image tasks, its interval includes zero.
Microsoft Word for Mac open check
In frontier iteration 1, we checked every delivered output in Word for Mac: 1,415 delivered outputs, with identical bytes opened once (1,230 unique files). We did not check later iterations in Word. Opening cleanly is separate from passing the edit scorer.
| Workflow | Clean opens / delivered | Repair prompts |
|---|---|---|
wolf-lifecycle | 353 / 353 | 0 |
wolf-3tool | 354 / 354 | 0 |
docx-skill | 353 / 354 | 1 |
python-docx | 354 / 354 | 0 |
Method and what you can reproduce
We built wolf 0.2.25 in release mode from source commit 959b39dab4cedb776c7e7a1c85a844298b3d259a. We authored the harness, tasks and scorer, and checked the recorded verdicts again with the frozen scorer. This is an in-house measurement.
An exact rerun requires the pinned harness and arm environments, held-out documents and edit instructions, split manifest, scoring code and expected outputs, and the original model route configuration. We do not provide that run bundle publicly. This page reports aggregate results; it does not include raw outputs, model transcripts or Word-open receipts.