Benchmark · wolf 0.2.25

# Held-out DOCX agent-edit benchmark

We ran 354 held-out DOCX tasks from the `test` split, including 37 image tasks. We repeated the same tasks across 3 iterations in all four editing workflows with gpt-6.1-sol. This run measured DOCX only. It did not measure PPTX or XLSX.

## Pass rates by editing workflow

A pass required the requested change without the collateral damage checked by our frozen scorer. We report mean pass rates and 95% Student t intervals across complete iterations. These intervals describe variation across repeated runs on the same tasks. They do not estimate variation across all DOCX tasks.

| Workflow | Mean pass rate | 95% t interval |
| --- | --- | --- |
| `wolf-lifecycle` Wolf document lifecycle tools | 95.9% | 87.1% to 100.0% |
| `wolf-3tool` Wolf with three tools | 92.5% | 86.8% to 98.1% |
| `docx-skill` DOCX skill with shell and OOXML editing | 86.8% | 80.5% to 93.2% |
| `python-docx` Python with the python-docx library | 84.1% | 80.0% to 88.1% |

**Route failures.** In iteration 2, we retried 468 route failures once on the same route. The 70 episodes that still failed count as failures in the pass rates.

## Paired differences against the DOCX skill

We subtract the frontier `docx-skill` pass rate from the `wolf-lifecycle` pass rate, pairing the same tasks within each iteration. Differences are in percentage points, with 95% Student t intervals in parentheses.

| Wolf lifecycle route | All 354 tasks | 317 tasks without images |
| --- | --- | --- |
| Frontier (gpt-6.1-sol) | +9.0 (+6.5 to +11.6) | +4.1 (+2.0 to +6.2) |
| Open weights (morph-dsv41flash-1017) | +9.4 (+3.3 to +15.5) | +4.5 (-1.4 to +10.5) |

The open-weights lifecycle arm averaged 96.2% across all tasks (95% t interval 95.4% to 97.0%). That comparison pairs tasks by iteration number across separate runs on different model routes. It does not isolate the effect of the editing workflow. Without the image tasks, its interval includes zero.

## Microsoft Word for Mac open check

In frontier iteration 1, we checked every delivered output in Word for Mac: 1,415 delivered outputs, with identical bytes opened once (1,230 unique files). We did not check later iterations in Word. Opening cleanly is separate from passing the edit scorer.

| Workflow | Clean opens / delivered | Repair prompts |
| --- | --- | --- |
| `wolf-lifecycle` | 353 / 353 | 0 |
| `wolf-3tool` | 354 / 354 | 0 |
| `docx-skill` | 353 / 354 | 1 |
| `python-docx` | 354 / 354 | 0 |

## Method and what you can reproduce

We built wolf 0.2.25 in release mode from source commit `959b39dab4cedb776c7e7a1c85a844298b3d259a`. We authored the harness, tasks and scorer, and checked the recorded verdicts again with the frozen scorer. This is an in-house measurement.

An exact rerun requires the pinned harness and arm environments, held-out documents and edit instructions, split manifest, scoring code and expected outputs, and the original model route configuration. We do not provide that run bundle publicly. This page reports aggregate results; it does not include raw outputs, model transcripts or Word-open receipts.
