WolfSDKWolfXLWolfPPTWolfDocxWolfRender

Benchmark · wolf 0.2.25

Held-out DOCX agent-edit benchmark

We ran 354 held-out DOCX tasks from the test split, including 37 image tasks. We repeated the same tasks across 3 iterations in all four editing workflows with gpt-6.1-sol. This run measured DOCX only. It did not measure PPTX or XLSX.

Pass rates by editing workflow

A pass required the requested change without the collateral damage checked by our frozen scorer. We report mean pass rates and 95% Student t intervals across complete iterations. These intervals describe variation across repeated runs on the same tasks. They do not estimate variation across all DOCX tasks.

Pass rates on gpt-6.1-sol
WorkflowMean pass rate95% t interval
wolf-lifecycle
Wolf document lifecycle tools
95.9%87.1% to 100.0%
wolf-3tool
Wolf with three tools
92.5%86.8% to 98.1%
docx-skill
DOCX skill with shell and OOXML editing
86.8%80.5% to 93.2%
python-docx
Python with the python-docx library
84.1%80.0% to 88.1%

Route failures. In iteration 2, we retried 468 route failures once on the same route. The 70 episodes that still failed count as failures in the pass rates.

Paired differences against the DOCX skill

We subtract the frontier docx-skill pass rate from the wolf-lifecycle pass rate, pairing the same tasks within each iteration. Differences are in percentage points, with 95% Student t intervals in parentheses.

Paired pass-rate differences in percentage points
Wolf lifecycle routeAll 354 tasks317 tasks without images
Frontier (gpt-6.1-sol)+9.0 (+6.5 to +11.6)+4.1 (+2.0 to +6.2)
Open weights (morph-dsv41flash-1017)+9.4 (+3.3 to +15.5)+4.5 (-1.4 to +10.5)

The open-weights lifecycle arm averaged 96.2% across all tasks (95% t interval 95.4% to 97.0%). That comparison pairs tasks by iteration number across separate runs on different model routes. It does not isolate the effect of the editing workflow. Without the image tasks, its interval includes zero.

Microsoft Word for Mac open check

In frontier iteration 1, we checked every delivered output in Word for Mac: 1,415 delivered outputs, with identical bytes opened once (1,230 unique files). We did not check later iterations in Word. Opening cleanly is separate from passing the edit scorer.

Word open results for delivered outputs
WorkflowClean opens / deliveredRepair prompts
wolf-lifecycle353 / 3530
wolf-3tool354 / 3540
docx-skill353 / 3541
python-docx354 / 3540

Method and what you can reproduce

We built wolf 0.2.25 in release mode from source commit 959b39dab4cedb776c7e7a1c85a844298b3d259a. We authored the harness, tasks and scorer, and checked the recorded verdicts again with the frozen scorer. This is an in-house measurement.

An exact rerun requires the pinned harness and arm environments, held-out documents and edit instructions, split manifest, scoring code and expected outputs, and the original model route configuration. We do not provide that run bundle publicly. This page reports aggregate results; it does not include raw outputs, model transcripts or Word-open receipts.