Every Failure Was the Harness

Task 6's worker finished its job — 331 tests green, committed as cfbe595 — and then didn't know how to say so. It doesn't have a "done" button. So it called a tool named finish, which doesn't exist in this harness. Three times. The runner counts unknown-tool calls against a malformed-call limit, hit the limit, and aborted the run. Final status: model_error. The work was fine. The harness just couldn't hear it say so.

That's this whole post in miniature. Across sixteen tasks and fifty recorded runs, the model rarely broke. The scaffolding around it did — repeatedly, in specific, fixable ways. The fixes are in the repo. So is every run that found the bugs. This post walks through both, in first person, with the numbers as they were recorded — not smoothed, not rounded up.

dirtywork 0.3.0 — host-mode hardening — shipped the morning of the 16th. Then we started on the thing that mattered more: a Docker sandbox, so a worker's bash commands run inside a container instead of directly on my machine. Spec and a sixteen-task plan were already written. What followed was two days of executing that plan, task by task, with the same local model that renamed this tool back in its first week now building the isolation layer meant to contain it.

Result, stated plainly up front so the rest of this post has somewhere to land: sixteen of sixteen tasks done, 482 unit tests, and eleven live tests — seven host-sentinel, three lifecycle, one process-flood — green both on my machine and in a new CI gate running real Docker on GitHub's own runners. PR #10 merged. PR #11, the pre-tag follow-up, is green. 0.4.0 is next.

Five roles, doing five different jobs:

Five of the fifty runs weren't qwen at all — three fix rounds (Tasks 12, 15, 16) got escalated to Sonnet when the local model's work was too broken or too integration-heavy to keep grinding on locally. More on that below.

Fifty recorded worker runs: 43 on qwen3-coder-next, 2 on Qwen3.6-35B-A3B for an A/B test, 5 escalated to Sonnet. Statuses: 35 completed, 12 hit the max-turns cap, 3 ended in model_error — including the one above. Ten representative rows, in order:

*Runner recorded this as "completed" — it wasn't. See the A/B section below.

Local runs took 1–4 minutes each. Sonnet and Opus reviews took 3–15 minutes each. Reviews dominated the wall-clock — not the model. All of this ran on one Apple M5 Max with 128 GB of RAM. The full ledger — every run, every ruling — lives in the repo at docs/superpowers/bench/, if you want to check my arithmetic.

Every hard failure in the early runs traced back to the scaffolding, not the model:

1. The guardrail blocked its own worktree. A rule meant to stop a worker from cd-ing out of its sandbox also blocked cd <absolute path> straight into the worktree it was already confined to. Ten blocked calls in one run alone. It derailed an entire fix round.
2. An extra field poisoned every bash call. qwen attached Claude Code's description argument to its bash tool calls. The tool executor didn't recognize it, threw a type error three times running, and aborted the run.
3. The finish tool that doesn't exist. Covered above — the model has no sanctioned way to say "I'm done," so it invents one, and the harness punishes it for asking.
4. git commit -am silently skips new files. The -a flag only stages tracked files. A worker that created new files and committed with -am would leave them uncommitted and not notice.

The first two were fixed the same day, in PR #9: a worktree-aware guardrail, and an executor that drops unrecognized tool arguments instead of choking on them. The fourth got a cheaper fix — an end-of-brief checklist baked into every worker prompt: run the suite, git add -A, commit, then summarize. The third, the missing "done" signal, is still open; it's first on the list for the next harness pass. Turns per task fell from 58 on Task 1 to a 12–27 range afterward. Same model. Different harness.

qwen3-coder-next transcribes a well-specified brief nearly perfectly. It is much less reliable at following instructions that arrive any other way. Rulings delivered as a preamble or an appendix get dropped: Task 2 missed three of them; Task 5 flatly rewrote a function it had been told, elsewhere in the same brief, to keep verbatim. The fix wasn't a smarter model — it was weaving every ruling directly into the body of the instructions the worker was already reading closely.

A related tax: any brief that said "compare your work with commit X" cost 20–40 turns of the model running git show | grep against itself, hunting for the target text. Inlining the exact text it needed to match made that cost disappear.

And a recurring finding that was really about the plan, not the worker: the reviewer treats duplicated code as a defect — my standing rule, DRY and SOLID both count as Important. Tasks 1, 2, 4, 6, and 11 all landed with verbatim-duplicated blocks, and in every case the duplication was already sitting in the plan text. The worker had typed exactly what it was told to type. Each became a small extracted helper in the fix round — which means five findings that were really findings about my own planning, caught because the review gate doesn't know or care who wrote the flawed instruction.

I asked for a controlled comparison, so we ran one: Task 7a, identical brief, identical starting commit, two models. qwen3-coder-next took 34 turns, 1.8 minutes, landed two Important findings, and committed exactly as instructed. Qwen3.6-35B-A3B took 23 turns, 2.1 minutes, found the same two defects — and did not commit. Both models made the identical mistake in edit_file, which says the brief invited the mistake more than either model made it independently.

Task 7b widened the gap. Qwen3.6 stopped after 16 turns with an empty final reply and two failing tests. The runner, seeing no tool calls and no text to act on, recorded the run as completed. It was not completed. That's a harness gap, not a model failure exactly — but it's the kind of gap that would ship a broken run under a green status if nobody was reading transcripts. It's on the list for the next release.

Verdict: keep qwen3-coder-next as the default. Qwen3.6 is attractive — 19 GB against qwen3-coder-next's much larger footprint — once the harness can actually detect an empty reply for what it is.

Unit tests, even 482 of them, don't touch a real Docker daemon. Two bugs only showed up once we did.

Under a 600-process flood, the sandbox was resetting twice per bash call, and the two resets raced each other into git init's template copy — intermittent sandbox_error, "File exists," the kind of flake that's murder to reproduce on demand. Fixed with git init --template=, a docker wait before restart, exactly one reset per call, and a lock around the reset path.

Separately: the worker image's Dockerfile installed .NET via an apt package that doesn't exist for arm64 on Debian 12. It built fine on an amd64 CI runner and failed silently everywhere else that mattered — including my own Mac. Switched to Microsoft's dotnet-install.sh, arch-agnostic, and verified by actually building the image locally rather than trusting the Dockerfile a second time.

Then the new CI gate — real Docker, on GitHub's ubuntu runners — found two more bugs on its first runs, both Linux-only, both invisible on my Mac:

• An image loaded via buildx carries a repo digest that Docker won't run directly — it tries to pull instead. Fixed by executing the image by its local Id.
• The LM Studio client raised EBADF on Linux under Python 3.13 — http.client was closing the socket early on short response bodies, which surfaced as "cannot reach LM Studio" for no visible reason.

Both of those would have hit every Linux user of 0.4.0, on day one, with no local repro. That's the whole argument for a CI gate that runs the real thing instead of a mock.

The model proposes; I still decide the things that don't have a technically correct answer. On the record from this run: keep .NET in the worker image, because invoicr — a real downstream project — needs it. Publish the worker image to GHCR, not Docker Hub, because I don't have a paid Docker Hub org. Push the CI Docker job into the follow-up PR instead of blocking the main one on it. Rewrite the README's "built in one day" story, because it stopped being true somewhere around Task 6. Keep DRY-and-SOLID as a standing review rule. Ask for the A/B in the first place. And when I floated adding live model metrics to this release — my own idea, and still a good one — defer it: it belongs in SP3's dirtywork bench, not stapled on here under deadline pressure.

The frontier model running this loop made its own mistakes, and the ledger records them the same way it records the model's. A shell-variable slip in a dispatch script left a fix round unlaunched and the whole pipeline stalled for about 20 minutes before anyone noticed. A tar-file test construction it prescribed for a fix round was simply wrong — TarFile.close() pads its output to a full 10,240-byte record, which meant the test it specified passed for the wrong reason. A second review caught that the test proved nothing and sent it back for a construction verified end to end instead of assumed.

There are exactly two documented exceptions, in this whole run, to "the controller never edits code directly": one docstring sentence, and one merge-conflict resolution taken in favor of code that had already cleared review. Both are logged as exceptions, not quietly absorbed into the normal flow.

This is one machine — an Apple M5 Max with 128 GB of RAM — running one plan, with one local model family doing almost all of the typing. It is not a benchmark. Any performance figures I've cited for the models themselves (context windows, published SWE-bench-style scores) are vendor self-reported; I haven't independently verified them and you shouldn't take them as more than a starting point.

Cost: the local model's inference was $0 — it's electricity on hardware I already own. The frontier spend went to planning, roughly forty review and re-review passes, and five escalated fix rounds. I can't put a dollar figure on that side, and I'm not going to invent one: I run Claude on a subscription, not a metered API key, so what I actually paid was a session budget — and I hit its ceiling a few times over the weekend and had to wait it out. That's the honest price: not money, but rationed frontier attention. Which is, not coincidentally, the resource this whole tool exists to spend carefully.

0.4.0 ships from this branch. Before that, a small "SP2.5" harness pass, mostly items this run surfaced against itself: a real completion signal so a model never has to invent a finish tool again; failure-kind tracking instead of one undifferentiated abort counter; a guard against an empty reply masquerading as completed; stall detection; and a way to resume a run instead of restarting it from zero. After that, SP3 — which is where dirtywork bench lives, so the next version of this table can build itself.

Coexistence, not replacement: a frontier model planned, reviewed, and ruled. A free local model did the typing. I decided what merges and ships. Every number in this post came from that same run's own record — which is the only way I know to ask you to believe it.