Testless against hono: the receipts

The last testless post ended on a hedge: the tool works, over-approximation is the cost. That’s a comfortable thing to say about your own fixtures, where every edge case is one you wrote on purpose. It’s a different claim on a codebase you don’t control, with a history you didn’t curate and a call graph nobody built to be readable. So I pointed testless at one and wrote down what happened, including the part that made me wince.

The setup

I picked honojs/hono, a real, actively maintained web framework with 2,703 tests across 384 files, cloned shallow at a recent commit. Not a toy repo, not one I’ve touched, not tuned in any way for testless to look good on. I indexed it, then walked back through the last fifty commits looking for ones that touch exactly one non-test source file with a small, single-function fix: no version bumps, no test-only commits, no docs or CI chore commits. Three qualified on the first pass, so I used all three instead of discarding the inconvenient one.

For each commit, the loop was the same:

git checkout <commit>~1
testless index
testless changes --from <commit>~1
testless select --from <commit>~1

changes prints the seeds testless found in the diff: the changed function, plus any new test cases. select walks the call graph from those seeds and prints the tests that could observe the change, ready to hand to a runner with --format args.

Two clean hits, one that isn’t

Here’s what came back:

Commit (changed function)	Selected / total	% of suite	Distinct test files
method-override fix (methodOverride)	12 / 2684	0.45%	1
cache fix (cache)	39 / 2681	1.45%	1
client/utils fix (replaceUrlParam)	1271 / 2683	47.4%	35

The first two are what you’d want from a tool with this pitch: a one-function fix to a middleware selects the tests in that middleware’s own test file and nothing else. Both landed in under 150ms, including the diff walk. The selected sets weren’t just numerically small either; I checked, and they visibly contain the test that actually exercises the changed function, plus its neighbors in the same file.

The third one is the outlier, and it gets equal billing here, not a footnote. replaceUrlParam is a low-level URL helper used by hono’s RPC client, which gets imported, directly or transitively, by dozens of runtime-tests/* files that spin up app.fetch across bun, deno, node, and workerd. That’s a real fan-out, not noise: I checked those selected tests too, and they do call code that runs through the changed function. But 35 files and 1271 tests for a one-function diff is still a lot of blast radius, and it’s worth being honest about why.

Why the 47% happens

testless stats on this repo reports roughly 40% of call edges unresolved, meaning testless can see a call site but can’t pin it to a specific definition. That’s what you get from dynamic dispatch, re-exports, and the kind of indirection a framework’s client code tends to accumulate. The guarantee from the last post still holds: when testless can’t resolve an edge, it doesn’t drop it and hope. It widens by name across the whole codebase instead, because a missed test is the one failure mode the design won’t allow.

That guarantee is exactly why the 47% happened. Widening by name is conservative by construction, and conservative means it can’t tell the difference between “this callback is genuinely reachable from forty files” and “this callback shares a name with forty unrelated things.” Right now it can’t tell those apart, so it assumes the worse one. The fix isn’t a different guarantee, it’s better resolution: SCIP-sharpened type information in place of name-based widening, which is next on the list, specifically because this benchmark is the number that justifies building it.

The other kind of fast

The other number worth writing down is what caching does to index itself. Cold, on a fresh clone with no cache, indexing 384 files took 2011ms of internal parse time. Warm, immediately after, with every file reused from cache: 53ms. That’s roughly 38x on the number that matters, and about 12x on wall time once you count process startup:

$ time testless index
indexed 384 files (384 parsed, 0 reused) in 2011ms
testless index  1.83s user 0.21s system 98% cpu 2.070 total

$ time testless index
indexed 384 files (0 parsed, 384 reused) in 53ms
testless index  0.05s user 0.11s system 95% cpu 0.165 total

select itself doesn’t show that gap. I edited a single function body and ran select cold, then warm, immediately after: 103ms versus 98ms, statistically indistinguishable. select only has to reparse the one changed file no matter what state the cache was in, so a fixed per-invocation overhead of roughly 90 to 100ms (process startup, the git diff) dominates the actual incremental-parse work. The caching win is on full re-index of a repository, not on the single-file select you’d run in CI on every commit.

What I won’t claim

Three commits is a sample, not a rate. I’m not going to tell you testless falls back to run_all 0% of the time on real repos because it did so zero times out of three tries; that’s a fact about three commits, not a distribution. The honest version is narrower: for isolated, single-function diffs at this codebase’s size, every attempt produced a real selection instead of giving up and running everything.

What this changes about CI math

Here’s the part that still holds even with the 47% sitting right there in the table: two of the three real fixes I tested selected 12 and 39 tests out of roughly 2,700, each under 1.5% of the suite, in under 150ms. Those aren’t cherry-picked; they’re two-thirds of every commit I tried. The third one ran half the suite, which is worse than I’d like and exactly as bad as an unresolved call graph makes it. Even averaged in, a tool that turns “rerun everything” into “rerun 1% of it” most of the time, and “rerun half of it” the rest, is still a CI bill that looks nothing like the one you’re paying today.