I got tired of CI re-running tests nobody doubted

By Itay Wolfish Updated 4 min read

  • rust
  • testing
  • tooling
A robotic arm on a factory line plucks the one glowing light bulb from a row of plain outline bulbs, while the rest roll off into a bin below.

I have spent enough time inside a large monolith to know a specific kind of pain. Touch one utility function, and forty unrelated tests fire, because the build tooling treats the file that function lives in as the unit that matters, not the function itself. Multiply that across a monorepo where dozens of services share a test runner, and every small change reruns entire suites, burning CI minutes re-proving things nobody doubted. That is the complaint behind testless, a Rust CLI that reads a code change and outputs the exact tests it affects, down to the function, using static analysis instead of a test runner’s coverage data.

File-level selectors like jest --findRelatedTests, vitest --changed, and Nx’s affected narrow a run to changed files, which still reruns forty tests for a one-function edit. Runtime coverage tools such as Wallaby and Datadog’s Test Impact Analysis go finer, but they cost money and need an instrumented test process before they produce anything. Nothing free and local selected at function granularity from a plain diff. That gap is what testless set out to close.

One guarantee, no exceptions

The design still commits to one rule: the selected set always contains every test truly impacted by a change. It over-approximates, never under. When indexing hits a file it cannot parse, a corrupt cache, or any other internal error, the output is not a smaller selection: it is run_all, with a reason attached. That rule is what makes everything below trustworthy enough to wire into CI.

From graph to answer

testless select --from <rev> is the command that did not exist a day ago. It walks the graph from the changed defs and prints the impacted tests, each with the full describe/it or package/func chain that identifies it. Exit code 0 means a real selection came back; exit code 2 means testless fell back to run_all and says why. testless changes --from <rev> shows the step before that: the classified seeds, before the walk turns them into tests.

select --format args renders each selected test as a runner invocation instead of a bare name:

vitest run src/math.test.ts -t 'add > handles negatives'
go test ./calc -run '^TestAdd$/^negatives$'
cargo test walk::tests::widens_module_init -- --exact

Each line is ready to paste into a shell or hand to a runner directly, one format per language testless understands.

Here’s that loop recorded end to end, from the diff to the runner-ready output:

testless select run in a terminal: a diff, select --from HEAD, and --format args output

How a diff becomes a selection

Three pieces do the work. A structural differ fingerprints each def’s signature and body separately, ignoring formatting and comments, so a reformatted file selects zero tests instead of the whole suite. A classifier turns each surviving change into a seed: a body edit, a signature edit, a new def, or a changed module top level. A reverse-reachability walk starts at those seeds and follows caller, reader, and container edges back to every test that could observe them. Output comes out in deterministic sorted order, so two runs against the same diff produce the same list.

Module-level seeds widen further, on purpose: importing a module reruns its top-level code, so a change there spreads to every file that imports it, transitively. A test helper keeps propagating impact to whatever calls it, instead of stopping at the helper itself.

Rust joins, and the tool tests itself

Rust is now a third supported language, with its own resolver crate sitting next to the TypeScript and Go ones behind the same shared trait. CI now indexes and selects against testless’s own repository on every push. A cold index runs first, then a warm-cache reindex that should reparse nothing, then a selection run against the working tree that should come back empty on a clean checkout. If the tool cannot analyze itself correctly, nothing else it says can be trusted.

Shipped

testless is on crates.io at 0.5.0, with release-please cutting versions and prebuilt binaries for Linux and macOS, on both x86_64 and aarch64. cargo install testless gets a binary instead of a clone and a build. There’s also a small site at testless.itaywol.tools with the same pitch, for anyone who wants it without a terminal:

The testless website

What over-approximation still costs

The design spec’s edge-case catalog sits at sixty-four entries now: thirty-two for TypeScript and JavaScript, eighteen for Go, fourteen that cut across languages. Each one gets a fixture and a snapshot of the expected selection. The plan is a ground-truth run on top of that: prove the selection is a superset of what a full before/after test run actually touched.

Over-approximation is a guarantee, not a compliment. An unresolved dynamic call widens by name across the whole codebase, so one untyped callback can drag in tests that have nothing to do with the change. There is still no why command to explain a selection, so a wider-than-expected run means reading code, not reading output. Go’s implicit interface satisfaction is not modeled yet, and SCIP-sharpened resolution, which would replace a lot of that name-based widening with real type information, is next. The tool works now. Making the over-approximation smaller is the rest of the job.