← All posts

Engineering · 2026.10.10 · 5 min

Using the CRAP Metric to Find the Functions That Most Need Tests

With mostly agent-written code, high line coverage couldn't show whether the gaps sat in the most complex functions; a CRAP threshold of 8 cleared 200-plus offending functions and turned up 72 issues to fix.

Reading
Theme
Size
Width

Most of this project's code was written by agents, and overall line coverage is high, but that number can't tell you whether the few untested lines that remain happen to sit in the most complex functions. After adopting the CRAP metric, it turned out they did. On the backend, even a loose threshold caught functions like the scheduler with untested branches, and on the frontend the request pipeline had no tests at all. The threshold ended up at 8, both frontend and backend were cleared to 0 functions over it, and I fixed 72 issues found while reading the code along the way.

What CRAP measures

CRAP (Change Risk Anti-Patterns) gives each function a score:

CRAP = complexity² × (1 − coverage)³ + complexity

Lots of branches with no tests means a high score, and a fully tested function scores exactly its complexity. What I wanted was a list pointing at which functions need tests or need splitting, plus a check at PR time. The backend uses the off-the-shelf cargo-crap together with cargo-llvm-cov, and the frontend followed the next day.

Threshold and rules

I started with the tool's default of 30, and across all 2,670 backend functions it caught only 3: the scheduler, the Discord ops notifications, and duration parsing. All three really did have untested branches. 30 was just too loose for this codebase. At 6, a function with three ifs and a match can't pass even when fully tested, so I settled on 8. Since a fully tested function scores its complexity, 8 also works as the complexity cap.

Number line of CRAP thresholds: the tool default of 30 was too loose, catching only 3 of 2,670 backend functions (the scheduler, Discord ops notifications, and duration parsing); 6 was too strict, failing three ifs and a match even when fully tested; 8 was chosen, and since a fully tested function scores its complexity, it also caps complexity.

By default cargo-crap adds 1 for every ?, but line coverage can't measure the path where the error propagates up. Counting them means a function with a long chain of DB calls still scores high when fully tested, and the checkout function's complexity would jump from 7 to 21. So I configured ? not to count as a branch.

The PR rule is that changed files can't contain functions over the threshold, and unchanged files aren't checked. That way there's no need for a baseline, and existing offenders don't block unrelated PRs. I didn't go with failing whenever any function in the repo exceeds the threshold, because as long as a single one remained on main, every PR would be red.

The backend check runs as its own job and isn't triggered by PRs that don't touch the backend. It isn't a GitHub required check, because then every PR would have to report it and burn an extra minute of quota. Instead, the merge script enforces it. Details are in the merge and deploy post.

Frontend: writing my own scorer

Before this, the frontend had only 8 Playwright tests and no unit tests. My rule is that ?. and ?? don't count as branches, and neither the off-the-shelf crap4ts nor ESLint's complexity rule has a setting to turn that off, so I wrote the scoring script myself on top of the TypeScript AST.

Coverage has to include branch exits: both sides of an if or a ternary, and every operand of && and ||, have to be exercised before a function counts as fully tested. Frontend branches mostly live inside a single statement, like JSX's {cond && …}. If you only count lines, a component hits 100% after rendering once and the score means nothing. For coverage I used Vitest's istanbul provider, because in my tests v8 counted lines after an early return as executed even though they never ran.

Cleanup: 200-plus functions, and the bugs found along the way

The moment the threshold dropped to 8, 120 backend functions and 88 frontend functions were over it. For the transition I kept a cleanup list. Files on the list temporarily kept the old threshold, every other file was held to 8 immediately, and each file came off the list once it was cleaned. Wherever the work stopped, nothing already cleaned could slide back.

Flow chart: when the threshold became 8, 120 backend and 88 frontend functions were over it. Files on the cleanup list kept the old threshold, every other file was held to 8, and each file came off the list once cleaned, ending at 0 on both sides. Unused code was deleted, uncovered code got tests, and real branching logic was split in place into named functions.

The principles: delete what nobody uses, add tests for what isn't covered, and only split functions with real branching logic, moving that logic in place into named functions without changing behavior. Before splitting a frontend page, I first wrote tests against the original page until they all passed. After the split I compared every DOM state, plus the order of API and navigation calls, across 966 scenarios in total. That comparison really did catch a bad split: after extracting a block with an await into a new async function, an API call and a navigation swapped order.

Problems found while reading code to write tests were filed as separate issues and fixed. For example:

  • New-login notifications labeled iPhones and iPads as macOS, because the OS check matched the Mac OS X string first
  • The duplicate-registration check used the email without lowercasing it, so a repeat signup with different casing returned 500
  • The reason users picked when pausing the bot only lived on the screen and was never actually sent

The cost, and looking back

Frontend CI checks went from under a minute to 1 min 47 s, an extra minute of quota on every frontend PR. Backend tests went from 2,655 to 2,767, and frontend tests from 0 to 2,604.

Bar chart: backend tests went from 2,655 to 2,767 and frontend tests from 0 to 2,604; frontend CI checks went from under a minute to 1 min 47 s.

The 3 functions caught at a backend threshold of 30 and the untested frontend request pipeline were all things nobody had noticed while line coverage looked high. CRAP put them at the top of the list, which gave me a basis for deciding what to test first. The 72 issues likewise only surfaced because I went to read that code in order to test it.

#Test Coverage#CRAP Metric#Rust#React

Letters

Loading comments…

Next

Engineering

Wiring TradingView Strategy Alerts to Automated Orders on Binance

A BTC daily strategy from backtest to automated Binance orders: only daily had enough samples, tuning the backtest wrapper beat tuning the indicator, the relay syncs to target position, and stops sit on the exchange.