CRAP Score as deterministic AI quality gate

AI produces code fast. The code generated is non-deterministic. Code complexity tends to increase.

Any new project started using AI, looks good. Until it doesn’t. It decays very fast without human steering.

Providing contextual style-guides, or rules as Markdown doesn’t work. You expect deterministic outputs from a non-deterministic process.

AI will either edit your current program, or extend it.

On edit, it will often reuse the same style and standards in the codebase its modifying.

When extending it, it will often do it in a different style.

This creates many “flavours” of code styles.

On code uniformity

For me diversity in code style does not add any benefits to a codebase. I’m not saying it’s bad. I’m saying it’s not good per-se.

My intuition tells me that it’s better to have a unified codebase (KISS. Keeping it boring. Reduce surprises). I agree it’s just a matter of opinion and taste.

Unified and consistent codebases are easier to read for humans. That’s why we have been using linters for the longest time. It might be easier to read for AI agents as well.

CRAP

In Pardon My French, But This Code Is C.R.A.P. (2), Alberto Savoia introduces the Change Risk Analysis and Predictions index:

C.R.A.P.⁡(m)=cyclomaticComplexity⁡(m)2(1−codeCoverage⁡(m)100)3+cyclomaticComplexity⁡(m)
\operatorname{C.R.A.P.}(m)
=
\operatorname{cyclomaticComplexity}(m)^2
\left(1 - \frac{\operatorname{codeCoverage}(m)}{100}\right)^3
+
\operatorname{cyclomaticComplexity}(m)

A high CRAP score (bad) is caused by not having test coverage or high cyclomatic complexity. If you have both, CRAP score is exponentially high.

CRAP index correlates to “risk of change”. The higher the index, the more risky is to make a change, and to introduce a bug. You want to keep it very low.

AI is good, fast and cheap minimising metrics iteratively.

Tools

There are tools for every language. For Go, I’m experimenting with go-crap:

$ go-crap scan --threshold 6 --min 6.0001
Running coverage tests  printonceandwatch ... done [1 in 2.861s]
Analyzing complexity ... done [0 in 191ms]
Processing results ... done [0 in 0s]

┌───┬────────┬────┬───────────────────┬─────────────────────────┬──────────────────────────────────────┐
│   │ CRAP   │ CC │ COVERAGE          │ FUNCTION                │ LOCATION                             │
├───┼────────┼────┼───────────────────┼─────────────────────────┼──────────────────────────────────────┤
│ ✗ │ 156.00 │ 12 │ ░░░░░░░░░░ 0.0%   │ main                    │ cmd/server/main.go:30                │
│ ✗ │ 110.00 │ 10 │ ░░░░░░░░░░ 0.0%   │ *Telegram.Mirror        │ internal/notifier/telegram.go:32     │
│ ✗ │ 110.00 │ 10 │ ░░░░░░░░░░ 0.0%   │ *Worker.DrainOnce       │ internal/printer/worker.go:68        │
│ ✗ │ 56.00  │  7 │ ░░░░░░░░░░ 0.0%   │ *Worker.Run             │ internal/printer/worker.go:42        │
│ ✗ │ 42.00  │  6 │ ░░░░░░░░░░ 0.0%   │ *Worker.waitForDisplay  │ internal/printer/worker.go:106       │
│ ✗ │ 36.78  │  9 │ ███░░░░░░░ 30.0%  │ sanitizeColumnByte      │ internal/escpos/escpos.go:107        │
│ ✗ │ 20.31  │ 14 │ ██████░░░░ 68.2%  │ *Encoder.PrintImage     │ internal/escpos/escpos.go:46         │
│ ✗ │ 12.00  │  3 │ ░░░░░░░░░░ 0.0%   │ *CUPSPrinter.PrintBlank │ internal/printer/cups.go:78          │
│ ✗ │ 12.00  │  3 │ ░░░░░░░░░░ 0.0%   │ *USBPrinter.PrintBlank  │ internal/printer/usb.go:87           │
│ ✗ │ 8.23   │  8 │ ████████░░ 84.6%  │ Validate                │ internal/submission/submission.go:85 │
│ ✗ │ 8.00   │  8 │ ██████████ 100.0% │ Atkinson                │ internal/imageprep/imageprep.go:66   │
│ ✗ │ 7.19   │  5 │ █████░░░░░ 55.6%  │ New                     │ internal/server/server.go:62         │
│ ✗ │ 6.73   │  6 │ ███████░░░ 72.7%  │ *USBPrinter.send        │ internal/printer/usb.go:121          │
│ ✗ │ 6.73   │  6 │ ███████░░░ 72.7%  │ *CUPSPrinter.Print      │ internal/printer/cups.go:49          │
│ ✗ │ 6.73   │  6 │ ███████░░░ 72.7%  │ *USBPrinter.Print       │ internal/printer/usb.go:58           │
│ ✗ │ 6.32   │  5 │ ██████░░░░ 62.5%  │ Open                    │ internal/db/db.go:46                 │
└───┴────────┴────┴───────────────────┴─────────────────────────┴──────────────────────────────────────┘
16/16 function(s) exceed threshold CRAP 6.
Combined CRAP: 818.03 | Average CRAP: 9.09

Each tool supports different formats, find the one most likely to be useful for your agent. This command returns the single worst CRAP function:

$ go-crap scan --threshold 6 --min 6.0001 --format json --top 1 | jq '.entries | map({file, line, function, coverage, cyclomatic, crap})'
Running coverage tests  printonceandwatch ... done [1 in 804ms]
Analyzing complexity ... done [0 in 193ms]
Processing results ... done [0 in 1ms]
[
  {
    "file": "cmd/server/main.go",
    "line": 30,
    "function": "main",
    "coverage": 0,
    "cyclomatic": 12,
    "crap": 156
  }
]

And use the prompt similar to this one, or create your own sub-agent:

/goal lower CRAP score to 6, when running the command `go-crap ...` . Stop if you cannot improve it.

Add checks in the CI and you’ll ensure that the code is simpler, and more tested.

Reference material