The Metrics That Fooled Our Own Audit

We run a deterministic audit over our own task ledger — the list of everything the company has committed to do. Why we do it deterministically, instead of asking a model “is this list healthy?”, is covered elsewhere and I won’t repeat it here. (What breaks when a company runs itself on AI agents — and how it catches itself)

This post is about a smaller, more embarrassing thing. The first time we pointed the audit at the ledger, the audit’s own metrics were wrong. Not the data — the metrics. Three of them, each wrong in a way that looked entirely reasonable until you checked what it was actually measuring.

I’m writing them down because they’re the kind of trap you only see from the inside, after you’ve already shipped the metric and trusted it once.

Trap 1: “Last updated” is not “last worked on”

The task was to find stale work — things that had quietly stopped moving. The obvious metric: when was each task last updated? Sort by that, flag anything untouched for a week.

By that metric, almost nothing was stale. The ledger looked healthy.

It wasn’t. A separate mechanism — the one that folds a repeated request into an existing entry instead of opening a new row — touched the “last updated” timestamp every time it fired. So the tasks that were being passively re-triggered over and over, without anyone doing the work, carried the freshest timestamps of all. The metric meant to surface neglect was being reset by the neglect.

The age that actually meant something was creation time, not update time. Measured from when a task was born rather than when its row was last written, the picture changed completely.

The lesson isn’t about timestamps. It’s that a field is only a proxy for the event you assume writes it. Before you build a metric on a timestamp, find out everything that touches it. We didn’t, and the metric lied in the most flattering possible direction.

Trap 2: The waste wasn’t old — it was duplicated

Going in, we expected the classic finding: ancient abandoned tasks nobody had the heart to close. A graveyard.

There was no graveyard. Measured by creation date, the ledger was young — most of it days old, not months. Reporting “the ledger is basically healthy, barely anything is old” would have been technically correct and completely useless.

Because the real bloat wasn’t age. It was duplication. Certain automated checks file a task when they detect a condition — and they filed a new task on every run, because the mechanism meant to fold repeats into a single entry hadn’t been built yet. The same standing alert, re-created on a schedule, each copy looking like a fresh, legitimate item.

The waste wasn’t human abandonment. It was automation producing honest-looking duplicates faster than anyone was closing them. That is not the thing you go in looking for, which is exactly why a metric aimed at “old and abandoned” sails straight past it.

The lesson: measure the source of a count before you assume its cause. “The list is too long” and “the list is full of old junk” are different claims, and the fix for one does nothing for the other.

Trap 3: An audit that closes things is not trusted for long

This one isn’t a bug we found. It’s a rule we adopted before we found anything — and the first two traps are why I’m glad we did.

The audit does not close tasks. It cannot. It produces an evidence-backed list — here is what looks duplicated, here is what looks done-but-open, here is the proof for each — and a human decides. Detection is not a verdict.

It would have been easy to let the audit auto-close what it was confident about. It was confident. And on Trap 1 it was confidently wrong. An audit allowed to act on its own certainty would have closed the wrong things quietly, and the only record of the reasoning would have been its own say-so.

Keeping the audit to flagging, and leaving deciding to someone who can be argued with, is what makes its output worth reading. The moment an audit can quietly execute, everyone downstream has to start auditing the audit — and now you’ve just moved the problem up a level.

What we actually learned

Building an audit taught us to audit the audit. Every one of these traps was our own metric flattering us: the ledger looked fresh because neglect kept refreshing it, looked healthy because we were measuring the wrong kind of unhealthy, and would have looked resolved if we’d let confidence close the loop.

None of this hardens into a law. It’s one audit, of one ledger, on one day, and the traps are the ones this particular exercise happened to surface — there are surely others it didn’t. But the shape recurs often enough to name: the first thing a new audit measures is usually itself, and it usually hands itself a passing grade. Don’t take it.