The last loose end from the incident issue —
`ce_compilation_queue_completed_total` incremented in a `finally` that
ran as soon as the job *returned its promise*, so it counted dequeues,
in lockstep with `ce_compilation_queue_dequeued_total`, and
`status().running` was almost always 0.
Completion is now counted when the job settles (fulfilled or rejected),
via a settlement callback rather than by awaiting in the wrapper. The
non-awaiting detail matters: my first attempt awaited `job()` so the
`finally` ran at settlement — and the existing "times out a job that
never settles" test immediately caught that this keeps `_running`
populated forever for a wedged job, reintroducing the exact
`busy`-forever wedge #8813 fixed. (A nice demonstration of that test
paying for itself.) With the callback approach, a never-settling job
correctly never counts as completed, so `dequeued − completed` now
exposes wedged/in-flight jobs — which would have made the original
incident visible directly in Grafana.
New test pins the semantics: counter unchanged while a job is running,
+1 once it settles.
Per discussion: no temp-dir sweeps of any kind (instances are replaced,
never restarted; and multiple CE processes may share a machine), so the
orphaned-dirs observation in #8811 is closed as won't-fix.
Closes#8811.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
19ae5abcf3