The last loose end from the incident issue —
`ce_compilation_queue_completed_total` incremented in a `finally` that
ran as soon as the job *returned its promise*, so it counted dequeues,
in lockstep with `ce_compilation_queue_dequeued_total`, and
`status().running` was almost always 0.
Completion is now counted when the job settles (fulfilled or rejected),
via a settlement callback rather than by awaiting in the wrapper. The
non-awaiting detail matters: my first attempt awaited `job()` so the
`finally` ran at settlement — and the existing "times out a job that
never settles" test immediately caught that this keeps `_running`
populated forever for a wedged job, reintroducing the exact
`busy`-forever wedge #8813 fixed. (A nice demonstration of that test
paying for itself.) With the callback approach, a never-settling job
correctly never counts as completed, so `dequeued − completed` now
exposes wedged/in-flight jobs — which would have made the original
incident visible directly in Grafana.
New test pins the semantics: counter unchanged while a job is running,
+1 once it settles.
Per discussion: no temp-dir sweeps of any kind (instances are replaced,
never restarted; and multiple CE processes may share a machine), so the
orphaned-dirs observation in #8811 is closed as won't-fix.
Closes#8811.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Part of #8811 (production disk-full incident). Companion to #8812 — this
is the systemic backstop so no future never-settling job can wedge the
system.
Two changes:
**Queue timeout was dead config.** `enqueue()` passed `timeout:
undefined` to p-queue's `add()`; p-queue spreads per-call options over
its defaults, so this overrode and *disabled* the queue-wide timeout
configured from `compilationEnvTimeoutMs` (default 300s). A job whose
promise never settled therefore occupied a queue slot forever — and
since temp dir cleanup only runs when the queue is fully idle, one
wedged slot permanently disabled cleanup and filled the disk. Removing
the override makes the timeout effective: in p-queue v9 a timed-out task
rejects with `TimeoutError`, freeing the slot. (The timeout doesn't kill
underlying work — that remains the exec layer's job — but the system
makes progress again.)
**Uncaught exceptions now actually stop the process.** The handler set
`process.exitCode = 1` assuming the app would "exit naturally", but a
process with live server listeners never does: during the incident the
instance limped on half-dead for hours, passing healthchecks while every
compilation failed. Now it exits after a 1s delay (letting winston flush
its transports), and the load balancer replaces the instance.
The new queue test fails against the previous code: a never-settling job
is never rejected and `status().busy` stays true forever. Also adds
basic enqueue/nested-enqueue coverage.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
