The last loose end from the incident issue —
`ce_compilation_queue_completed_total` incremented in a `finally` that
ran as soon as the job *returned its promise*, so it counted dequeues,
in lockstep with `ce_compilation_queue_dequeued_total`, and
`status().running` was almost always 0.
Completion is now counted when the job settles (fulfilled or rejected),
via a settlement callback rather than by awaiting in the wrapper. The
non-awaiting detail matters: my first attempt awaited `job()` so the
`finally` ran at settlement — and the existing "times out a job that
never settles" test immediately caught that this keeps `_running`
populated forever for a wedged job, reintroducing the exact
`busy`-forever wedge #8813 fixed. (A nice demonstration of that test
paying for itself.) With the callback approach, a never-settling job
correctly never counts as completed, so `dequeued − completed` now
exposes wedged/in-flight jobs — which would have made the original
incident visible directly in Grafana.
New test pins the semantics: counter unchanged while a job is running,
+1 once it settles.
Per discussion: no temp-dir sweeps of any kind (instances are replaced,
never restarted; and multiple CE processes may share a machine), so the
orphaned-dirs observation in #8811 is closed as won't-fix.
Closes#8811.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Part of #8811 (production disk-full incident). Companion to #8812 — this
is the systemic backstop so no future never-settling job can wedge the
system.
Two changes:
**Queue timeout was dead config.** `enqueue()` passed `timeout:
undefined` to p-queue's `add()`; p-queue spreads per-call options over
its defaults, so this overrode and *disabled* the queue-wide timeout
configured from `compilationEnvTimeoutMs` (default 300s). A job whose
promise never settled therefore occupied a queue slot forever — and
since temp dir cleanup only runs when the queue is fully idle, one
wedged slot permanently disabled cleanup and filled the disk. Removing
the override makes the timeout effective: in p-queue v9 a timed-out task
rejects with `TimeoutError`, freeing the slot. (The timeout doesn't kill
underlying work — that remains the exec layer's job — but the system
makes progress again.)
**Uncaught exceptions now actually stop the process.** The handler set
`process.exitCode = 1` assuming the app would "exit naturally", but a
process with live server listeners never does: during the incident the
instance limped on half-dead for hours, passing healthchecks while every
compilation failed. Now it exits after a 1s delay (letting winston flush
its transports), and the load balancer replaces the instance.
The new queue test fails against the previous code: a never-settling job
is never rejected and `status().busy` stays true forever. Also adds
basic enqueue/nested-enqueue coverage.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
- bump pQueue:
https://github.com/sindresorhus/p-queue/releases/tag/v9.0.0
- throwOnTimeout is removed (and is always `true`)
- 0 is not a valid timeout, so, updated tests to pass a big number (no
normal code path assumes 0)
- happy dom: breaking changes: removed commonjs, new jest; nothing
affects us
Removes an old TODO of mine :)
p-queue conditionally returns `undefined` if throwOnTimeout is not set
to true. We currently operate our own timeout check, so I haven't
changed that.
Setting timeout to undefined should give us the correct type without
affecting behavior.
https://github.com/sindresorhus/p-queue/blob/main/source/index.ts#L253
After some time we know either the client or CloudFront will give up on
pending compilations. As such, if we continue to process compilations
after the client's timed out we're just clogging up the compilation
queue with pointless work.
As such, this change now supports the notion of "stale" work which will
be abandoned once it's made it to the front of the queue. Only compiles
coming from the user will be abandoned, so discovery and health checks
are unaffected.
Hopefully this will mitigate the number of nodes marked unhealthy due to
being overloaded: work they were doing was "pointless" anyway, and them
being killed by going unhealthy is equivalent to abandoning all the work
in flight anyway, but this means there's a fighting chance the node will
recover quickly enough to return a "healthy" status.
Makes the Compiler Explorer app, and all the tooling ESM compatible.
Things that have been done:
1. The package.json has `type: module` now
2. All relative imports have a .js ending
3. All directory imports are now directory/index.js to comply with ESM
standards
4. Dependency node-graceful is now imported into tree, because the
package is broken under esm
5. Dependency p-queue has been bumped to 7.x with ESM support
6. Dependency profanities has been bumped to 3.x with ESM support
7. Webpack config is now both ESM and CommonJS compatible
8. Non-ESM compatible imports have been rewritten
9. ESLint configuration has been tweaked to not fail on .js imports
10. Mocha is now hacked together and ran with ts-node-esm
11. Webpack is now hacked together and ran with ts-node-esm
12. Webpack config is now ESM compatible, so that it can be used in the
dev server
13. Cypress code still runs commonjs, and has been excluded from the
tsconfig
14. All sinon mock tests have been commented out, because sinon module
mocks do not work with ESModules (because ESModules are immutable)
A lot of tests are now giving warnings/errors to stdout, yet still pass.
Docenizer codegenerator scripts have been updated, but I did not re-run
them, and instead just changed their code.
---------
Co-authored-by: Matt Godbolt <matt@godbolt.org>
