mirror of
https://github.com/compiler-explorer/compiler-explorer.git
synced 2026-07-22 02:56:57 -04:00
8e6033825a
Part of #8811 (production disk-full incident). During the incident `/healthcheck` stayed green for hours while every compilation failed with ENOSPC: the empty-job queue check passed (one of two queue slots still worked), and the EFS health-file read doesn't touch the root filesystem — so the load balancer never replaced the instance. This adds a free-space check on the filesystem where compilation temp dirs are actually created: `lib/temp.ts` gains `getTempRoot()`, used both by `temp.mkdir()` and the healthcheck, so the two can't drift apart. (That resolves via `os.tmpdir()`; `--tmp-dir` flows into it as `$TMP` via `setupTempDir()` — `/nosym/tmp` in prod.) Below the threshold the healthcheck returns 500 and the load balancer replaces the instance *before* hard failure. A `statfs` failure is itself treated as unhealthy. Configuration: `healthCheckMinFreeSpaceMiB` — required constructor argument (no code default, per review); `amazon.properties` sets 2048 (at the incident's ~300MB/min fill rate that gives the LB several minutes to act); the ceProps fallback is 100 for unconfigured/local instances; 0 disables the check. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>