mirror of
https://github.com/compiler-explorer/compiler-explorer.git
synced 2026-07-22 02:56:57 -04:00
a80b7eb210
Part of #8811 (production disk-full incident). ## The incident bug If a conan library package download's connection died mid-transfer, `downloadAndExtractPackage()`: - threw an **uncaught exception** (`TypeError: terminated`): the error was emitted on the `Readable.from(res.body)` wrapper, which had no `'error'` listener — `.pipe()` neither attaches nor forwards one; - **never settled its promise**: gunzip/tar saw neither an error nor an end, so neither `resolve` nor `reject` was ever called. This wedged a compilation queue slot forever, which permanently disabled temp dir cleanup (it only runs when the queue is idle) and filled the disk on prod. The download path is rewritten around `stream.pipeline()`, which propagates errors through every stage and guarantees settlement. Per-entry file writes also go through `pipeline()`, so write errors propagate (via `extract.destroy`) instead of being silently dropped, and `next()` fires after the file is fully flushed rather than on stream `'end'`. ## Extraction hardening (from review + an adversarial security pass) - **Zip-slip guard anchored at the per-library extraction root** (`downloadPath/<libId>`; plain `downloadPath` for `extractAllToRoot`): a package can no longer write over a sibling library's files or the compilation's own. The check is `path.relative`-based — immune to prefix collisions (`/tmp/pkg` vs `/tmp/pkg-evil`) and to directories merely *named* with leading dots. - **Only regular-file entries are ever written**: directories, symlinks, hardlinks and other types are drained and skipped, including malformed entries (e.g. a directory claiming non-zero size) whose body could otherwise land on disk as a file. CE never creates links of any kind; prod additionally runs under the `/nosym/tmp` nosymfollow mount. - **Zero-length files extract correctly** (the pre-pipeline code created them as a side effect of an early `createWriteStream` — with a leaked fd; the first pipeline version dropped them entirely). - **Logs preserve stacks** (error objects passed to winston, not interpolated) and the archive-controlled entry name is JSON-stringified. Deliberately out of scope (filed as #8817): decompressed-size caps and `packageUrl` scheme/redirect validation — defense-in-depth against our own conan server, not blockers. See the review-convergence comment below for vectors evaluated and rejected with rationale. ## Tests `test/buildenvsetup-ceconan-tests.ts` (new), against a local HTTP server: - **Severed mid-stream download rejects rather than hanging** — against the pre-fix code this reproduces the production failure exactly (unhandled `TypeError: terminated` + timeout). - Happy-path extraction including a zero-length file. - 404 → rejection. - Zip-slip: full escape and sibling-library escape are skipped; a `..`-named directory inside the library root still extracts. - A malformed sized-directory entry settles (no wedged promise) and writes nothing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>