## What & why
We used to enforce the BSD-2-Clause license banner on every source file
via `eslint-plugin-header`. That check was lost when we migrated from
ESLint to Biome (#7033), which has no equivalent rule. This PR
reinstates it as a standalone node script, modelled on the existing
`etc/scripts/check-frontend-imports.js`.
### Why a script rather than a Biome rule
Biome 2.x has no built-in license/header rule and no plugin equivalent
to `eslint-plugin-header`; its experimental GritQL plugins aren't suited
to whole-file-prefix matching. A script needs zero new dependencies and
gives full control over scope and exemptions.
## The check (`etc/scripts/check-license-headers.js`)
A file "has an appropriate banner" if, ignoring an optional shebang, it
opens with a `// Copyright (c) …` line **and** contains the BSD-2-Clause
disclaimer body. The year and copyright holder are intentionally **not**
constrained — the tree legitimately has many holders (Compiler Explorer
Authors, Arm, Microsoft, HRT, individuals). The `(c)`/`(C)` marker is
matched case-insensitively.
**Scope:** `.ts/.js/.mjs/.cjs` under `lib/ static/ shared/ types/ test/
cypress/`.
**Exempt:** generated files (`lib/asm-docs/generated`), vendored
(`docenizer/vendor`), `.d.ts`, and three third-party ports that carry
their own upstream license — `static/ansi-to-html.ts` (MIT),
`lib/node-graceful.ts` (MIT), `shared/rison.ts` (Nanonid/rison port).
**Wired into:** CI (`test-and-deploy.yml`), `npm run check`, the `make
pre-commit` target, the husky pre-commit hook, and `lint-staged` (per
staged file).
Usage:
```
node ./etc/scripts/check-license-headers.js # scan the tracked tree
node ./etc/scripts/check-license-headers.js <files...> # scan specific files (lint-staged)
```
## Backfill
The check surfaced **42 CE-authored files** missing the banner. This PR
backfills them all with the standard `Copyright (c) <year>, Compiler
Explorer Authors` banner, using each file's **git creation year**
(added-at-this-path, so no `--follow` rename artifacts).
## Verification
- `check-license-headers` → clean (was 42 failures)
- `biome check` on all source → no fixes needed (banner format matches
existing convention)
- `tsc` backend + frontend + tests → clean
- pre-commit gauntlet (lint, ts-check, related tests: 528 passed) ran on
commit
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes#8817. Kept deliberately simple per discussion — the conan server
is CE's own infrastructure, so these are hygiene bounds, not attack
mitigations:
- A generous fixed 2GiB cap on the total declared size of extracted
files (tar-stream enforces entry bodies match their headers, so summing
`header.size` bounds bytes written). Real packages are tens to a few
hundred MiB; hitting this means a packaging error or a corrupt/bombed
archive, and the extraction rejects cleanly through the existing error
path.
- The package URL conan returns must be http(s). No redirect
restrictions (conan may legitimately hand out redirecting/presigned
URLs), no config plumbing.
Both paths tested (cap exercised by lowering the limit on the instance
under test; scheme via a `file://` URL).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Part of #8811 (production disk-full incident).
## The incident bug
If a conan library package download's connection died mid-transfer,
`downloadAndExtractPackage()`:
- threw an **uncaught exception** (`TypeError: terminated`): the error
was emitted on the `Readable.from(res.body)` wrapper, which had no
`'error'` listener — `.pipe()` neither attaches nor forwards one;
- **never settled its promise**: gunzip/tar saw neither an error nor an
end, so neither `resolve` nor `reject` was ever called. This wedged a
compilation queue slot forever, which permanently disabled temp dir
cleanup (it only runs when the queue is idle) and filled the disk on
prod.
The download path is rewritten around `stream.pipeline()`, which
propagates errors through every stage and guarantees settlement.
Per-entry file writes also go through `pipeline()`, so write errors
propagate (via `extract.destroy`) instead of being silently dropped, and
`next()` fires after the file is fully flushed rather than on stream
`'end'`.
## Extraction hardening (from review + an adversarial security pass)
- **Zip-slip guard anchored at the per-library extraction root**
(`downloadPath/<libId>`; plain `downloadPath` for `extractAllToRoot`): a
package can no longer write over a sibling library's files or the
compilation's own. The check is `path.relative`-based — immune to prefix
collisions (`/tmp/pkg` vs `/tmp/pkg-evil`) and to directories merely
*named* with leading dots.
- **Only regular-file entries are ever written**: directories, symlinks,
hardlinks and other types are drained and skipped, including malformed
entries (e.g. a directory claiming non-zero size) whose body could
otherwise land on disk as a file. CE never creates links of any kind;
prod additionally runs under the `/nosym/tmp` nosymfollow mount.
- **Zero-length files extract correctly** (the pre-pipeline code created
them as a side effect of an early `createWriteStream` — with a leaked
fd; the first pipeline version dropped them entirely).
- **Logs preserve stacks** (error objects passed to winston, not
interpolated) and the archive-controlled entry name is JSON-stringified.
Deliberately out of scope (filed as #8817): decompressed-size caps and
`packageUrl` scheme/redirect validation — defense-in-depth against our
own conan server, not blockers. See the review-convergence comment below
for vectors evaluated and rejected with rationale.
## Tests
`test/buildenvsetup-ceconan-tests.ts` (new), against a local HTTP
server:
- **Severed mid-stream download rejects rather than hanging** — against
the pre-fix code this reproduces the production failure exactly
(unhandled `TypeError: terminated` + timeout).
- Happy-path extraction including a zero-length file.
- 404 → rejection.
- Zip-slip: full escape and sibling-library escape are skipped; a
`..`-named directory inside the library root still extracts.
- A malformed sized-directory entry settles (no wedged promise) and
writes nothing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Enable Biome's `organizeImports` with groups matching the original
ESLint `import/order` configuration:
1. **Node builtins** (`node:fs`, `path`, etc.)
2. *(blank line)*
3. **Third-party packages** (`express`, `@sentry/node`, etc.)
4. *(blank line)*
5. **Local/relative imports** (`../foo.js`, `./bar.js`, aliases)
This resolves the inconsistency where Biome wasn't enforcing import
grouping, meaning new files would lose the blank-line separation that
the old ESLint config enforced.
### Impact
- **354 files** updated out of 738 checked (~48%)
- **+188 / -240 lines** (net -52) — almost entirely single blank line
additions/removals between import groups
- No import reordering; purely group separator consistency
Fixes#7373🤖 Generated by LLM (Claude, via OpenClaw)
- latest biome, and fix its configuration
- fixes "static" content to be globally configured too (instead of
per-line)
- fixes issues:
- imports fixed up
- `Date.now()` vs `+new Date()`
- some unused things `_` prefixed
After discussion with the team, turned off the unused parameter warning.
- Uses strings across the board in the UI part (no functional change
here from before; all state is the same as it was before).
- Sends _arrays_ in the POST, using the same `splitArguments` code as
the backend.
- Backend _still supports_ strings (though doesn't advertise), also
using same `splitArguments`.
- Moves `splitArguments` into common utils, and rephrases to avoid
unnecessary use of underscore and ES2021+ code.
Tested locally:
- with both old and new client code (ran new backend and old webcode to
show sending strings still works)
- with creating and removing tool windows (checked with `ldd` locally)
- with various strings on the client `moo foo "this is bad" #moo` and
even "error" things like `this is "badger` (with a missing close quote).
All works as you'd expect
Happy to break the "move the splitArguments" code into a separate PR if
that'd be easier to review separately.
Fixes#7195
This includes among others:
- Proper tsification of various tools code,
- Elimination of the `CompilationInfo2` type,
- Use of `CompilationInfo` instead of `Record<any, any>` in some places
- These lines in CompilationResult:
```
// Temp hack until we get all code to agree on type of asm
asm?: ResultLine[] | string;
```
The next task would be to get all code to agree on the type of
CompilationResult.asm, thereby enabling fixing of most the remaining
TSification.
Mindless replacements of the form
`_.filter(options, option =>...` --> `options.filter(option =>...`.
One not *entirely* mindless replacement at the bottom of
compiler-dropin-tool.ts :
```
- return _.filter(pathFilteredFlags) as string[];
+ return pathFilteredFlags.filter(Boolean) as string[];
```
6 files can now stop importing underscore.
- use async file access when looking for llc
- don't use the random customCwd in the cache hash
- don't lookup "log cache accesses" in props every access
- turn on compiler cache access logging in discovery mode
- error on missing wine executable config
- atomically cache on disk file cache
Makes the Compiler Explorer app, and all the tooling ESM compatible.
Things that have been done:
1. The package.json has `type: module` now
2. All relative imports have a .js ending
3. All directory imports are now directory/index.js to comply with ESM
standards
4. Dependency node-graceful is now imported into tree, because the
package is broken under esm
5. Dependency p-queue has been bumped to 7.x with ESM support
6. Dependency profanities has been bumped to 3.x with ESM support
7. Webpack config is now both ESM and CommonJS compatible
8. Non-ESM compatible imports have been rewritten
9. ESLint configuration has been tweaked to not fail on .js imports
10. Mocha is now hacked together and ran with ts-node-esm
11. Webpack is now hacked together and ran with ts-node-esm
12. Webpack config is now ESM compatible, so that it can be used in the
dev server
13. Cypress code still runs commonjs, and has been excluded from the
tsconfig
14. All sinon mock tests have been commented out, because sinon module
mocks do not work with ESModules (because ESModules are immutable)
A lot of tests are now giving warnings/errors to stdout, yet still pass.
Docenizer codegenerator scripts have been updated, but I did not re-run
them, and instead just changed their code.
---------
Co-authored-by: Matt Godbolt <matt@godbolt.org>
This PR turns comma-dangle and indent eslint rules on for lib/. These
are rules inherited from the eslint config for static/, this PR just
makes things more consistent. Also turned
@typescript-eslint/no-var-requires back on while I was here.
Happy Superbowl Sunday!
Watching 3rd down conversions and converted code.
Converted base-compiler-tests
Snuck in analysis-tests & compilfer-finder tests because they are small
Added a couple of testing helpers to tests/utils.ts and went back and
implemented them in cache-tests
<!-- THIS COMMENT IS INVISIBLE IN THE FINAL PR, BUT FEEL FREE TO REMOVE
IT
Thanks for taking the time to improve CE. We really appreciate it.
Before opening the PR, please make sure that the tests & linter pass
their checks,
by running `make check`.
In the best case scenario, you are also adding tests to back up your
changes,
but don't sweat it if you don't. We can discuss them at a later date.
Feel free to append your name to the CONTRIBUTORS.md file
Thanks again, we really appreciate this!
-->
---------
Signed-off-by: Marc Poulhiès <dkm@kataplop.net>
Co-authored-by: Jeremy Rifkin <51220084+jeremy-rifkin@users.noreply.github.com>
Co-authored-by: Gaurav Gautam <gautam1168@users.noreply.github.com>
Co-authored-by: Matt Godbolt <matt@godbolt.org>
Co-authored-by: Goooler <wangzongler@gmail.com>
Co-authored-by: Marc Poulhiès <dkm@kataplop.net>
Co-authored-by: Mats Jun Larsen <me@supergrecko.com>
Co-authored-by: Jake Hemstad <jhemstad@nvidia.com>
Co-authored-by: Ross Brunton <bruntonross+github@gmail.com>
Co-authored-by: Fábio de Souza Villaça Medeiros <fabiosvm@outlook.com>
Co-authored-by: Joel Falcou <joel.falcou@lri.fr>
Co-authored-by: Patrick Quist <partouf@gmail.com>
* Handle zero-sized files
It appears that having a zero-sized file won't call our `next()` handler which means we wedge forever and eventually time out.
This is a workaround. Upstream issue filed as https://github.com/mafintosh/tar-stream/issues/145
- latest sentry, tar-stream, which, some yamljs versions
- latest eslint-* stuff
- latest webpack manifest
- Applies all the automatic fixes for newer lint rules
- Bump the webpack version
applies new tslint stuff
* The Grand Reformat
- everything made prettier...literally
- some tweaks to include a few more files, including documentation
- minor changes to format style
- some tiny `// prettier-ignore` changes to keep a few things the way we like them
- a couple of super minor tweaks to embedded document types to ensure they format correctly
