mirror of
https://github.com/compiler-explorer/compiler-explorer.git
synced 2026-09-11 11:22:08 -04:00
f0644934e5
Clears the 6 open Dependabot **urllib3** alerts (high/medium), all in `etc/scripts/shortlinkmigration/uv.lock`. The lock pinned `urllib3 1.26.20` (for py<3.10) and `2.6.3`, both vulnerable. The root cause was `requires-python = ">=3.8"` — the outlier among the `etc/scripts` tools (siblings are `>=3.10`/`>=3.12`), which forced urllib3 onto the 1.x line for old Python markers. Change: raise `requires-python` to `>=3.10` (this is a one-off internal migration script; the dev env is 3.12) and re-lock. urllib3 now resolves to **2.7.0** everywhere, and the dual py-marker entries for boto3/botocore/etc. collapse to single versions. Not touched: the **dompurify** alerts (transitive under monaco-editor@0.55.1, latest) — `npm audit` only offers a `--force` fix that downgrades monaco, so per policy those are left for an upstream monaco bump. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shortlink Migration Script
This script migrates Compiler Explorer shortlinks from local file storage to AWS S3/DynamoDB.
Prerequisites
- Python 3.10+
- uv (for dependency management)
- AWS credentials configured (via AWS CLI, environment variables, or IAM role)
Installation
Install dependencies using uv:
cd etc/scripts/shortlinkmigration
uv sync
Usage
Dry Run (recommended first)
uv run python migrate_shortlinks.py \
--local-storage-dir ./lib/storage/data/ \
--s3-bucket storage.godbolt.org \
--s3-prefix ce/ \
--dynamodb-table links \
--aws-region us-east-1 \
--dry-run
Actual Migration
uv run python migrate_shortlinks.py \
--local-storage-dir ./lib/storage/data/ \
--s3-bucket storage.godbolt.org \
--s3-prefix ce/ \
--dynamodb-table links \
--aws-region us-east-1
With Verification
uv run python migrate_shortlinks.py \
--local-storage-dir ./lib/storage/data/ \
--s3-bucket storage.godbolt.org \
--s3-prefix ce/ \
--dynamodb-table links \
--aws-region us-east-1 \
--verify \
--verify-sample-size 20
Options
--local-storage-dir: Path to local storage directory (required)--s3-bucket: S3 bucket name (required)--s3-prefix: S3 key prefix (optional, default: '')--dynamodb-table: DynamoDB table name (required)--aws-region: AWS region (required)--batch-size: Number of files to process in each batch (default: 100)--verify: Run verification after migration--verify-sample-size: Number of random samples to verify (default: 10)--dry-run: Simulate migration without making changes--verbose: Enable detailed logging
How it Works
- Loading: Reads all files from local storage directory
- Sorting: Sorts files by creation time to preserve chronological order
- Deduplication: Checks if content already exists in DynamoDB
- Collision Handling: Extends subhash length if collisions occur (local uses 6+ chars, S3 uses 9+ chars)
- Migration:
- Uploads config to S3:
{prefix}/{fullHash} - Creates DynamoDB entry with metadata
- Uploads config to S3:
- Verification: Optionally verifies random samples
Migration Details
Local Storage Structure
- Files named by their unique subhash (minimum 6 characters)
- Each file contains:
{prefix, uniqueSubHash, fullHash, config}
S3/DynamoDB Structure
- S3 key:
{s3_prefix}{6-char-prefix}/{fullHash} - DynamoDB partition key: 6-character prefix
- DynamoDB sort key: unique_subhash (minimum 9 characters)
Notes
- The script preserves creation timestamps from file metadata
- Existing entries are skipped (deduplication)
- Collision resolution may result in longer subhashes
- Progress is logged every batch_size entries
- Errors are logged but don't stop the migration