compiler-explorer/docs/Configuration.md

# Compiler Explorer Configuration System

This document describes the configuration system used by Compiler Explorer, which is based on `.properties` files that
follow a hierarchical structure with inheritance and specialized handling for various property types.

## Configuration File Structure

Compiler Explorer uses `.properties` files located in the `etc/config/` directory for all its configuration needs. These
files follow a simple key-value format:

```
key=value
```

Leading and trailing whitespace around the value is trimmed.

Comments are lines that start with a hash character, and are not processed:

```
# This is a comment
key=value # This is also a comment
```

### Property Append Syntax

You can append to an existing string property using the `+=` operator.
Leading whitespace in the appended value is preserved, while trailing whitespace is trimmed.

```
base.options=-Wall
base.options+= -Wextra
```

This results in `base.options` having the value `-Wall -Wextra`.

> [!NOTE]
> The `+=` operator only works for string properties that have already been defined.

### File Naming Convention

Configuration files follow a specific naming pattern:

```
CATEGORY.ENVIRONMENT.properties
```

Where:

- `CATEGORY` represents a configuration category (like `c++`, `rust`, `compiler-explorer`, `aws`)
- `ENVIRONMENT` represents where the configuration will be used (like `defaults`, `amazon`, `local`)

For example:

- `c++.defaults.properties` - Default C++ configuration
- `c++.amazon.properties` - C++ configuration for the Amazon environment (used in production)
- `c++.local.properties` - Local C++ configuration (ignored by git, for personal settings)

## Configuration Hierarchy

The system loads configuration files in a specific order, creating a cascade of settings. The actual hierarchy used for
property resolution, from lowest to highest priority, is:

1. `defaults` - Base configuration that applies to all environments
2. Environment-specific settings (such as `dev`, `beta`, `staging`, `amazon`)
3. Platform-specific environment settings (like `dev.linux`, `staging.darwin`)
4. Platform-specific settings (like `linux`, `darwin`, `win32`)
5. Host-specific settings (using the machine's hostname)
6. `local` - User-specific configuration that overrides all others (can be disabled with the `--no-local` flag)

For example, if you're running on a Linux machine named "myserver" in the "staging" environment, a property would be
looked up in this order:

1. `etc/config/category.defaults.properties`
2. `etc/config/category.staging.properties`
3. `etc/config/category.staging.linux.properties`
4. `etc/config/category.linux.properties`
5. `etc/config/category.myserver.properties`
6. `etc/config/category.local.properties`

This means if the same property is defined in multiple files, the one from the most specific environment will be used.

### Important Note on Group Properties

The hierarchy/cascade only works for top-level properties. Group properties (discussed below) do not inherit across
different environment files. That is, if you define a group property in `c++.defaults.properties`, but that group is
defined without that property in `c++.amazon.properties`, the property will not be carried forward to the amazon
environment.

## Property Types

While all properties are stored as strings in the files, the system automatically converts values to appropriate types
using the `toProperty` function:

1. **Strings** - The default type for all properties
   ```
   compiler.clang.name=Clang
   ```

2. **Booleans** - Values `true`, `yes`, `false`, `no` are converted to boolean values
   ```
   compiler.clang.supportsBinary=true
   ```

3. **Numbers** - Numeric strings are converted to integers or floats
   ```
   compiler.clang.timeout=10
   ```

4. **Special Version Properties** - Properties ending with `.version` or `.semver` are never converted to numbers, even
   if they look like numbers. This preserves version formatting with leading zeros or other special characters:
   ```
   compiler.gcc123.version=9.0.0    # Remains the string "9.0.0" instead of becoming a number
   ```

## Lists and Separators

Many properties in Compiler Explorer represent lists of values. These use specific separators:

1. **Colon-separated lists** (`:`) - Most commonly used for lists of identifiers, compiler names, etc.
   ```
   compilers=gcc:clang:msvc
   ```

2. **Pipe-separated lists** (`|`) - Used for argument lists, particularly when options might contain colons
   ```
   compiler.clang.demanglerArgs=-n|-C|--no-verbose
   ```

## Compiler and Group Configuration

### Basic Compiler Configuration

Compilers are configured using properties with the `compiler.ID` prefix:

```
compiler.gcc.name=GCC
compiler.gcc.exe=/usr/bin/gcc
compiler.gcc.options=-Wall
```

### Group System with &IDENTIFIER

A key feature of the configuration system is the ability to define groups of compilers with shared settings using `&`
syntax:

```
compilers=&gcc:&clang:specific_compiler

group.gcc.compilers=gcc7:gcc8:gcc9
group.gcc.groupName=GCC
group.gcc.supportsBinary=true

group.clang.compilers=clang9:clang10
group.clang.groupName=Clang
group.clang.intelAsm=-mllvm --x86-asm-syntax=intel
```

In this example:

1. The `compilers` list includes two groups (`&gcc` and `&clang`) and one individual compiler
2. Each group defines its own list of compilers
3. Properties set at the group level (like `supportsBinary` or `intelAsm`) are inherited by all compilers in that group

Groups can contain other groups by using the `&` syntax within a group's compilers list:

```
group.newer.compilers=&clang:&gcc
group.newer.groupName=Modern Compilers
```

### Property Inheritance

Properties are inherited from groups to individual compilers. If both a group and a compiler define the same property,
the compiler's value takes precedence. This allows for group-wide defaults with compiler-specific overrides.
**There are some notable exceptions to this rule**: some unique-per-compiler properties are not inherited from groups,
but we hope to fix this. See [this GitHub issue](https://github.com/compiler-explorer/compiler-explorer/issues/7150).

## Configuration Keys

Common configuration keys include:

| Key Name       | Type    | Description                                         |
|----------------|---------|-----------------------------------------------------|
| name           | String  | Human-readable name displayed to users              |
| exe            | Path    | Path to the executable                              |
| options        | String  | Default compiler options                            |
| compilerType   | String  | Refers to the handler class in `lib/compilers/*.ts` |
| intelAsm       | String  | Flags for Intel assembly syntax                     |
| supportsX      | Boolean | Capability flags for various features               |
| versionFlag    | String  | Flag to pass to compiler to get version             |
| demanglerArgs  | String  | Arguments for the demangler (pipe-separated)        |
| objdumperArgs  | String  | Arguments for the object dumper (pipe-separated)    |
| instructionSet | String  | Default instruction set for the compiler            |

## Variable Substitution

Some properties support variable substitution to make configuration more flexible. The most common variables are:

### Path Variables

1. **${exePath}**: Replaced with the directory path of the compiler executable
   ```
   ldPath=/opt/compiler-explorer/lib/|${exePath}/../lib/
   ```

2. **${ceToolsPath}**: Replaced with the path to the Compiler Explorer tools directory
   ```
   demangler=${ceToolsPath}/demangler
   ```

### Special Properties for Environment Variables

Environment variables can be configured using a special format:

```
envVars=VAR1=value1:VAR2=value2
```

This is parsed and converted into environment variable key-value pairs that are passed to the compiler when it's
executed.

### Path Lists

Multiple types of path lists exist in the system, mainly using two separator styles:

1. **Colon-separated**: Used for general lists (compilers, versions, etc.)
   ```
   compilers=gcc:clang:msvc
   ```

2. **Pipe-separated**: Used for path lists and command arguments
   ```
   ldPath=/path/to/lib1|/path/to/lib2
   demanglerArgs=-n|-C|--no-verbose
   ```

Note that path-style properties often support variable substitution as shown above.

## Advanced Configuration Features

### Property Debugging

You can enable detailed debugging of property resolution by using the `--prop-debug` flag when starting Compiler
Explorer. This shows every property lookup, including where properties are being overridden and which configuration
source they come from.

### Remote Compiler Configuration

Compiler Explorer supports defining remote compilers using a special syntax in the configuration:

```
compilers=local1:local2:remote@hostname:port
```

When a compiler ID contains the `@` symbol followed by a hostname and port, it is interpreted as a remote compiler. This
allows running compilers on separate machines and accessing them through the main Compiler Explorer instance.

### Orphaned Property Detection

The system has built-in detection for "orphaned" properties - configurations for compilers or groups that are not
referenced anywhere. This helps maintain configuration cleanliness by identifying unused configuration entries.

## Configuration System Implementation

The configuration system is implemented primarily in the following files:

- `lib/properties.ts` - Core implementation of the properties system
- `lib/properties.interfaces.ts` - TypeScript interfaces for the properties system
- `lib/compiler-finder.ts` - Handles compiler discovery and group expansion
- `lib/utils.ts` - Contains property value conversion functions

## Debugging Configuration

If you need to troubleshoot configuration issues, you can run Compiler Explorer with debug output:

```
make EXTRA_ARGS='--prop-debug' dev
```

This will show detailed logs about property resolution, including which properties are being overridden and from which
source.