## Summary
Fixes the duplicate issue detection algorithm to strip `[TAGS]` from
issue titles before calculating similarity. This eliminates massive
false-positive groups caused by shared tag prefixes like `[LIB REQUEST]`
or `[COMPILER REQUEST]`.
## Problem
The previous implementation would create groups of 98+ completely
unrelated issues just because they shared common tag prefixes. For
example:
- `[LIB REQUEST] Add ULib Library`
- `[LIB REQUEST] musl vs glibc`
- `[REQUEST] Float explorer support`
- `[REQUEST] Support logging in`
These would all be grouped together despite being completely different
requests.
## Solution
- Strip `[TAGS]` before calculating text similarity using a compiled
regex pattern
- Compare only the actual content: "Add ULib Library" vs "musl vs glibc"
→ low similarity ✓
## Additional Changes
- Added `ruff` as a project dependency for consistent code quality
- Fixed linting issues (unused imports, updated to `datetime.UTC`)
- Updated tests to reflect new tag-stripping behavior
## Results
Testing on actual CE issues shows dramatic improvement:
- **Before**: 83 groups, with Group 1 containing 98 unrelated issues
(98% false positives)
- **After**: 63 groups, with Group 1 containing 2 legitimate "Forth"
duplicates (actual duplicates)
Most groups are now legitimate duplicates like:
- Three "Problem with [opcode]" bugs
- Two TI ARM compiler requests
- Multiple MSVC version requests
## Test Plan
- [x] All existing tests pass
- [x] Tested on real CE issue data showing 20+ group reduction
- [x] Code passes ruff linting and formatting
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
This PR adds a new Python CLI tool for automating GitHub repository
management tasks.
## Overview
The initial implementation provides duplicate issue detection using text
similarity analysis. This is the first step toward automating repository
triage tasks.
## Features
- **Click-based CLI** with subcommands for future extensibility
- **find-duplicates command** for detecting duplicate issues using text
similarity
- Uses **gh CLI** for GitHub API access (no token management needed)
- Text similarity using `difflib.SequenceMatcher` (ratio-based
algorithm)
- Configurable similarity threshold (default: 0.6)
- Progress bar for long-running comparisons
- Age filtering support (`--min-age` parameter)
- Standard Python src-layout with **uv** for dependency management
- **Comprehensive test suite** with pytest (integrated into CI)
## Project Structure
```
etc/scripts/gh_tool/
├── src/gh_tool/ # Main package
│ ├── cli.py # Click-based CLI interface
│ └── duplicate_finder.py # Core duplicate detection logic
├── tests/ # Test suite
│ └── test_duplicate_finder.py
├── docs/ # Documentation
│ ├── TRIAGE-CRITERIA.md # Triage guidelines from manual review
│ └── PHASE1-FINDINGS.md # Historical analysis of 855 issues
├── pyproject.toml # Package configuration
└── README.md # Usage documentation
```
## Usage
```bash
cd etc/scripts/gh_tool
uv sync
uv run gh_tool find-duplicates /tmp/report.md
```
**Options:**
- `--threshold FLOAT` - Similarity threshold 0-1 (default: 0.6)
- `--state {all,open,closed}` - Issue state to check (default: open)
- `--min-age DAYS` - Only check issues older than N days (default: 0)
- `--limit INTEGER` - Maximum number of issues to fetch (default: 1000)
- `--repo TEXT` - GitHub repository in owner/repo format (default:
compiler-explorer/compiler-explorer)
**Example:**
```bash
# Find high-confidence duplicates in open issues
uv run gh_tool find-duplicates /tmp/report.md --threshold 0.85
# Check all issues older than 30 days
uv run gh_tool find-duplicates /tmp/report.md --state all --min-age 30
```
## Testing
The tool includes comprehensive test coverage:
- Unit tests for similarity calculation
- Integration tests for duplicate detection
- Edge case handling (transitive grouping, age filtering, threshold
sensitivity)
- Report generation validation
**Run tests:**
```bash
cd etc/scripts/gh_tool
uv run pytest -v
```
Tests are integrated into CI and run on every push.
## Documentation
- **`README.md`**: Complete usage guide with examples
- **`docs/TRIAGE-CRITERIA.md`**: Comprehensive triage guidelines
developed during manual review of 22+ issues
- **`docs/PHASE1-FINDINGS.md`**: Historical analysis context from
initial 855 issue review
## CI Integration
The tool is integrated into the GitHub Actions workflow:
- `uv` is installed via `astral-sh/setup-uv@v6`
- Tests run automatically on every push
- Ensures tool remains functional as codebase evolves
## Next Steps
Future enhancements planned for follow-up PRs:
- GitHub Action for automatic duplicate detection on new issues
- Additional automation tools (upstream health checker, label validator,
etc.)
- Automated triage reports
## Changes in this PR
- ✅ Core duplicate detection implementation
- ✅ Comprehensive test suite (192 lines)
- ✅ CI integration
- ✅ Complete documentation
- ✅ Example triage criteria and findings
---------
Co-authored-by: Claude <noreply@anthropic.com>
