mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-08-30 14:33:24 -04:00
f94563fac2
When `perf record` samples a multi-threaded process and one of the target threads exits during the session, perf itself may start burning 100% CPU (up to 200% across two cores) until the session ends. A single dead fd is sufficient to trigger this; it can be reproduced with 15 pthreads in a compute loop where one thread exits halfway through. The root cause is two independent instances of the same defect: dead perf_event ring-buffer fds are left in a pollfd array. When a monitored thread exits, the kernel closes its ring-buffer fd, which then returns POLLHUP. POSIX specifies that poll() always reports POLLHUP and POLLERR regardless of the events mask, so any dead fd left in the array makes poll() return immediately every time, spinning in a tight loop: 3 seconds: 256,600 poll() calls, 0 context switches, only 21 write() Woken up count goes from ~0 to 1,300,000+ There are two affected poll paths, fixed together here: 1. Record main loop, via fdarray__filter() (tools/lib/api/fd/array.c). Since commit59b4412f27("libperf: Avoid internal moving of fdarray fds") it only zeroes events/revents without setting fd to -1, so poll() keeps reporting POLLHUP for the entry. Setting fd = -1 makes poll() skip it, matching the pattern already used in the control-fd path at tools/perf/builtin-record.c:1673. 2. BPF sideband thread, perf_evlist__poll_thread() (tools/perf/util/sideband_evlist.c). This thread polls for PERF_RECORD_BPF_EVENT but, unlike the main record loop, never calls fdarray__filter() at all, so dead fds accumulate forever and it spins at 100% CPU: Before fix: dJiffies=101, wchan=0 (running) After fix: dJiffies=0, wchan=do_sys_poll (blocking) Fixed by calling the existing evlist__filter_pollfd() helper after evlist__poll(), mirroring the main record loop. <poll.h> is included for the POLLERR/POLLHUP macros (previously unused there). The two fixes compose: fix 1 makes poll() ignore dead fds (fd=-1); fix 2 ensures the sideband thread actually performs the filtering. Both paths are affected in all kernels from v5.1/v5.9 to the current master (7.2-rc1); the source of both functions is byte-identical across them. BPF event recording is preserved: after the fix, perf.data still contains PERF_RECORD_BPF_EVENT records and bpf_prog_info entries. Verified on perf 6.1.76, 6.6.143 and 7.2-rc1 with a minimal reproducer (Woken up 1,300,000 -> 3, CPU 100% -> 0%) and an A/B orthogonal test: keeping the unpatched binary but preventing the target thread from exiting also makes the storm disappear, confirming the trigger. Fixes:59b4412f27("libperf: Avoid internal moving of fdarray fds") Fixes:657ee55319("perf evlist: Introduce side band thread") Signed-off-by: Jiawei Sun <abyssmystery@gmail.com> Reviewed-by: Ian Rogers <irogers@google.com> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Linux kernel ============ The Linux kernel is the core of any Linux operating system. It manages hardware, system resources, and provides the fundamental services for all other software. Quick Start ----------- * Report a bug: See Documentation/admin-guide/reporting-issues.rst * Get the latest kernel: https://kernel.org * Build the kernel: See Documentation/admin-guide/quickly-build-trimmed-linux.rst * Join the community: https://lore.kernel.org/ Essential Documentation ----------------------- All users should be familiar with: * Building requirements: Documentation/process/changes.rst * Code of Conduct: Documentation/process/code-of-conduct.rst * License: See COPYING Documentation can be built with make htmldocs or viewed online at: https://www.kernel.org/doc/html/latest/ Who Are You? ============ Find your role below: * New Kernel Developer - Getting started with kernel development * Academic Researcher - Studying kernel internals and architecture * Security Expert - Hardening and vulnerability analysis * Backport/Maintenance Engineer - Maintaining stable kernels * System Administrator - Configuring and troubleshooting * Maintainer - Leading subsystems and reviewing patches * Hardware Vendor - Writing drivers for new hardware * Distribution Maintainer - Packaging kernels for distros * AI Coding Assistant - LLMs and AI-powered development tools For Specific Users ================== New Kernel Developer -------------------- Welcome! Start your kernel development journey here: * Getting Started: Documentation/process/development-process.rst * Your First Patch: Documentation/process/submitting-patches.rst * Coding Style: Documentation/process/coding-style.rst * Build System: Documentation/kbuild/index.rst * Development Tools: Documentation/dev-tools/index.rst * Kernel Hacking Guide: Documentation/kernel-hacking/hacking.rst * Core APIs: Documentation/core-api/index.rst Academic Researcher ------------------- Explore the kernel's architecture and internals: * Researcher Guidelines: Documentation/process/researcher-guidelines.rst * Memory Management: Documentation/mm/index.rst * Scheduler: Documentation/scheduler/index.rst * Networking Stack: Documentation/networking/index.rst * Filesystems: Documentation/filesystems/index.rst * RCU (Read-Copy Update): Documentation/RCU/index.rst * Locking Primitives: Documentation/locking/index.rst * Power Management: Documentation/power/index.rst Security Expert --------------- Security documentation and hardening guides: * Security Documentation: Documentation/security/index.rst * LSM Development: Documentation/security/lsm-development.rst * Self Protection: Documentation/security/self-protection.rst * Reporting Vulnerabilities: Documentation/process/security-bugs.rst * CVE Procedures: Documentation/process/cve.rst * Embargoed Hardware Issues: Documentation/process/embargoed-hardware-issues.rst * Security Features: Documentation/userspace-api/seccomp_filter.rst Backport/Maintenance Engineer ----------------------------- Maintain and stabilize kernel versions: * Stable Kernel Rules: Documentation/process/stable-kernel-rules.rst * Backporting Guide: Documentation/process/backporting.rst * Applying Patches: Documentation/process/applying-patches.rst * Subsystem Profile: Documentation/maintainer/maintainer-entry-profile.rst * Git for Maintainers: Documentation/maintainer/configure-git.rst System Administrator -------------------- Configure, tune, and troubleshoot Linux systems: * Admin Guide: Documentation/admin-guide/index.rst * Kernel Parameters: Documentation/admin-guide/kernel-parameters.rst * Sysctl Tuning: Documentation/admin-guide/sysctl/index.rst * Tracing/Debugging: Documentation/trace/index.rst * Performance Security: Documentation/admin-guide/perf-security.rst * Hardware Monitoring: Documentation/hwmon/index.rst Maintainer ---------- Lead kernel subsystems and manage contributions: * Maintainer Handbook: Documentation/maintainer/index.rst * Pull Requests: Documentation/maintainer/pull-requests.rst * Managing Patches: Documentation/maintainer/modifying-patches.rst * Rebasing and Merging: Documentation/maintainer/rebasing-and-merging.rst * Development Process: Documentation/process/maintainer-handbooks.rst * Maintainer Entry Profile: Documentation/maintainer/maintainer-entry-profile.rst * Git Configuration: Documentation/maintainer/configure-git.rst Hardware Vendor --------------- Write drivers and support new hardware: * Driver API Guide: Documentation/driver-api/index.rst * Driver Model: Documentation/driver-api/driver-model/driver.rst * Device Drivers: Documentation/driver-api/infrastructure.rst * Bus Types: Documentation/driver-api/driver-model/bus.rst * Device Tree Bindings: Documentation/devicetree/bindings/ * Power Management: Documentation/driver-api/pm/index.rst * DMA API: Documentation/core-api/dma-api.rst Distribution Maintainer ----------------------- Package and distribute the kernel: * Stable Kernel Rules: Documentation/process/stable-kernel-rules.rst * ABI Documentation: Documentation/ABI/README * Kernel Configuration: Documentation/kbuild/kconfig.rst * Module Signing: Documentation/admin-guide/module-signing.rst * Kernel Parameters: Documentation/admin-guide/kernel-parameters.rst * Tainted Kernels: Documentation/admin-guide/tainted-kernels.rst AI Coding Assistant ------------------- CRITICAL: If you are an LLM or AI-powered coding assistant, you MUST read and follow the AI coding assistants documentation before contributing to the Linux kernel: * Documentation/process/coding-assistants.rst This documentation contains essential requirements about licensing, attribution, and the Developer Certificate of Origin that all AI tools must comply with. Communication and Support ========================= * Mailing Lists: https://lore.kernel.org/ * IRC: #kernelnewbies on irc.oftc.net * Bugzilla: https://bugzilla.kernel.org/ * MAINTAINERS file: Lists subsystem maintainers and mailing lists * Email Clients: Documentation/process/email-clients.rst