mirrors/linux
2
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-08-27 23:25:13 -04:00
Code Issues Packages Projects Releases Wiki Activity

eventpoll: compute timer slack lazily in ep_poll()

Browse Source 
ep_poll() computes the timer slack via select_estimate_accuracy() up front,
before checking whether events are already available.
select_estimate_accuracy() reads the clock (ktime_get_ts64()), and the
resulting slack is only consumed by the schedule_hrtimeout_range() call on
the blocking path.

A busy poller such as an L7 proxy event loop calls epoll_wait() at a very
high rate and often finds events already pending, returning via
ep_try_send_events() without ever blocking.  In that case the up-front
slack estimation - including its clock read - is pure overhead.  read_tsc()
attributable to select_estimate_accuracy() sometimes shows up in perf profiles
of such a workload via the epoll_wait() path.

Move the slack estimation to the point where the thread is actually about
to sleep.  The timeout passed to ep_poll() is already an absolute deadline
(ep_timeout_to_timespec()), so deferring the estimate does not change the
wakeup time; taken closer to the sleep it is, if anything, marginally more
accurate.  On the common non-blocking path the clock read is skipped
entirely.

Measured on a host running a Meta production workload with the following
bpftrace script:

  #!/usr/bin/bpftrace
  fentry:__x64_sys_epoll_wait,
  fentry:__x64_sys_epoll_pwait   { @in[tid] = 1; }
  fexit:__x64_sys_epoll_wait,
  fexit:__x64_sys_epoll_pwait    { delete(@in, tid); }
  fentry:select_estimate_accuracy /@in[tid]/ { @sea++; }
  fentry:schedule_hrtimeout_range /@in[tid]/ { @shr++; }
  interval:s:30 {
      printf("sea=%lld shr=%lld wasted=%lld (%d%%)\n",
             @sea, @shr, @sea - @shr, (@sea - @shr) * 100 / @sea);
      exit();
  }

Over a 30s window:

sea=3,587,704 shr=3,003,920 wasted=583,784 (16%)

So ~16% of ep_poll invocations of select_estimate_accuracy have no
consumer.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
Link: https://patch.msgid.link/20260707190238.3478608-1-usama.arif@linux.dev
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
This commit is contained in:
Usama Arif
2026-07-07 12:02:38 -07:00
committed by Christian Brauner
parent 0c97d2a165
commit f797d7b64e
1 changed files with 4 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
fs/eventpoll.c
View File

@@ -2248,7 +2248,6 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
lockdep_assert_irqs_enabled();
if (timeout && (timeout->tv_sec | timeout->tv_nsec)) {
slack = select_estimate_accuracy(timeout);
to = &expires;
*to = timespec64_to_ktime(*timeout);
} else if (timeout) {
@@ -2327,10 +2326,13 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
spin_unlock_irq(&ep->lock);
if (!eavail)
if (!eavail) {
if (to)
slack = select_estimate_accuracy(timeout);
timed_out = !ep_schedule_timeout(to) ||
!schedule_hrtimeout_range(to, slack,
HRTIMER_MODE_ABS);
}
__set_current_state(TASK_RUNNING);
/*