ep_poll() computes the timer slack via select_estimate_accuracy() up front,
before checking whether events are already available.
select_estimate_accuracy() reads the clock (ktime_get_ts64()), and the
resulting slack is only consumed by the schedule_hrtimeout_range() call on
the blocking path.
A busy poller such as an L7 proxy event loop calls epoll_wait() at a very
high rate and often finds events already pending, returning via
ep_try_send_events() without ever blocking. In that case the up-front
slack estimation - including its clock read - is pure overhead. read_tsc()
attributable to select_estimate_accuracy() sometimes shows up in perf profiles
of such a workload via the epoll_wait() path.
Move the slack estimation to the point where the thread is actually about
to sleep. The timeout passed to ep_poll() is already an absolute deadline
(ep_timeout_to_timespec()), so deferring the estimate does not change the
wakeup time; taken closer to the sleep it is, if anything, marginally more
accurate. On the common non-blocking path the clock read is skipped
entirely.
Measured on a host running a Meta production workload with the following
bpftrace script:
#!/usr/bin/bpftrace
fentry:__x64_sys_epoll_wait,
fentry:__x64_sys_epoll_pwait { @in[tid] = 1; }
fexit:__x64_sys_epoll_wait,
fexit:__x64_sys_epoll_pwait { delete(@in, tid); }
fentry:select_estimate_accuracy /@in[tid]/ { @sea++; }
fentry:schedule_hrtimeout_range /@in[tid]/ { @shr++; }
interval:s:30 {
printf("sea=%lld shr=%lld wasted=%lld (%d%%)\n",
@sea, @shr, @sea - @shr, (@sea - @shr) * 100 / @sea);
exit();
}
Over a 30s window:
sea=3,587,704 shr=3,003,920 wasted=583,784 (16%)
So ~16% of ep_poll invocations of select_estimate_accuracy have no
consumer.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Link: https://patch.msgid.link/20260707190238.3478608-1-usama.arif@linux.dev
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Noah Orlando <Noah.Orlando@deshaw.com> says:
When a mount namespace is destroyed, put_mnt_ns() disconnects its mounts
from their mount points. A file descriptor still open on the parent of a
mount point can then be used to look under the mount point.
Locked mounts are kept connected to prevent this. However, a mount is
only locked when its tree is copied across a user namespace boundary. A
mount namespace set up by a privileged component has no locked mounts,
so its mounts are disconnected.
Pass UMOUNT_CONNECTED so every mount is kept connected, as locked mounts
already are.
* patches from https://patch.msgid.link/20260706182559.2496448-2-Noah.Orlando@deshaw.com:
selftests/filesystems: add mntns cleanup test
put_mnt_ns(): leave mounts connected
Link: https://patch.msgid.link/20260706182559.2496448-2-Noah.Orlando@deshaw.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
When a mount namespace is destroyed, put_mnt_ns() disconnects its mounts
from their mount points. A file descriptor still open on the parent of a
mount point can then be used to look under the mount point.
Locked mounts are kept connected to prevent this. However, a mount is
only locked when its tree is copied across a user namespace boundary. A
mount namespace set up by a privileged component has no locked mounts,
so its mounts are disconnected.
Pass UMOUNT_CONNECTED so every mount is kept connected, as locked mounts
already are.
Signed-off-by: Noah Orlando <Noah.Orlando@deshaw.com>
Link: https://patch.msgid.link/20260706182559.2496448-2-Noah.Orlando@deshaw.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Marco Crivellari <marco.crivellari@suse.com> says:
fs: Move long delayed work on system_dfl_long_wq
Hello,
Currently the code uses the per-cpu workqueue system_long_wq to schedule
long running works.
Unbound works could benefit from scheduler task placement, to optimize
performance and power consumption. Another good reason to have this unbound,
is the "queue_delayed_work()" function, used to enqueue the work item.
More details on this will follow in the next section.
Recently, a new unbound workqueue specific for long running work has been
added:
c116737e97 ("workqueue: Add system_dfl_long_wq for long unbound works")
~~~ Details about queue_delayed_work ~~~
system_long_wq is a per-cpu workqueue and it is used as a parameter of
queue_delayed_work(). This function schedule an item that it will later
be enqueued (once the timer will fire). __queue_delayed_work() does the job
receiving as "cpu" WORK_CPU_UNBOUND:
if (housekeeping_enabled(HK_TYPE_TIMER)) {
// [....]
} else {
if (likely(cpu == WORK_CPU_UNBOUND))
add_timer_global(timer);
else
add_timer_on(timer, cpu);
}
The timer is global, so can fire everywhere, and the work item will be
enqueued where the timer fired.
Since the workqueue work doesn't rely on per-cpu variables, there is no
obvious reason that justify the use of a per-cpu workqueue. So change the
workqueue with the new system_dfl_long_wq, so that the used workqueue is
now unbound and can benefit from scheduler task placement.
* patches from https://patch.msgid.link/20260706105443.173697-1-marco.crivellari@suse.com:
affs: Move long delayed work on system_dfl_long_wq
hfs: Move long delayed work on system_dfl_long_wq
hfsplus: Move long delayed work on system_dfl_long_wq
fs/jffs2: Move long delayed work on system_dfl_long_wq
ufs: Move long delayed work on system_dfl_long_wq
Link: https://patch.msgid.link/20260706105443.173697-1-marco.crivellari@suse.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Currently the code enqueue work items using {queue|mod}_delayed_work(),
using system_long_wq. This workqueue should be used when long works are
expected and it is a per-cpu workqueue.
The function(s) end up calling __queue_delayed_work(), which set a global
timer that could fire anywhere, enqueuing the work where the timer fired.
Unbound works could benefit from scheduler task placement, to optimize
performance and power consumption. Long work shouldn't stick to a single
CPU.
Recently, a new unbound workqueue specific for long running work has
been added:
c116737e972e ("workqueue: Add system_dfl_long_wq for long unbound works")
Since the workqueue work doesn't rely on per-cpu variables, there is no
obvious reason that justify the use of a per-cpu workqueue. So change
system_long_wq with system_dfl_long_wq so that the work may benefit from
scheduler task placement.
Cc: David Sterba <dsterba@suse.com>
Cc: linux-fsdevel@vger.kernel.org
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
Link: https://patch.msgid.link/20260706105443.173697-6-marco.crivellari@suse.com
Acked-by: David Sterba <dsterba@suse.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Currently the code enqueue work items using {queue|mod}_delayed_work(),
using system_long_wq. This workqueue should be used when long works are
expected and it is a per-cpu workqueue.
The function(s) end up calling __queue_delayed_work(), which set a global
timer that could fire anywhere, enqueuing the work where the timer fired.
Unbound works could benefit from scheduler task placement, to optimize
performance and power consumption. Long work shouldn't stick to a single
CPU.
Recently, a new unbound workqueue specific for long running work has
been added:
c116737e972e ("workqueue: Add system_dfl_long_wq for long unbound works")
Since the workqueue work doesn't rely on per-cpu variables, there is no
obvious reason that justify the use of a per-cpu workqueue. So change
system_long_wq with system_dfl_long_wq so that the work may benefit from
scheduler task placement.
Cc: Viacheslav Dubeyko <slava@dubeyko.com>
Cc: John Paul Adrian Glaubitz
Cc: Yangtao Li <frank.li@vivo.com>
Cc: linux-fsdevel@vger.kernel.org
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
Link: https://patch.msgid.link/20260706105443.173697-5-marco.crivellari@suse.com
Reviewed-by: Viacheslav Dubeyko <slava@dubeyko.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Currently the code enqueue work items using {queue|mod}_delayed_work(),
using system_long_wq. This workqueue should be used when long works are
expected and it is a per-cpu workqueue.
The function(s) end up calling __queue_delayed_work(), which set a global
timer that could fire anywhere, enqueuing the work where the timer fired.
Unbound works could benefit from scheduler task placement, to optimize
performance and power consumption. Long work shouldn't stick to a single
CPU.
Recently, a new unbound workqueue specific for long running work has
been added:
c116737e972e ("workqueue: Add system_dfl_long_wq for long unbound works")
Since the workqueue work doesn't rely on per-cpu variables, there is no
obvious reason that justify the use of a per-cpu workqueue. So change
system_long_wq with system_dfl_long_wq so that the work may benefit from
scheduler task placement.
Cc: Viacheslav Dubeyko <slava@dubeyko.com>
Cc: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Cc: Yangtao Li <frank.li@vivo.com>
Cc: linux-fsdevel@vger.kernel.org
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
Link: https://patch.msgid.link/20260706105443.173697-4-marco.crivellari@suse.com
Reviewed-by: Viacheslav Dubeyko <slava@dubeyko.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Currently the code enqueue work items using {queue|mod}_delayed_work(),
using system_long_wq. This workqueue should be used when long works are
expected and it is a per-cpu workqueue.
The function(s) end up calling __queue_delayed_work(), which set a global
timer that could fire anywhere, enqueuing the work where the timer fired.
Unbound works could benefit from scheduler task placement, to optimize
performance and power consumption. Long work shouldn't stick to a single
CPU.
Recently, a new unbound workqueue specific for long running work has
been added:
c116737e972e ("workqueue: Add system_dfl_long_wq for long unbound works")
Since the workqueue work doesn't rely on per-cpu variables, there is no
obvious reason that justify the use of a per-cpu workqueue. So change
system_long_wq with system_dfl_long_wq so that the work may benefit from
scheduler task placement.
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Richard Weinberger <richard@nod.at>
Cc: linux-mtd@lists.infradead.org
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
Link: https://patch.msgid.link/20260706105443.173697-3-marco.crivellari@suse.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
In setup_namespace(), f_mountinfo is opened with fopen() at line 115
but is never closed. Multiple ksft_exit_fail_msg() calls exit the
program without closing this file descriptor, and the cleanup_namespace()
function registered with atexit() also doesn't close it.
Add fclose(f_mountinfo) in cleanup_namespace() to ensure the file
descriptor is properly closed on both normal and error exit paths,
since cleanup_namespace() is already registered as an atexit handler.
Signed-off-by: Malaya Kumar Rout <malayarout91@gmail.com>
Link: https://patch.msgid.link/20260704120437.99851-1-malayarout91@gmail.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Currently the code enqueue work items using {queue|mod}_delayed_work(),
using system_long_wq. This workqueue should be used when long works are
expected and it is a per-cpu workqueue.
The function(s) end up calling __queue_delayed_work(), which set a global
timer that could fire anywhere, enqueuing the work where the timer fired.
Unbound works could benefit from scheduler task placement, to optimize
performance and power consumption. Long work shouldn't stick to a single
CPU.
Recently, a new unbound workqueue specific for long running work has
been added:
c116737e972e ("workqueue: Add system_dfl_long_wq for long unbound works")
Since the workqueue work doesn't rely on per-cpu variables, there is no
obvious reason that justify the use of a per-cpu workqueue. So change
system_long_wq with system_dfl_long_wq so that the work may benefit from
scheduler task placement.
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Kees Cook <kees@kernel.org>
Cc: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
Link: https://patch.msgid.link/20260706105443.173697-2-marco.crivellari@suse.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
romfs_iget() follows on-disk hard link entries until it reaches a non-hard
link inode:
pos = be32_to_cpu(ri.spec) & ROMFH_MASK;
The target position is image-controlled, and the loop does not detect
cycles. A crafted romfs image can make the root inode a hard link. The hard
link can point back to itself and leave mount(2) spinning in the kernel.
Reject excessive hard link indirection with -ELOOP. Normal romfs images do
not need long hard link chains. This bounds corrupted-image traversal.
Propagate romfs_iget() errors from lookup because hard link traversal can
now fail with -ELOOP.
Signed-off-by: 이상호 <kudo3228@gmail.com>
Link: https://patch.msgid.link/20260701220729.822112-1-kudo3228@gmail.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Jori Koolstra <jkoolstra@xs4all.nl> says:
vfs: pass S_IFDIR mode to vfs_prepare_mode()
There is a comment in vfs_prepare_mode() that says:
Note that it's currently valid for @type to be 0 if a directory is
created. Filesystems raise that flag individually and we need to check
whether each filesystem can deal with receiving S_IFDIR from the vfs
before we enforce a non-zero type.
It is useful to do this clean-up ahead of O_CREAT|O_DIRECTORY.
Specifically, in lookup_open() we need to replace the vfs_prepare_mode()
with something that also handles dirs. I don't really want to push the
odd
mode = vfs_prepare_mode(idmap, dir, mode, S_IRWXUGO | S_ISVTX, 0);
further into that code, and neither do I want this to be different from
the regular vfs_mkdir() path. We can then also match on S_IFMT in
may_o_create(), instead of passing a bool to signal whether we are
creating a dir (and assuming 0 means a dir is really ugly).
It is a bit challenging to verify that passing S_IFDIR is safe, as there
are many filesystems. Claude Opus 4.8 was used to generate the context
for each mkdir implementation from which it can be judged whether
passing S_IFDIR is OK. The result was then verified by hand by looking
at how the mode argument is used in each case. To check whether all
mkdir implementations are covered, 'rg "\.mkdir" ' was used and checked
against the list of uses Claude found.
It is safe to do this clean-up except that three filesystems (fuse,
cifs, and coda) forward the mkdir @mode unchanged to something outside
the kernel. Mask S_IFDIR back out in coda_mkdir(), fuse_mkdir() and
cifs_mkdir() so that what is sent outside the kernel is unchanged.
Their maintainers can drop the mask once they have confirmed it is safe.
For the other filesystems redundant S_IFDIR OR'ing is dropped.
* patches from https://patch.msgid.link/20260630105400.68459-1-jkoolstra@xs4all.nl: (31 commits)
ntfs: drop redundant S_IFDIR from mkdir
xfs: drop redundant S_IFDIR from mkdir
ubifs: drop redundant S_IFDIR from mkdir
nfs: drop redundant S_IFDIR from mkdir
ufs: drop redundant S_IFDIR from mkdir
udf: drop redundant S_IFDIR from mkdir
ramfs: drop redundant S_IFDIR from mkdir
orangefs: drop redundant S_IFDIR from mkdir
omfs: drop redundant S_IFDIR from mkdir
ocfs2: dlmfs: drop redundant S_IFDIR from mkdir
ocfs2: drop redundant S_IFDIR from mkdir
ntfs3: drop redundant S_IFDIR from mkdir
nilfs2: drop redundant S_IFDIR from mkdir
minix: drop redundant S_IFDIR from mkdir
jfs: drop redundant S_IFDIR from mkdir
jffs2: drop redundant S_IFDIR from mkdir
hugetlbfs: drop redundant S_IFDIR from mkdir
hpfs: drop redundant S_IFDIR from mkdir
hfsplus: drop redundant S_IFDIR from mkdir
hfs: drop redundant S_IFDIR from mkdir
...
Link: https://patch.msgid.link/20260630105400.68459-1-jkoolstra@xs4all.nl
Suggested-by: Christian Brauner (Amutable) <brauner@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
There is a comment in vfs_prepare_mode() that says:
Note that it's currently valid for @type to be 0 if a directory is
created. Filesystems raise that flag individually and we need to check
whether each filesystem can deal with receiving S_IFDIR from the vfs
before we enforce a non-zero type.
It is safe to do this clean-up except that three filesystems (fuse,
cifs, and coda) forward the mkdir @mode unchanged to something outside
the kernel. Mask S_IFDIR back out in coda_mkdir(), fuse_mkdir() and
cifs_mkdir() so that what is sent outside the kernel is unchanged.
Their maintainers can drop the mask once they have confirmed it is safe.
Assisted-by: LLM
Signed-off-by: Jori Koolstra <jkoolstra@xs4all.nl>
Link: https://patch.msgid.link/20260630105400.68459-2-jkoolstra@xs4all.nl
Reviewed-by: NeilBrown <neil@brown.name>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
The super_block shrinker is registered with SHRINKER_MEMCG_AWARE because its
dentry and inode LRUs are memcg-aware (via list_lru). But the optional
->nr_cached_objects() hooks that the shrinker also drives are not memcg-aware:
btrfs extent maps and xfs inode reclaim operate on filesystem-global
state, and shmem's unused-huge shrinker walks a per-superblock shrinklist.
None of them filter by sc->memcg.
The mismatch shows up under memcg-heavy slab reclaim. shrink_slab_memcg()
calls do_shrink_slab() once per (memcg, NUMA node) pair for every memcg
whose bit is set in the per-superblock shrinker bitmap, which on a busy
host means hundreds of calls per reclaim pass. Each scan queues the same
global shrinker work item that's already kicked from the root path.
Because btrfs/xfs global count is typically non-zero on any in-use filesystem,
the returned total stays positive even if a memcg's own dentry/inode LRUs
are empty. shrink_slab_memcg() therefore never clears the SB shrinker bit
in the memcg bitmap, so subsequent reclaim passes from the same memcg
re-enter super_cache_count() and pay for the global counter walk again.
Restrict ->nr_cached_objects() to the global shrink path (sc->memcg NULL
or root). The memcg-aware dentry/inode LRUs keep being counted and
scanned per memcg as before; only the global fs-specific hooks are skipped.
The root/global shrink path still drives those hooks; only their
invocation from non-root memcg slab reclaim is removed.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Link: https://patch.msgid.link/20260609123047.1948242-1-usama.arif@linux.dev
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
proc-pidns.c uses open() and O_* flags, but does not include
<fcntl.h>. This breaks the proc selftests build with errors such as:
error: implicit declaration of function 'open'
error: 'O_WRONLY' undeclared
error: 'O_CREAT' undeclared
error: 'O_RDONLY' undeclared
Include <fcntl.h> to provide the declaration and flag definitions.
Fixes: 5554d820f7 ("selftests/proc: add tests for new pidns APIs")
Tested with:
make -C tools/testing/selftests TARGETS=proc
Signed-off-by: Amin Vakil <info@aminvakil.com>
Link: https://patch.msgid.link/20260618151444.124739-1-info@aminvakil.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
