The nvme_tcp_ctrl_list and nvme_tcp_ctrl::list are protected by
nvme_tcp_ctrl_mutex. Define nvme_tcp_ctrl_list using
LIST_HEAD_GUARDED(nvme_tcp_ctrl_list, nvme_tcp_ctrl_mutex) and
annotate nvme_tcp_ctrl::list using
__guarded_by(&nvme_tcp_ctrl_mutex) so that Clang's context analysis
can validate accesses against the corresponding locking requirements.
It is safe to initialize nvme_tcp_ctrl::list while allocating the
controller object because the list entry has not yet been added to
nvme_tcp_ctrl_list. Annotate the initialization with context_unsafe()
to suppress the corresponding Clang warning.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
After adding Clang lock context annotations in rdma.c, Clang reports
the following warning when context analysis is enabled:
drivers/nvme/host/rdma.c:972:24: warning: passing pointer to variable 'list' requires holding mutex 'nvme_rdma_ctrl_mutex'
[-Wthread-safety-pointer]
972 | if (list_empty(&ctrl->list))
| ^
The warning is triggered because ctrl->list is annotated as being
protected by nvme_rdma_ctrl_mutex, but list_empty(&ctrl->list) is
invoked without holding that mutex.
Replace list_empty() with list_empty_careful(), which is intended
for lockless inspection of list heads during teardown when no concurrent
list modifications are expected. This suppresses the corresponding
context analysis warning while preserving the existing behavior.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
device_list and nvme_rdma_device::entry are protected by
device_list_mutex. Define device_list using
LIST_HEAD_GUARDED(device_list, device_list_mutex) and annotate
nvme_rdma_device::entry with __guarded_by(&device_list_mutex) so that
Clang's context analysis can validate accesses against the corresponding
locking requirements.
Similarly, nvme_rdma_ctrl_list and nvme_rdma_ctrl::list are
protected by nvme_rdma_ctrl_mutex. Define nvme_rdma_ctrl_list using
LIST_HEAD_GUARDED(nvme_rdma_ctrl_list, nvme_rdma_ctrl_mutex) and
annotate nvme_rdma_ctrl::list with __guarded_by(&nvme_rdma_ctrl_mutex).
It is safe to initialize nvme_rdma_ctrl::list while allocating the
controller object because the list entry has not yet been added to
nvme_rdma_ctrl_list. Annotate the initialization with context_unsafe()
to suppress the corresponding Clang context analysis warning.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvme_queue::sq_tail, nvme_queue::last_sq_tail and nvme_queue::sq_cmds
are protected by nvme_queue::sq_lock. Annotate each field with
__guarded_by(&sq_lock) and annotate helpers that access them with
__must_hold(&sq_lock) so that Clang's context analysis can validate
the locking requirements.
Access to nvme_queue::sq_tail used solely for tracing is annotated with
data_race(), as they only require a lockless snapshot of the value.
nvme_init_queue() initializes nvme_queue::sq_tail and
nvme_queue::last_sq_tail before the queue is published and thus do not
require nvme_queue::sq_lock protection. So annotate nvme_init_queue()
with context_unsafe() to suppress false positive context analyzer
warning.
nvme_free_queue() operate on queues which are no longer reachable, and
therefore do not require nvme_queue::sq_lock protection. Similarly,
nvme_alloc_sq_cmds() allocates memory for nvme_queue::sq_cmds for the
queue which is not yet published or in use and hence it's safe to
annotate all these helpers using context_unsafe.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
The global nvmf_transports list is protected by nvmf_transports_rwsem
and the global nvmf_hosts list is protected by nvmf_hosts_mutex.
Define both lists using LIST_HEAD_GUARDED() so that Clang's context
analysis can validate accesses to the lists against the corresponding
locking requirements.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
The global nvme_subsystems list, nvme_subsystem::entry,
nvme_subsystem::ctrls, and nvme_ctrl::subsys_entry are protected by
nvme_subsystems_lock. Annotate these objects with
__guarded_by(&nvme_subsystems_lock) so that Clang's context analysis
can validate accesses to them.
__nvme_find_get_subsystem() and nvme_validate_cntlid() traverse the
global subsystem list and subsystem controller list and therefore
require callers to hold nvme_subsystems_lock. Annotate both helpers
with __must_hold(&nvme_subsystems_lock).
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvme_parse_ana_log() accesses ANA state protected by ctrl->ana_lock and
therefore requires callers to hold the lock.
Annotate nvme_parse_ana_log() with __must_hold(&ctrl->ana_lock) so that
Clang's lock context analysis can verify the locking requirement at
compile time.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Several helpers access or traverse data structures protected by
nvme_subsystem::lock and therefore require callers to hold the lock.
Annotate nvme_mpath_unfreeze(), nvme_mpath_wait_freeze(),
nvme_mpath_start_freeze(), nvme_find_ns_head(), nvme_alloc_ns_head()
and nvme_subsys_check_duplicate_ids() with __must_hold(&subsys->lock)
so that Clang's lock context analysis can validate the locking
requirements at compile time.
Also annotate nvme_subsystem::nsheads and
nvme_ns_head::delayed_removal_secs with __guarded_by(&subsys->lock),
as both are protected by the subsystem lock.
Annotate nvme_init_subsystem() with __context_unsafe(), as it
initializes these lock-protected members before the object is published,
suppressing a false positive from Clang's context analysis.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvme_ns_head is allocated with kzalloc(), so explicitly initializing
nvme_ns_head::delayed_removal_secs to 0 in nvme_mpath_alloc_disk() is
redundant.
Removing the redundant initialization also avoids a false positive from
Clang's context analysis once nvme_ns_head::delayed_removal_secs is
annotated with __guarded_by(nvme_subsystem::lock).
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvme_setup_io_queues_trylock() conditionally acquires dev->shutdown_lock
using mutex_trylock(). The function returns 0 when the lock is
successfully acquired and a negative error code otherwise.
Annotate the function with __cond_acquires(0, &dev->shutdown_lock) so
that Clang's lock context analysis can track the lock state based on
the return value and verify correct lock usage at call sites.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Annotate nvme_ns_head::current_path[] with __rcu_guarded so that
Clang's context analysis can validate accesses to the SRCU/RCU
protected pointer.
Cc: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvme_ns_head::requeue_list is protected by
nvme_ns_head::requeue_lock. Annotate requeue_list with
__guarded_by(&requeue_lock) so that Clang's context analysis can
validate accesses to the list.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
bio_list_init() is a no-op for zero-initialized objects. Remove the
redundant initialization of nvme_ns_head::requeue_list from
nvme_mpath_alloc_disk().
Besides simplifying the code, this also avoids a false positive from
Clang's context analysis once nvme_ns_head::requeue_list is annotated
with __guarded_by(&requeue_lock).
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add Clang lock context annotations for helpers that operate under
head->srcu read-side protection.
The path selection helpers invoked by nvme_find_path() access SRCU-
protected data through srcu_dereference() or list APIs which iterate
through rcu protected list and therefore require the caller to hold
head->srcu. Annotate these helpers and nvme_find_path() with
__must_hold_shared(&head->srcu) so that Clang's lock context analysis
can verify the SRCU locking requirements across the call chain.
Also update nvme_ns_head_ctrl_ioctl() to use __releases_shared()
to match the shared SRCU read-side lock acquired through
srcu_read_lock().
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Annotate nvme_passthru_start() and nvme_passthru_end() for Clang
context/thread-safety analysis.
The __cond_acquires() and __cond_releases() annotations model
conditional lock acquisition and release based on a function's return
value. Use a nonzero return value as the abstract condition denoting
that the associated locks have been acquired or released.
This allows the analyzer to track the lock state across the
nvme_passthru_start() / nvme_passthru_end() pair and verify correct
locking semantics.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Change nvme_passthru_end() to return the command effects value passed
to it.
This is a preparatory change for Clang's context/thread-safety analysis
support. The conditional release annotations (__cond_releases()) model
lock release based on a function's return value. Returning the existing
effects value allows a subsequent patch to annotate nvme_passthru_end()
as conditionally releasing locks acquired by nvme_passthru_start().
No functional change intended.
A follow-up patch will add the corresponding context analysis
annotations.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
With Context Analysis (viz. Clang's Thread Safety Analysis), list_heads
that are __guarded_by(..) require holding the appropriate context lock
when accessing and manipulating them via the list API. Because Clang's
warning diagnostics do not perform inter-procedural analysis, this is
enforced by Clang with -Wthread-safety-pointer in the caller at the call
boundary; a warning is produced when passing a pointer to a guarded
variable without holding the appropriate context locks:
warning: passing pointer to variable 'list' requires holding [...] [-Wthread-safety-pointer]
if (list_empty(&ctrl->list))
An exception is list_empty_careful(), which is like list_empty(), except
that it is permitted to use without holding any context lock (carefully).
Mark list_empty_careful() __context_unsafe, which disables context
analysis within list_empty_careful(), but also suppresses warnings
generated in callers related to its pointer arguments.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Marco Elver <elver@google.com>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Introduce LIST_HEAD_GUARDED(name, lock) to define a struct list_head
annotated with __guarded_by(lock). This provides a convenient shorthand
for defining lock-protected list heads and allows compiler context
analysis to validate accesses to the list against the associated lock.
The new helper also reduces boilerplate and improves consistency across
callers that annotate struct list_head objects with __guarded_by().
This is a preparatory change for subsequent patches that annotate
LIST_HEAD() instances with their protecting lock.
Suggested-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
When fuzzing the nvme target code, I tripped a kernel warning in
nvmet_tcp_map_data() because the length passed into the allocator is
controlled by the remote initiator.
A remote initiator that sends a command with an SGL claiming a huge
number, can create a scatterlist and iovec allocation of over 1 million
entries, which causes the backing kmalloc call to exceed MAX_PAGE_ORDER
and then the page allocator will trip on a WARN_ON_ONCE_GFP() message:
WARNING: mm/page_alloc.c:5280 __alloc_frozen_pages_noprof
Workqueue: nvmet_tcp_wq nvmet_tcp_io_work
...
sgl_alloc_order
nvmet_tcp_map_data
nvmet_tcp_try_recv_pdu
As it's never good to trip a kernel warning remotely due to many systems
having panic-on-warn enabled, let's silence it by just add GFP_NOWARN to
the allocation flags.
Assisted-by: gkh_clanker_2000
Cc: stable <stable@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add a wrapper for getting a reference to the NS head.
This would be used in scenarios when we know that getting a reference
would not fail.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: John Garry <john.g.garry@oracle.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvme_query_fdp_info() allocates the RUH status buffer for at most S8_MAX
- 1 descriptors, and then copies ruhs->ruhsd[] into head->plids[] using
the controller reported ruhs->nruhsd directly as the loop bound.
However, that count wasn't taken into account for the actual buffer's
size, so there was a chance for a controller reporting a larger nruhsd
to cause the copy to overflow the buffer. Clamp nr_plids to the same
bound used for the allocation.
Assisted-by: gkh_clanker_t1000
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hari Mishal <harimishal1@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvme_identify_ns_descs() allocates a buffer and gives it to the
controller, which populates it and then iterates the buffer with
variable byte increments that vary by type and body size. But, there is
no bounds check inside the iteration itself except the loop bound
itself. Fix this by checking and stopping iteration if the next header
or its declared body would go past the buffer itself.
Assisted-by: gkh_clanker_t1000
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hari Mishal <harimishal1@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add documentation file entries that were missing from the NVM EXPRESS
DRIVER and NVM EXPRESS TARGET DRIVER sections, so patches touching
these files are properly routed to the NVMe mailing list and
maintainers.
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add Documentation/ABI/stable/configfs-nvmet documenting all NVMe
target configfs attributes, covering port attributes, subsystem
attributes, namespace attributes, host authentication, passthrough
mode, and ANA configuration.
Each entry has been traced to its original introducing commit to
provide accurate Date, KernelVersion, and Contact information.
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add Documentation/ABI/stable/sysfs-nvme documenting all NVMe host
sysfs attributes, covering controller attributes under
/sys/class/nvme/nvmeX/, namespace attributes under
/sys/block/nvmeXnY/, and subsystem attributes under
/sys/class/nvme-subsystem/nvme-subsysX/.
Each entry has been traced to its original introducing commit to
provide accurate Date, KernelVersion, and Contact information.
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add a 'reservation' debugfs file under each namespace directory that
shows the persistent reservation state, including enable status,
generation counter, notify mask, current holder info, and the full
registrant list with hostid and reservation key.
Each attribute is emitted as a single "key=value" line so the output is
easy to parse from scripts. The registrant list is emitted as repeated
"reg=" lines. The notify mask is emitted as a comma-separated list of
masked notification names. Empty values are reported as "none". Example
output:
enable=1
generation=2
notify_mask=reg_preempted,resv_released,resv_preempted
rtype=write_exclusive
holder=11111111-1111-1111-1111-111111111111,0x1111
reg=11111111-1111-1111-1111-111111111111,0x1111
reg=22222222-2222-2222-2222-222222222222,0x2222
When reservation is not enabled only "enable=0" is printed.
The output uses rcu_read_lock() for safe access to the holder and
registrant_list, consistent with other PR read paths.
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add per-namespace debugfs directory support under the subsystem debugfs
directory. Each enabled namespace gets a ns<nsid>/ directory created
during nvmet_ns_enable() and removed during nvmet_ns_disable().
This provides the infrastructure for exposing namespace-specific debug
information in subsequent patches.
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
In nvme_ns_head_ctrl_ioctl(), once we drop the SRCU read lock we should
not reference the NS to lookup the controller, so use the available
controller pointer directly.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: John Garry <john.g.garry@oracle.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
sashiko bot reported a potential issue in the requeue handling in [0] -
the code there is same as the NVMe driver.
The issue is that when we schedule the requeue work, if a bio is added to
the requeue list afterwards in nvme_ns_head_submit_bio(), it is missed by
the requeue worker.
This issue can be recreated by hacking a large delay in the bio submission
requeue path:
} else if (nvme_available_path(head)) {
dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");
+ msleep(30000);
spin_lock_irq(&head->requeue_lock);
bio_list_add(&head->requeue_list, bio);
spin_unlock_irq(&head->requeue_lock);
Then if we issue a write after removing all paths, a hang can be seen:
# echo 20 > /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1/delayed_removal_secs
#
# ./ini_nvme_teardown.sh
[ 25.877224] nvme nvme1: Removing ctrl: NQN "nvme-test-target"
[ 25.939569] nvme nvme2: Removing ctrl: NQN "nvme-test-target"
#
# xfs_io -d -C "pwrite -b 64k -V 1 -D 0 64k" /dev/nvme1n1p1
[ 29.883653] block nvme1n1: no usable path - requeuing I/O
Fix by re-ordering the SRCU synchronization and scheduling the requeue
work.
[0] https://lore.kernel.org/linux-scsi/20260703102918.3723667-1-john.g.garry@oracle.com/T/#m72af1f29deb0ebfb2973464207f201f1be1f660c
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: John Garry <john.g.garry@oracle.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvmet_execute_auth_receive() allocates the response buffer with kmalloc()
sized by the host-supplied AUTH_RECEIVE allocation length, but the
DH-HMAC-CHAP builders write only a fixed-size message into it. The full
allocation length is then copied to the wire by nvmet_copy_to_sgl(), so a
remote initiator receives the bytes past the built message -- up to nearly
a page of uninitialized slab -- during the pre-authentication handshake.
Allocate the buffer with kzalloc() so the unwritten tail is zeroed before
it is sent; conforming responses are unaffected.
Fixes: db1312dd95 ("nvmet: implement basic In-Band Authentication")
Cc: stable@vger.kernel.org
Signed-off-by: Bryam Vargas <hexlabsecurity@proton.me>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
DH-HMAC-CHAP authentication compares HMAC response digests with memcmp().
Standard memcmp() may stop at the first differing byte, which can leak
timing information to a remote attacker and allow incremental recovery
of the expected digest.
Use crypto_memneq() for constant-time comparison on both the host path
that validates the controller Success1 response and the target path that
validates the host Reply digest. Other memcmp() uses in the NVMe auth
code (e.g. fixed string prefix checks) are not security-sensitive and
are left unchanged.
Signed-off-by: Xixin Liu <liuxixin@kylinos.cn>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Function __nvme_submit_sync_cmd() returns a positive error code for NVMe
errors. Otherwise, we get 0 for success or a negative error code for a
kernel error.
In nuse_show() -> ns_{head}_update_nuse() -> nvme_identify_ns() ->
nvme_submit_sync_cmd() -> __nvme_submit_sync_cmd(), we then may get a
positive error code returned.
Function nuse_show() - being a device attr handler - should return the
number of bytes written to the buffer or a negative error code.
Convert any positive NVMe error code to -EIO.
Signed-off-by: John Garry <john.g.garry@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
When an nvme target with rdma transport is removed while I/Os are in
flight, a response can be posted but its send completion is never
delivered before the connection is torn down. As a result
nvmet_rdma_send_done() and nvmet_rdma_release_rsp() are never called for
the response, and this leaks the allocated RDMA read/write context and
request SGLs.
These leaks are recreated by running blktests nvme/061 with the rdma
transport and the siw driver. Kernel kmemleak feature reports them as
follows:
unreferenced object 0xffff88812bc490c0 (size 32):
comm "kworker/2:1H", pid 409, jiffies 4307744490
backtrace (crc 89afd339):
__kmalloc_noprof+0x5f9/0x890
sgl_alloc_order+0x7b/0x380
nvmet_req_alloc_sgls+0x290/0x4f0 [nvmet]
nvmet_rdma_map_sgl_keyed+0x241/0x12e0 [nvmet_rdma]
nvmet_rdma_handle_command+0x73e/0xb80 [nvmet_rdma]
__ib_process_cq+0x149/0x4c0 [ib_core]
ib_cq_poll_work+0x49/0x160 [ib_core]
process_one_work+0x8b2/0x1640
worker_thread+0x5fd/0xfe0
kthread+0x367/0x460
ret_from_fork+0x655/0x9d0
ret_from_fork_asm+0x1a/0x30
unreferenced object 0xffff88814bd05e80 (size 64):
comm "kworker/3:1H", pid 148, jiffies 4295195428
backtrace (crc e35510cb):
__kmalloc_noprof+0x5f9/0x890
rdma_rw_ctx_init+0x333/0x1fa0 [ib_core]
nvmet_rdma_map_sgl_keyed+0x5c8/0x12e0 [nvmet_rdma]
nvmet_rdma_handle_command+0x73e/0xb80 [nvmet_rdma]
__ib_process_cq+0x149/0x4c0 [ib_core]
ib_cq_poll_work+0x49/0x160 [ib_core]
process_one_work+0x8b2/0x1640
worker_thread+0x5fd/0xfe0
kthread+0x367/0x460
ret_from_fork+0x655/0x9d0
ret_from_fork_asm+0x1a/0x30
To avoid the memory leaks, reclaim the memory of the in-flight responses
when the queue QP is torn down. Call nvmet_rdma_free_rsp_resources()
that frees up the RDMA read/write context and the request SGLs of such
responses.
Fixes: 8f000cac6e ("nvmet-rdma: add a NVMe over Fabrics RDMA target driver")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Move the RDMA read/write context teardown and the request SGL freeing
out of nvmet_rdma_release_rsp() into a new helper function
nvmet_rdma_free_rsp_resources().
This is a refactoring with no functional change, in preparation for the
following patch that uses nvmet_rdma_free_rsp_resources().
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Refactor nvme rdma I/O queue setup to use async API, combining
allocation and startup into a single parallel operation per queue. This
reduces connection and reconnection setup time when there are delays in
establishing connections, which is especially important for
high-core-count hosts.
Key changes:
- Use async API to facilitate parallel calls for io queue setup.
- Add nvme_rdma_setup_ctx for propagating errors from async workers.
- Remove nvme_rdma_alloc_io_queues() and nvme_rdma_start_io_queues();
their logic is folded into nvme_rdma_setup_io_queues() and
nvme_rdma_configure_io_queues().
- Move queue count negotiation (nvme_set_queue_count,
nvmf_set_io_queues) from the removed nvme_rdma_alloc_io_queues()
into nvme_rdma_configure_io_queues().
Testing on a 64-core host with 64 IO-queues shows
nvme-rdma connection time reduced from ~1.4s to 416ms.
Signed-off-by: Surabhi Gogte <sgogte@purestorage.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Callers are responsible for initializing queue->ctrl and queue->queue_size
before calling nvme_rdma_alloc_queue(), which now derives ctrl and idx
from the queue pointer directly. This removes redundant assignments inside
the function and simplifies the interface.
Signed-off-by: Surabhi Gogte <sgogte@purestorage.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Include the zone index when reporting an invalid zone type during zone
descriptor parsing.
Signed-off-by: Xixin Liu <liuxixin@kylinos.cn>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
With Partial Report (PR=1), the Number of Zones (NZ) field in the report
header must equal the number of zone descriptors fully transferred in the
DMA buffer (ZNS Command Set Specification Rev 1.2, section 3.4.2).
nvme_ns_report_zones() does not cap the parse loop by max_in_buf derived
from buflen. Cap nz with min3() over the device-reported count, nr_zones -
zone_idx, and max_in_buf.
Signed-off-by: Xixin Liu <liuxixin@kylinos.cn>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
apple_nvme_init_queue() initializes queue state and then marks the queue
enabled. The interrupt and request paths check enabled before using that
queue state.
The old wmb() after WRITE_ONCE(enabled, true) does not publish the
earlier initialization before enabled becomes visible. Use a release store
when enabling the queue and acquire loads when testing it.
Although the shutdown-side enabled accesses are not used for publishing
queue initialization, use helpers for them as well for consistency.
Fixes: 5bd2927ace ("nvme-apple: Add initial Apple SoC NVMe driver")
Signed-off-by: Gui-Dong Han <hanguidong02@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Zoned multipath namespace heads get BLK_FEAT_ZONED and their limits are
refreshed from the paths, but the zone state for the head disk is never
initialized. The previous nr_zones assignment only updated a single
field and did not allocate or populate the block layer's per-zone state.
The failure was found with xfstests xfs/643 and xfs/646 on an NVMe
ZNS multipath namespace. Tracing showed regular REQ_OP_WRITE I/O being
submitted to sequential zones through the multipath head.
That leaves the head disk without valid zone condition information. Code
using the head device, such as bdev_zone_is_seq(), can then treat a
sequential zone as non-sequential and submit regular writes to it.
Add a small helper to run blk_revalidate_disk_zones() for a live zoned
namespace head after the path limits have been committed and when a path
becomes live. Return the error to the namespace update path, and keep the
live path transition as a warning-only update. Drop the nr_zones copy, as
blk_revalidate_disk_zones() updates it together with the rest of the zoned
disk state.
Signed-off-by: Yao Sang <sangyao@kylinos.cn>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
queue/wbt_lat_usec controls both the stored WBT latency target and the
effective WBT enable state.
The old no-op check skipped updates whenever the converted latency
matched the stored min_lat_nsec. That check ignored whether the current
WBT state already matched the state requested by the write. For a queue
disabled by default, attempting to enable WBT by writing the default
value through sysfs could return success while the enable state was left
unchanged.
Treat a write as a no-op only when both the stored latency and the
effective WBT enabled state already match the converted value.
Signed-off-by: Guzebing <guzebing1612@gmail.com>
Link: https://patch.msgid.link/20260621014030.1625306-1-guzebing1612@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
blk_hctx_poll() can busy-poll until a completion is found or
need_resched() becomes true. On preemptible kernels, the scheduler can
set TIF_NEED_RESCHED on the timer tick and preempt the task at IRQ
return before the loop condition re-evaluates it. After the context
switch, the flag is cleared, so the poller can continue spinning instead
of returning to its caller.
This can happen with io_uring IOPOLL reads inside iocb_bio_iopoll(),
which holds the rcu_read_lock() while calling bio_poll(). If another
poller on the same polled queue drains the available completions, this
poller may repeatedly find no completions and remain inside the RCU
read-side critical section long enough to trigger RCU stall reports:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: Tasks blocked on level-1 rcu_node (CPUs 0-9): P3961
rcu: (detected by 3, t=60002 jiffies, g=18533, q=4943 ncpus=20)
task:fio state:R running task stack:0 pid:3961
Call Trace:
<TASK>
? nvme_poll+0x36/0xa0 [nvme]
? blk_hctx_poll+0x39/0x90
? blk_mq_poll+0x30/0x60
? bio_poll+0x87/0x170
? iocb_bio_iopoll+0x32/0x50
? io_uring_classic_poll+0x25/0x50
? io_do_iopoll+0x216/0x420
? __do_sys_io_uring_enter+0x2c7/0x7c0
Reproducible with:
fio -filename=/dev/nvme0n1 -direct=1 -size=4g -rw=randread \
--numjobs=32 -bs=4K -ioengine=io_uring -hipri=1 -iodepth=1 \
--registerfiles=1 --group_reporting --thread
Record the starting jiffy and exit the loop once jiffies has advanced.
This bounds each blk_hctx_poll() invocation while also covering the
case where the reschedule flag was cleared by the context switch
before the loop condition could observe it.
Fixes: f22ecf9c14 ("blk-mq: delete task running check in blk_hctx_poll()")
Reviewed-by: Fengnan Chang <changfengnan@bytedance.com>
Suggested-by: Fengnan Chang <changfengnan@bytedance.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Signed-off-by: Alok Rathore <alok.rathore@samsung.com>
Link: https://patch.msgid.link/20260617155051.1266079-1-anuj20.g@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Otherwise zone append commands will miss their integrity data. While
this works "fine" for auto-PI, it break file system PI and non-PI
metadata.
With this XFS on ZNS namespace with non-PI metadata and 512 byte sectors
with PI work, while PI 4k sector formats with PI work only when Caleb's
"block: fix integrity offset/length conversions" is applied as well.
Note that unlike regular writes, zone append does need remapping as
partitions are not supported on zoned block devices.
Fixes: df3c485e0e ("block: switch on bio operation in bio_integrity_prep")
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
Link: https://patch.msgid.link/20260624080014.1998650-3-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
bio_integrity_alloc_buf usage of GFP_ flags is messed up. For one it
mixes GFP_NOFS and GFP_NOIO for neighbouring allocations, but it also
makes the allocations fail more often than needed. That code was copied
from bio_alloc_bioset which needs to do that so that it can punt to the
rescuer workqueue, but none of that is needed for the integrity
allocations that either sits in the file system or at the very bottom
of the I/O stack. Failing early means we'll do a fully waiting
allocation from the mempool ->alloc callback which is usually much
larger than required.
Fix this by passing a gfp_t so that the file system path can pass
GFP_NOFS and the auto-integrity code can pass GFP_NOIO, and don't
modify the allocation type except for disabling warnings.
Fixes: ec7f31b2a2 ("block: make bio auto-integrity deadlock safe")
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
Link: https://patch.msgid.link/20260624080014.1998650-2-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
