mirrors/linux
2
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-07-16 17:57:38 -04:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'for-7.2/block-20260615' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux

Browse Source 
Pull block updates from Jens Axboe:

 - NVMe pull request via Keith:
     - Per-controller admin and IO timeout sysfs attributes, and
       letting the block layer set request timeouts (Maurizio,
       Maximilian)
     - Multipath passthrough iostats, and PCI P2PDMA enablement for
       multipath devices (Keith, Kiran)
     - A new diag sysfs attribute group exporting per-controller
       counters (retries, multipath failover, error counters, requeue
       and failure counts, reset and reconnect events) (Nilay)
     - FDP configuration validation and bounds check fixes (liuxixin)
     - Various nvmet fixes, including a pre-auth out-of-bounds read in
       the Discovery Get Log Page handler, auth payload bounds
       validation, and tcp error-path leak fixes (Bryam, Tianchu,
       Geliang)
     - nvme-tcp lockdep and workqueue fixes (Shin'ichiro, Kuniyuki,
       Eric)
     - Assorted other fixes and cleanups (John, Yao, Chao, Mateusz,
       Achkinazi, Wentao)

 - MD pull request via Yu Kuai:
     - raid1/raid10 fixes for a deadlock in the read error recovery
       path, error-path detection and bio accounting with cloned bios,
       and an nr_pending leak in the REQ_ATOMIC bad-block error path
       (Abd-Alrhman)
     - PCI P2PDMA propagation from member devices to the RAID device
       (Kiran)
     - dm-raid bio requeue fix, and various smaller fixes and cleanups
       (Benjamin, Chen, Li, Thorsten)

 - Enable Clang lock context analysis for the block layer, with the
   accompanying annotations across queue limits, the blk_holder_ops
   callbacks, crypto, cgroup, iocost, kyber and mq-deadline (Bart)

 - Block status code infrastructure work: a tagged status table, a
   str_to_blk_op() helper, a bio_endio_status() helper, and on top of
   that a new configurable block-layer error injection facility
   (Christoph)

 - DRBD netlink rework, replacing the genl_magic machinery with explicit
   netlink serialization and moving the DRBD UAPI headers to
   include/uapi/linux/ (Christoph Böhmwalder)

 - bvec improvements: a bvec_folio() helper and making the bvec_iter
   helpers proper inline functions (Willy, Christoph)

 - ublk cleanups and a canceling-flag fix for the disk-not-allocated
   case (Caleb, Ming)

 - Partition handling fixes: bound the AIX pp_count scan, fix an of_node
   refcount leak, and replace __get_free_page() with kmalloc() (Bryam,
   Wentao, Mike)

 - Convert numa_node to int in blk_mq_hw_ctx and ->init_request, and add
   WQ_PERCPU to the block workqueue users (Mateusz, Marco)

 - Block statistics and tracing: propagate in-flight to the whole disk
   on partition IO, export passthrough stats, and a new
   block_rq_tag_wait tracepoint (Tang, Keith, Aaron)

 - A round of removals, unexports and cleanups across bio, direct-io and
   the bvec helpers (Christoph)

 - Various driver fixes (mtip32xx use-after-free, rbd snap_count
   validation and strscpy conversion, nbd socket lockdep reclassify,
   virtio-blk zone report clamp, floppy) and a batch of MAINTAINERS
   email/list updates (Coly, Li, Yu, Christoph Böhmwalder)

 - Other little fixes and cleanups all over

* tag 'for-7.2/block-20260615' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (117 commits)
  MAINTAINERS: Update Coly Li's email address
  block: check bio split for unaligned bvec
  nbd: Reclassify sockets to avoid lockdep circular dependency
  block: add configurable error injection
  block: add a str_to_blk_op helper
  block: add a "tag" for block status codes
  block: add a macro to initialize the status table
  floppy: Drop unused pnp driver data
  block: propagate in_flight to whole disk on partition I/O
  virtio-blk: clamp zone report to the report buffer capacity
  block: optimize I/O merge hot path with unlikely() hints
  drivers/block/rbd: Use strscpy() to copy strings into arrays
  partitions: aix: bound the pp_count scan to the ppe array
  block: Enable lock context analysis
  block/mq-deadline: Make the lock context annotations compatible with Clang
  block/Kyber: Make the lock context annotations compatible with Clang
  block/blk-mq-debugfs: Improve lock context annotations
  block/blk-iocost: Inline iocg_lock() and iocg_unlock()
  block/blk-iocost: Split ioc_rqos_throttle()
  block/crypto: Annotate the crypto functions
  ...
This commit is contained in:
Linus Torvalds
2026-06-16 13:02:47 +05:30
parent 9b40ba14ed c7c76f9232
commit ba9c792c82
98 changed files with 5782 additions and 2323 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
Documentation/block/error-injection.rst Normal file
View File

@@ -0,0 +1,59 @@
.. SPDX-License-Identifier: GPL-2.0
============================
Configurable Error Injection
============================
Overview
--------
Configurable error injection allows injecting specific block layer status codes
for sector ranges of a block device. Errors can be injected unconditionally, or
with a given probability.
To use configurable error injection, CONFIG_BLK_ERROR_INJECTION must be enabled.
The only interface is the error_injection debugfs file, which is created for
each registered gendisk. Writes to this file are used to create or delete rules
and reads return a list of the current error injection sites.
Options
-------
The following options specify the operations:
=================== =======================================================
add add a new rule
removeall remove all existing rules
=================== =======================================================
The following options specify the details of the rule for the add operation:
=================== =======================================================
op=<string> block layer operation this rule applies to. This uses
the XYZ for each REQ_OP_XYZ operation, e.g. READ, WRITE
or DISCARD. Mandatory.
status=<string> Status to return. This uses XYZ for each BLK_STS_XYZ
code, e.g. IOERR or MEDIUM. Mandatory.
start=<number> First block layer sector the rule applies to.
Optional, defaults to 0.
nr_sectors=<number> Number of sectors this rule applies.
Optional, defaults to the remainder of the device.
chance=<number> Only return a failure with a likelihood of 1/chance.
Optional, defaults to 1 (always).
=================== =======================================================
Example
-------
Return BLK_STS_IOERR for one in 10 reads of sector 0 of /dev/nvme0n1:
$ echo 'add,op=READ,start=0,status=IOERR,chance=10' > /sys/kernel/debug/block/nvme0n1/error_injection
Return BLK_STS_MEDIUM for every write to /dev/nvme0n1:
$ echo 'add,op=WRITE,start=0,status=MEDIUM' > /sys/kernel/debug/block/nvme0n1/error_injection
Remove all rules for /dev/nvme0n1:
$ echo 'removeall' > /sys/kernel/debug/block/nvme0n1/error_injection

1
Documentation/block/index.rst
View File

@@ -22,3 +22,4 @@ Block
switching-sched
writeback_cache_control
ublk
error-injection

1
Documentation/core-api/kernel-api.rst
View File

@@ -307,6 +307,7 @@ Accounting Framework
Block Devices
=============
.. kernel-doc:: include/linux/bvec.h
.. kernel-doc:: include/linux/bio.h
.. kernel-doc:: block/blk-core.c
:export:

10
MAINTAINERS
View File

@@ -4462,7 +4462,7 @@ F: include/uapi/linux/batman_adv.h
F: net/batman-adv/
BCACHE (BLOCK LAYER CACHE)
M: Coly Li <colyli@fnnas.com>
M: Coly Li <colyli@fygo.io>
M: Kent Overstreet <kent.overstreet@linux.dev>
L: linux-bcache@vger.kernel.org
S: Maintained
@@ -4500,7 +4500,7 @@ F: Documentation/filesystems/befs.rst
F: fs/befs/
BFQ I/O SCHEDULER
M: Yu Kuai <yukuai@fnnas.com>
M: Yu Kuai <yukuai@fygo.io>
L: linux-block@vger.kernel.org
S: Odd Fixes
F: Documentation/block/bfq-iosched.rst
@@ -7804,7 +7804,7 @@ DRBD DRIVER
M: Philipp Reisner <philipp.reisner@linbit.com>
M: Lars Ellenberg <lars.ellenberg@linbit.com>
M: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
L: drbd-dev@lists.linbit.com
L: drbd-dev@lists.linux.dev
S: Supported
W: http://www.drbd.org
T: git git://git.linbit.com/linux-drbd.git
@@ -24896,8 +24896,8 @@ F: include/linux/property.h
SOFTWARE RAID (Multiple Disks) SUPPORT
M: Song Liu <song@kernel.org>
M: Yu Kuai <yukuai@fnnas.com>
R: Li Nan <linan122@huawei.com>
M: Yu Kuai <yukuai@fygo.io>
R: Li Nan <magiclinan@didiglobal.com>
R: Xiao Ni <xiao@kernel.org>
L: linux-raid@vger.kernel.org
S: Supported

8
block/Kconfig
View File

@@ -221,6 +221,14 @@ config BLOCK_HOLDER_DEPRECATED
config BLK_MQ_STACKING
bool
config BLK_ERROR_INJECTION
bool "Enable block layer error injection"
select JUMP_LABEL if HAVE_ARCH_JUMP_LABEL
help
Enable inserting arbitrary block errors through a debugfs interface.
See Documentation/block/error-injection.rst for details.
source "block/Kconfig.iosched"
endif # BLOCK

3
block/Makefile
View File

@@ -3,6 +3,8 @@
# Makefile for the kernel block layer
#
CONTEXT_ANALYSIS := y
obj-y := bdev.o fops.o bio.o elevator.o blk-core.o blk-sysfs.o \
blk-flush.o blk-settings.o blk-ioc.o blk-map.o \
blk-merge.o blk-timeout.o blk-lib.o blk-mq.o \
@@ -11,6 +13,7 @@ obj-y := bdev.o fops.o bio.o elevator.o blk-core.o blk-sysfs.o \
genhd.o ioprio.o badblocks.o partitions/ blk-rq-qos.o \
disk-events.o blk-ia-ranges.o early-lookup.o
obj-$(CONFIG_BLK_ERROR_INJECTION) += error-injection.o
obj-$(CONFIG_BLK_DEV_BSG_COMMON) += bsg.o
obj-$(CONFIG_BLK_DEV_BSGLIB) += bsg-lib.o
obj-$(CONFIG_BLK_CGROUP) += blk-cgroup.o

13
block/bdev.c
View File

@@ -446,15 +446,10 @@ EXPORT_SYMBOL_GPL(blockdev_superblock);
void __init bdev_cache_init(void)
{
int err;
bdev_cachep = kmem_cache_create("bdev_cache", sizeof(struct bdev_inode),
0, (SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|
SLAB_ACCOUNT|SLAB_PANIC),
init_once);
err = register_filesystem(&bd_type);
if (err)
panic("Cannot register bdev pseudo-fs");
blockdev_mnt = kern_mount(&bd_type);
if (IS_ERR(blockdev_mnt))
panic("Cannot create bdev pseudo-fs");
@@ -1250,7 +1245,13 @@ void bdev_mark_dead(struct block_device *bdev, bool surprise)
bdev->bd_holder_ops->mark_dead(bdev, surprise);
else {
mutex_unlock(&bdev->bd_holder_lock);
sync_blockdev(bdev);
/*
* On surprise removal the device is already gone; syncing is
* futile and can hang forever waiting on I/O that will never
* complete. Match fs_bdev_mark_dead(), which also skips it.
*/
if (!surprise)
sync_blockdev(bdev);
}
invalidate_bdev(bdev);

54
block/bfq-cgroup.c
View File

@@ -300,6 +300,25 @@ static struct bfq_group *bfqg_parent(struct bfq_group *bfqg)
return pblkg ? blkg_to_bfqg(pblkg) : NULL;
}
static void bfqg_stats_exit(struct bfqg_stats *stats)
{
blkg_rwstat_exit(&stats->bytes);
blkg_rwstat_exit(&stats->ios);
#ifdef CONFIG_BFQ_CGROUP_DEBUG
blkg_rwstat_exit(&stats->merged);
blkg_rwstat_exit(&stats->service_time);
blkg_rwstat_exit(&stats->wait_time);
blkg_rwstat_exit(&stats->queued);
bfq_stat_exit(&stats->time);
bfq_stat_exit(&stats->avg_queue_size_sum);
bfq_stat_exit(&stats->avg_queue_size_samples);
bfq_stat_exit(&stats->dequeue);
bfq_stat_exit(&stats->group_wait_time);
bfq_stat_exit(&stats->idle_time);
bfq_stat_exit(&stats->empty_time);
#endif
}
struct bfq_group *bfqq_group(struct bfq_queue *bfqq)
{
struct bfq_entity *group_entity = bfqq->entity.parent;
@@ -321,8 +340,10 @@ static void bfqg_get(struct bfq_group *bfqg)
static void bfqg_put(struct bfq_group *bfqg)
{
if (refcount_dec_and_test(&bfqg->ref))
if (refcount_dec_and_test(&bfqg->ref)) {
bfqg_stats_exit(&bfqg->stats);
kfree(bfqg);
}
}
static void bfqg_and_blkg_get(struct bfq_group *bfqg)
@@ -433,25 +454,6 @@ void bfq_init_entity(struct bfq_entity *entity, struct bfq_group *bfqg)
entity->sched_data = &bfqg->sched_data;
}
static void bfqg_stats_exit(struct bfqg_stats *stats)
{
blkg_rwstat_exit(&stats->bytes);
blkg_rwstat_exit(&stats->ios);
#ifdef CONFIG_BFQ_CGROUP_DEBUG
blkg_rwstat_exit(&stats->merged);
blkg_rwstat_exit(&stats->service_time);
blkg_rwstat_exit(&stats->wait_time);
blkg_rwstat_exit(&stats->queued);
bfq_stat_exit(&stats->time);
bfq_stat_exit(&stats->avg_queue_size_sum);
bfq_stat_exit(&stats->avg_queue_size_samples);
bfq_stat_exit(&stats->dequeue);
bfq_stat_exit(&stats->group_wait_time);
bfq_stat_exit(&stats->idle_time);
bfq_stat_exit(&stats->empty_time);
#endif
}
static int bfqg_stats_init(struct bfqg_stats *stats, gfp_t gfp)
{
if (blkg_rwstat_init(&stats->bytes, gfp) ||
@@ -552,7 +554,6 @@ static void bfq_pd_free(struct blkg_policy_data *pd)
{
struct bfq_group *bfqg = pd_to_bfqg(pd);
bfqg_stats_exit(&bfqg->stats);
bfqg_put(bfqg);
}
@@ -1051,9 +1052,13 @@ static ssize_t bfq_io_set_device_weight(struct kernfs_open_file *of,
blkg_conf_init(&ctx, buf);
ret = blkg_conf_open_bdev(&ctx);
if (ret)
return ret;
ret = blkg_conf_prep(blkcg, &blkcg_policy_bfq, &ctx);
if (ret)
goto out;
goto close_bdev;
if (sscanf(ctx.body, "%llu", &v) == 1) {
/* require "default" on dfl */
@@ -1074,8 +1079,11 @@ static ssize_t bfq_io_set_device_weight(struct kernfs_open_file *of,
bfq_group_set_weight(bfqg, bfqg->entity.weight, v);
ret = 0;
}
out:
blkg_conf_exit(&ctx);
blkg_conf_unprep(&ctx);
close_bdev:
blkg_conf_close_bdev(&ctx);
return ret ?: nbytes;
}

52
block/bio.c
View File

@@ -635,15 +635,15 @@ struct bio *bio_kmalloc(unsigned short nr_vecs, gfp_t gfp_mask)
}
EXPORT_SYMBOL(bio_kmalloc);
void zero_fill_bio_iter(struct bio *bio, struct bvec_iter start)
void zero_fill_bio(struct bio *bio)
{
struct bio_vec bv;
struct bvec_iter iter;
__bio_for_each_segment(bv, bio, iter, start)
bio_for_each_segment(bv, bio, iter)
memzero_bvec(&bv);
}
EXPORT_SYMBOL(zero_fill_bio_iter);
EXPORT_SYMBOL(zero_fill_bio);
/**
* bio_truncate - truncate the bio to small size of @new_size
@@ -1300,7 +1300,7 @@ static void bio_free_folios(struct bio *bio)
int i;
bio_for_each_bvec_all(bv, bio, i) {
struct folio *folio = page_folio(bv->bv_page);
struct folio *folio = bvec_folio(bv);
if (!is_zero_folio(folio))
folio_put(folio);
@@ -1409,7 +1409,7 @@ int bio_iov_iter_bounce(struct bio *bio, struct iov_iter *iter, size_t maxlen,
static void bvec_unpin(struct bio_vec *bv, bool mark_dirty)
{
struct folio *folio = page_folio(bv->bv_page);
struct folio *folio = bvec_folio(bv);
size_t nr_pages = (bv->bv_offset + bv->bv_len - 1) / PAGE_SIZE -
bv->bv_offset / PAGE_SIZE + 1;
@@ -1443,7 +1443,7 @@ static void bio_iov_iter_unbounce_read(struct bio *bio, bool is_error,
bvec_unpin(&bio->bi_io_vec[1 + i], mark_dirty);
}
folio_put(page_folio(bio->bi_io_vec[0].bv_page));
folio_put(bvec_folio(&bio->bi_io_vec[0]));
}
/**
@@ -1578,26 +1578,6 @@ void __bio_advance(struct bio *bio, unsigned bytes)
}
EXPORT_SYMBOL(__bio_advance);
void bio_copy_data_iter(struct bio *dst, struct bvec_iter *dst_iter,
struct bio *src, struct bvec_iter *src_iter)
{
while (src_iter->bi_size && dst_iter->bi_size) {
struct bio_vec src_bv = bio_iter_iovec(src, *src_iter);
struct bio_vec dst_bv = bio_iter_iovec(dst, *dst_iter);
unsigned int bytes = min(src_bv.bv_len, dst_bv.bv_len);
void *src_buf = bvec_kmap_local(&src_bv);
void *dst_buf = bvec_kmap_local(&dst_bv);
memcpy(dst_buf, src_buf, bytes);
kunmap_local(dst_buf);
kunmap_local(src_buf);
bio_advance_iter_single(src, src_iter, bytes);
bio_advance_iter_single(dst, dst_iter, bytes);
}
}
EXPORT_SYMBOL(bio_copy_data_iter);
/**
* bio_copy_data - copy contents of data buffers from one bio to another
@@ -1612,7 +1592,21 @@ void bio_copy_data(struct bio *dst, struct bio *src)
struct bvec_iter src_iter = src->bi_iter;
struct bvec_iter dst_iter = dst->bi_iter;
bio_copy_data_iter(dst, &dst_iter, src, &src_iter);
while (src_iter.bi_size && dst_iter.bi_size) {
struct bio_vec src_bv = bio_iter_iovec(src, src_iter);
struct bio_vec dst_bv = bio_iter_iovec(dst, dst_iter);
unsigned int bytes = min(src_bv.bv_len, dst_bv.bv_len);
void *src_buf = bvec_kmap_local(&src_bv);
void *dst_buf = bvec_kmap_local(&dst_bv);
memcpy(dst_buf, src_buf, bytes);
kunmap_local(dst_buf);
kunmap_local(src_buf);
bio_advance_iter_single(src, &src_iter, bytes);
bio_advance_iter_single(dst, &dst_iter, bytes);
}
}
EXPORT_SYMBOL(bio_copy_data);
@@ -1659,7 +1653,6 @@ void bio_set_pages_dirty(struct bio *bio)
folio_unlock(fi.folio);
}
}
EXPORT_SYMBOL_GPL(bio_set_pages_dirty);
/*
* bio_check_pages_dirty() will check that all the BIO's pages are still dirty.
@@ -1718,7 +1711,6 @@ void bio_check_pages_dirty(struct bio *bio)
spin_unlock_irqrestore(&bio_dirty_lock, flags);
schedule_work(&bio_dirty_work);
}
EXPORT_SYMBOL_GPL(bio_check_pages_dirty);
static inline bool bio_remaining_done(struct bio *bio)
{
@@ -1884,7 +1876,7 @@ EXPORT_SYMBOL_GPL(bio_trim);
* create memory pools for biovec's in a bio_set.
* use the global biovec slabs created for general use.
*/
int biovec_init_pool(mempool_t *pool, int pool_entries)
static int biovec_init_pool(mempool_t *pool, int pool_entries)
{
struct biovec_slab *bp = bvec_slabs + ARRAY_SIZE(bvec_slabs) - 1;

98
block/blk-cgroup.c
View File

@@ -755,7 +755,7 @@ EXPORT_SYMBOL_GPL(__blkg_prfill_u64);
*
* Initialize @ctx which can be used to parse blkg config input string @input.
* Once initialized, @ctx can be used with blkg_conf_open_bdev() and
* blkg_conf_prep(), and must be cleaned up with blkg_conf_exit().
* blkg_conf_prep().
*/
void blkg_conf_init(struct blkg_conf_ctx *ctx, char *input)
{
@@ -771,10 +771,7 @@ EXPORT_SYMBOL_GPL(blkg_conf_init);
* @ctx->input and get and store the matching bdev in @ctx->bdev. @ctx->body is
* set to point past the device node prefix.
*
* This function may be called multiple times on @ctx and the extra calls become
* NOOPs. blkg_conf_prep() implicitly calls this function. Use this function
* explicitly if bdev access is needed without resolving the blkcg / policy part
* of @ctx->input. Returns -errno on error.
* Returns: -errno on error.
*/
int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx)
{
@@ -783,8 +780,8 @@ int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx)
struct block_device *bdev;
int key_len;
if (ctx->bdev)
return 0;
if (WARN_ON_ONCE(ctx->bdev))
return -EINVAL;
if (sscanf(input, "%u:%u%n", &major, &minor, &key_len) != 2)
return -EINVAL;
@@ -813,38 +810,7 @@ int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx)
ctx->bdev = bdev;
return 0;
}
/*
* Similar to blkg_conf_open_bdev, but additionally freezes the queue,
* ensures the correct locking order between freeze queue and q->rq_qos_mutex.
*
* This function returns negative error on failure. On success it returns
* memflags which must be saved and later passed to blkg_conf_exit_frozen
* for restoring the memalloc scope.
*/
unsigned long __must_check blkg_conf_open_bdev_frozen(struct blkg_conf_ctx *ctx)
{
int ret;
unsigned long memflags;
if (ctx->bdev)
return -EINVAL;
ret = blkg_conf_open_bdev(ctx);
if (ret < 0)
return ret;
/*
* At this point, we haven’t started protecting anything related to QoS,
* so we release q->rq_qos_mutex here, which was first acquired in blkg_
* conf_open_bdev. Later, we re-acquire q->rq_qos_mutex after freezing
* the queue to maintain the correct locking order.
*/
mutex_unlock(&ctx->bdev->bd_queue->rq_qos_mutex);
memflags = blk_mq_freeze_queue(ctx->bdev->bd_queue);
mutex_lock(&ctx->bdev->bd_queue->rq_qos_mutex);
return memflags;
}
EXPORT_SYMBOL_GPL(blkg_conf_open_bdev);
/**
* blkg_conf_prep - parse and prepare for per-blkg config update
@@ -857,22 +823,20 @@ unsigned long __must_check blkg_conf_open_bdev_frozen(struct blkg_conf_ctx *ctx)
* following MAJ:MIN, @ctx->bdev points to the target block device and
* @ctx->blkg to the blkg being configured.
*
* blkg_conf_open_bdev() may be called on @ctx beforehand. On success, this
* blkg_conf_open_bdev() must be called on @ctx beforehand. On success, this
* function returns with queue lock held and must be followed by
* blkg_conf_exit().
* blkg_conf_close_bdev().
*/
int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
struct blkg_conf_ctx *ctx)
__acquires(&bdev->bd_queue->queue_lock)
{
struct gendisk *disk;
struct request_queue *q;
struct blkcg_gq *blkg;
int ret;
ret = blkg_conf_open_bdev(ctx);
if (ret)
return ret;
if (WARN_ON_ONCE(!ctx->bdev))
return -EINVAL;
disk = ctx->bdev->bd_disk;
q = disk->queue;
@@ -970,43 +934,29 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
EXPORT_SYMBOL_GPL(blkg_conf_prep);
/**
* blkg_conf_exit - clean up per-blkg config update
* blkg_conf_unprep - counterpart of blkg_conf_prep()
* @ctx: blkg_conf_ctx initialized with blkg_conf_init()
*
* Clean up after per-blkg config update. This function must be called on all
* blkg_conf_ctx's initialized with blkg_conf_init().
*/
void blkg_conf_exit(struct blkg_conf_ctx *ctx)
__releases(&ctx->bdev->bd_queue->queue_lock)
__releases(&ctx->bdev->bd_queue->rq_qos_mutex)
void blkg_conf_unprep(struct blkg_conf_ctx *ctx)
{
if (ctx->blkg) {
spin_unlock_irq(&bdev_get_queue(ctx->bdev)->queue_lock);
ctx->blkg = NULL;
}
if (ctx->bdev) {
mutex_unlock(&ctx->bdev->bd_queue->rq_qos_mutex);
blkdev_put_no_open(ctx->bdev);
ctx->body = NULL;
ctx->bdev = NULL;
}
WARN_ON_ONCE(!ctx->blkg);
spin_unlock_irq(&ctx->bdev->bd_disk->queue->queue_lock);
ctx->blkg = NULL;
}
EXPORT_SYMBOL_GPL(blkg_conf_exit);
EXPORT_SYMBOL_GPL(blkg_conf_unprep);
/*
* Similar to blkg_conf_exit, but also unfreezes the queue. Should be used
* when blkg_conf_open_bdev_frozen is used to open the bdev.
/**
* blkg_conf_close_bdev - counterpart of blkg_conf_open_bdev()
* @ctx: blkg_conf_ctx initialized with blkg_conf_init()
*/
void blkg_conf_exit_frozen(struct blkg_conf_ctx *ctx, unsigned long memflags)
void blkg_conf_close_bdev(struct blkg_conf_ctx *ctx)
{
if (ctx->bdev) {
struct request_queue *q = ctx->bdev->bd_queue;
blkg_conf_exit(ctx);
blk_mq_unfreeze_queue(q, memflags);
}
mutex_unlock(&ctx->bdev->bd_queue->rq_qos_mutex);
blkdev_put_no_open(ctx->bdev);
ctx->body = NULL;
ctx->bdev = NULL;
}
EXPORT_SYMBOL_GPL(blkg_conf_close_bdev);
static void blkg_iostat_add(struct blkg_iostat *dst, struct blkg_iostat *src)
{

13
block/blk-cgroup.h
View File

@@ -218,12 +218,15 @@ struct blkg_conf_ctx {
};
void blkg_conf_init(struct blkg_conf_ctx *ctx, char *input);
int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx);
unsigned long blkg_conf_open_bdev_frozen(struct blkg_conf_ctx *ctx);
int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx)
__cond_acquires(0, &ctx->bdev->bd_queue->rq_qos_mutex);
int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
struct blkg_conf_ctx *ctx);
void blkg_conf_exit(struct blkg_conf_ctx *ctx);
void blkg_conf_exit_frozen(struct blkg_conf_ctx *ctx, unsigned long memflags);
struct blkg_conf_ctx *ctx)
__cond_acquires(0, &ctx->bdev->bd_disk->queue->queue_lock);
void blkg_conf_unprep(struct blkg_conf_ctx *ctx)
__releases(ctx->bdev->bd_disk->queue->queue_lock);
void blkg_conf_close_bdev(struct blkg_conf_ctx *ctx)
__releases(&ctx->bdev->bd_queue->rq_qos_mutex);
/**
* bio_issue_as_root_blkg - see if this bio needs to be issued as root blkg

104
block/blk-core.c
View File

@@ -50,6 +50,7 @@
#include "blk-cgroup.h"
#include "blk-throttle.h"
#include "blk-ioprio.h"
#include "error-injection.h"
struct dentry *blk_debugfs_root;
@@ -132,39 +133,56 @@ inline const char *blk_op_str(enum req_op op)
}
EXPORT_SYMBOL_GPL(blk_op_str);
enum req_op str_to_blk_op(const char *op)
{
int i;
for (i = 0; i < ARRAY_SIZE(blk_op_name); i++)
if (blk_op_name[i] && !strcmp(blk_op_name[i], op))
return (enum req_op)i;
return REQ_OP_LAST;
}
#define ENT(_tag, _errno, _desc) \
[BLK_STS_##_tag] = { \
.errno = _errno, \
.tag = __stringify(_tag), \
.name = _desc, \
}
static const struct {
int errno;
const char *tag;
const char *name;
} blk_errors[] = {
[BLK_STS_OK] = { 0, "" },
[BLK_STS_NOTSUPP] = { -EOPNOTSUPP, "operation not supported" },
[BLK_STS_TIMEOUT] = { -ETIMEDOUT, "timeout" },
[BLK_STS_NOSPC] = { -ENOSPC, "critical space allocation" },
[BLK_STS_TRANSPORT] = { -ENOLINK, "recoverable transport" },
[BLK_STS_TARGET] = { -EREMOTEIO, "critical target" },
[BLK_STS_RESV_CONFLICT] = { -EBADE, "reservation conflict" },
[BLK_STS_MEDIUM] = { -ENODATA, "critical medium" },
[BLK_STS_PROTECTION] = { -EILSEQ, "protection" },
[BLK_STS_RESOURCE] = { -ENOMEM, "kernel resource" },
[BLK_STS_DEV_RESOURCE] = { -EBUSY, "device resource" },
[BLK_STS_AGAIN] = { -EAGAIN, "nonblocking retry" },
[BLK_STS_OFFLINE] = { -ENODEV, "device offline" },
ENT(OK, 0, ""),
ENT(NOTSUPP, -EOPNOTSUPP, "operation not supported"),
ENT(TIMEOUT, -ETIMEDOUT, "timeout"),
ENT(NOSPC, -ENOSPC, "critical space allocation"),
ENT(TRANSPORT, -ENOLINK, "recoverable transport"),
ENT(TARGET, -EREMOTEIO, "critical target"),
ENT(RESV_CONFLICT, -EBADE, "reservation conflict"),
ENT(MEDIUM, -ENODATA, "critical medium"),
ENT(PROTECTION, -EILSEQ, "protection"),
ENT(RESOURCE, -ENOMEM, "kernel resource"),
ENT(DEV_RESOURCE, -EBUSY, "device resource"),
ENT(AGAIN, -EAGAIN, "nonblocking retry"),
ENT(OFFLINE, -ENODEV, "device offline"),
/* device mapper special case, should not leak out: */
[BLK_STS_DM_REQUEUE] = { -EREMCHG, "dm internal retry" },
ENT(DM_REQUEUE, -EREMCHG, "dm internal retry"),
/* zone device specific errors */
[BLK_STS_ZONE_OPEN_RESOURCE] = { -ETOOMANYREFS, "open zones exceeded" },
[BLK_STS_ZONE_ACTIVE_RESOURCE] = { -EOVERFLOW, "active zones exceeded" },
ENT(ZONE_OPEN_RESOURCE, -ETOOMANYREFS, "open zones exceeded"),
ENT(ZONE_ACTIVE_RESOURCE, -EOVERFLOW, "active zones exceeded"),
/* Command duration limit device-side timeout */
[BLK_STS_DURATION_LIMIT] = { -ETIME, "duration limit exceeded" },
[BLK_STS_INVAL] = { -EINVAL, "invalid" },
ENT(DURATION_LIMIT, -ETIME, "duration limit exceeded"),
ENT(INVAL, -EINVAL, "invalid"),
/* everything else not covered above: */
[BLK_STS_IOERR] = { -EIO, "I/O" },
ENT(IOERR, -EIO, "I/O"),
};
#undef ENT
blk_status_t errno_to_blk_status(int errno)
{
@@ -197,7 +215,32 @@ const char *blk_status_to_str(blk_status_t status)
return "<null>";
return blk_errors[idx].name;
}
EXPORT_SYMBOL_GPL(blk_status_to_str);
const char *blk_status_to_tag(blk_status_t status)
{
int idx = (__force int)status;
if (WARN_ON_ONCE(idx >= ARRAY_SIZE(blk_errors) || !blk_errors[idx].tag))
return "<null>";
return blk_errors[idx].tag;
}
blk_status_t tag_to_blk_status(const char *tag)
{
int i;
for (i = 0; i < ARRAY_SIZE(blk_errors); i++) {
if (blk_errors[i].tag &&
!strcmp(blk_errors[i].tag, tag))
return (__force blk_status_t)i;
}
/*
* Return BLK_STS_OK for mismatches as this function is intended to
* parse error status values.
*/
return BLK_STS_OK;
}
/**
* blk_sync_queue - cancel any pending callbacks on a queue
@@ -637,12 +680,10 @@ static void __submit_bio(struct bio *bio)
struct gendisk *disk = bio->bi_bdev->bd_disk;
if ((bio->bi_opf & REQ_POLLED) &&
!(disk->queue->limits.features & BLK_FEAT_POLL)) {
bio->bi_status = BLK_STS_NOTSUPP;
bio_endio(bio);
} else {
!(disk->queue->limits.features & BLK_FEAT_POLL))
bio_endio_status(bio, BLK_STS_NOTSUPP);
else
disk->fops->submit_bio(bio);
}
blk_queue_exit(disk->queue);
}
@@ -727,6 +768,9 @@ static void __submit_bio_noacct_mq(struct bio *bio)
void submit_bio_noacct_nocheck(struct bio *bio, bool split)
{
if (unlikely(blk_error_inject(bio)))
return;
blk_cgroup_bio_start(bio);
if (!bio_flagged(bio, BIO_TRACE_COMPLETION)) {
@@ -887,8 +931,7 @@ void submit_bio_noacct(struct bio *bio)
not_supported:
status = BLK_STS_NOTSUPP;
end_io:
bio->bi_status = status;
bio_endio(bio);
bio_endio_status(bio, status);
}
EXPORT_SYMBOL(submit_bio_noacct);
@@ -1042,7 +1085,7 @@ unsigned long bdev_start_io_acct(struct block_device *bdev, enum req_op op,
{
part_stat_lock();
update_io_ticks(bdev, start_time, false);
part_stat_local_inc(bdev, in_flight[op_is_write(op)]);
bdev_inc_in_flight(bdev, op);
part_stat_unlock();
return start_time;
@@ -1073,7 +1116,7 @@ void bdev_end_io_acct(struct block_device *bdev, enum req_op op,
part_stat_inc(bdev, ios[sgrp]);
part_stat_add(bdev, sectors[sgrp], sectors);
part_stat_add(bdev, nsecs[sgrp], jiffies_to_nsecs(duration));
part_stat_local_dec(bdev, in_flight[op_is_write(op)]);
bdev_dec_in_flight(bdev, op);
part_stat_unlock();
}
EXPORT_SYMBOL(bdev_end_io_acct);
@@ -1270,7 +1313,6 @@ void blk_io_schedule(void)
else
io_schedule();
}
EXPORT_SYMBOL_GPL(blk_io_schedule);
int __init blk_dev_init(void)
{

9
block/blk-crypto-fallback.c
View File

@@ -362,8 +362,7 @@ static void blk_crypto_fallback_encrypt_bio(struct bio *src_bio)
status = blk_crypto_get_keyslot(blk_crypto_fallback_profile,
bc->bc_key, &slot);
if (status != BLK_STS_OK) {
src_bio->bi_status = status;
bio_endio(src_bio);
bio_endio_status(src_bio, status);
return;
}
__blk_crypto_fallback_encrypt_bio(src_bio,
@@ -438,8 +437,7 @@ static void blk_crypto_fallback_decrypt_bio(struct work_struct *work)
}
mempool_free(f_ctx, bio_fallback_crypt_ctx_pool);
bio->bi_status = status;
bio_endio(bio);
bio_endio_status(bio, status);
}
/**
@@ -500,8 +498,7 @@ bool blk_crypto_fallback_bio_prep(struct bio *bio)
if (!__blk_crypto_cfg_supported(blk_crypto_fallback_profile,
&bc->bc_key->crypto_cfg)) {
bio->bi_status = BLK_STS_NOTSUPP;
bio_endio(bio);
bio_endio_status(bio, BLK_STS_NOTSUPP);
return false;
}

2
block/blk-crypto-profile.c
View File

@@ -43,6 +43,7 @@ struct blk_crypto_keyslot {
};
static inline void blk_crypto_hw_enter(struct blk_crypto_profile *profile)
__acquires(&profile->lock)
{
/*
* Calling into the driver requires profile->lock held and the device
@@ -55,6 +56,7 @@ static inline void blk_crypto_hw_enter(struct blk_crypto_profile *profile)
}
static inline void blk_crypto_hw_exit(struct blk_crypto_profile *profile)
__releases(&profile->lock)
{
up_write(&profile->lock);
if (profile->dev)

3
block/blk-crypto.c
View File

@@ -267,8 +267,7 @@ bool __blk_crypto_submit_bio(struct bio *bio)
if (!IS_ENABLED(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK)) {
pr_warn_once("%pg: crypto API fallback disabled; failing request.\n",
bdev);
bio->bi_status = BLK_STS_NOTSUPP;
bio_endio(bio);
bio_endio_status(bio, BLK_STS_NOTSUPP);
return false;
}
return blk_crypto_fallback_bio_prep(bio);

310
block/blk-iocost.c
View File

@@ -727,26 +727,6 @@ static void iocg_commit_bio(struct ioc_gq *iocg, struct bio *bio,
put_cpu_ptr(gcs);
}
static void iocg_lock(struct ioc_gq *iocg, bool lock_ioc, unsigned long *flags)
{
if (lock_ioc) {
spin_lock_irqsave(&iocg->ioc->lock, *flags);
spin_lock(&iocg->waitq.lock);
} else {
spin_lock_irqsave(&iocg->waitq.lock, *flags);
}
}
static void iocg_unlock(struct ioc_gq *iocg, bool unlock_ioc, unsigned long *flags)
{
if (unlock_ioc) {
spin_unlock(&iocg->waitq.lock);
spin_unlock_irqrestore(&iocg->ioc->lock, *flags);
} else {
spin_unlock_irqrestore(&iocg->waitq.lock, *flags);
}
}
#define CREATE_TRACE_POINTS
#include <trace/events/iocost.h>
@@ -1589,9 +1569,17 @@ static enum hrtimer_restart iocg_waitq_timer_fn(struct hrtimer *timer)
ioc_now(iocg->ioc, &now);
iocg_lock(iocg, pay_debt, &flags);
iocg_kick_waitq(iocg, pay_debt, &now);
iocg_unlock(iocg, pay_debt, &flags);
if (pay_debt) {
spin_lock_irqsave(&iocg->ioc->lock, flags);
spin_lock(&iocg->waitq.lock);
iocg_kick_waitq(iocg, pay_debt, &now);
spin_unlock(&iocg->waitq.lock);
spin_unlock_irqrestore(&iocg->ioc->lock, flags);
} else {
spin_lock_irqsave(&iocg->waitq.lock, flags);
iocg_kick_waitq(iocg, pay_debt, &now);
spin_unlock_irqrestore(&iocg->waitq.lock, flags);
}
return HRTIMER_NORESTART;
}
@@ -2614,6 +2602,88 @@ static u64 calc_size_vtime_cost(struct request *rq, struct ioc *ioc)
return cost;
}
enum over_budget_action {
action_retry,
action_commit,
action_wait,
action_return,
};
static enum over_budget_action
iocg_handle_over_budget(struct rq_qos *rqos, struct ioc_gq *iocg,
struct bio *bio, struct ioc_now *now,
struct iocg_wait *wait, bool use_debt, bool ioc_locked,
u64 abs_cost, u64 cost)
{
lockdep_assert_held(&iocg->waitq.lock);
/*
* @iocg must stay activated for debt and waitq handling. Deactivation
* is synchronized against both ioc->lock and waitq.lock and we won't
* get deactivated as long as we're waiting or have debt, so we're good
* if we're activated here. In the unlikely cases that we aren't, just
* issue the IO.
*/
if (unlikely(list_empty(&iocg->active_list)))
return action_commit;
/*
* We're over budget. If @bio has to be issued regardless, remember
* the abs_cost instead of advancing vtime. iocg_kick_waitq() will pay
* off the debt before waking more IOs.
*
* This way, the debt is continuously paid off each period with the
* actual budget available to the cgroup. If we just wound vtime, we
* would incorrectly use the current hw_inuse for the entire amount
* which, for example, can lead to the cgroup staying blocked for a
* long time even with substantially raised hw_inuse.
*
* An iocg with vdebt should stay online so that the timer can keep
* deducting its vdebt and [de]activate use_delay mechanism
* accordingly. We don't want to race against the timer trying to
* clear them and leave @iocg inactive w/ dangling use_delay heavily
* penalizing the cgroup and its descendants.
*/
if (use_debt) {
iocg_incur_debt(iocg, abs_cost, now);
if (iocg_kick_delay(iocg, now))
blkcg_schedule_throttle(rqos->disk,
(bio->bi_opf & REQ_SWAP) ==
REQ_SWAP);
return action_return;
}
/* guarantee that iocgs w/ waiters have maximum inuse */
if (!iocg->abs_vdebt && iocg->inuse != iocg->active) {
if (!ioc_locked)
return action_retry;
lockdep_assert_held(&iocg->ioc->lock);
propagate_weights(iocg, iocg->active, iocg->active, true, now);
}
/*
* Append self to the waitq and schedule the wakeup timer if we're
* the first waiter. The timer duration is calculated based on the
* current vrate. vtime and hweight changes can make it too short
* or too long. Each wait entry records the absolute cost it's
* waiting for to allow re-evaluation using a custom wait entry.
*
* If too short, the timer simply reschedules itself. If too long,
* the period timer will notice and trigger wakeups.
*
* All waiters are on iocg->waitq and the wait states are
* synchronized using waitq.lock.
*/
init_wait_func(&wait->wait, iocg_wake_fn);
wait->bio = bio;
wait->abs_cost = abs_cost;
wait->committed = false; /* will be set true by waker */
__add_wait_queue_entry_tail(&iocg->waitq, &wait->wait);
iocg_kick_waitq(iocg, ioc_locked, now);
return action_wait;
}
static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)
{
struct blkcg_gq *blkg = bio->bi_blkg;
@@ -2623,6 +2693,7 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)
struct iocg_wait wait;
u64 abs_cost, cost, vtime;
bool use_debt, ioc_locked;
enum over_budget_action action;
unsigned long flags;
/* bypass IOs if disabled, still initializing, or for root cgroup */
@@ -2662,81 +2733,34 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)
use_debt = bio_issue_as_root_blkg(bio) || fatal_signal_pending(current);
ioc_locked = use_debt || READ_ONCE(iocg->abs_vdebt);
retry_lock:
iocg_lock(iocg, ioc_locked, &flags);
/*
* @iocg must stay activated for debt and waitq handling. Deactivation
* is synchronized against both ioc->lock and waitq.lock and we won't
* get deactivated as long as we're waiting or has debt, so we're good
* if we're activated here. In the unlikely cases that we aren't, just
* issue the IO.
*/
if (unlikely(list_empty(&iocg->active_list))) {
iocg_unlock(iocg, ioc_locked, &flags);
if (ioc_locked) {
spin_lock_irqsave(&iocg->ioc->lock, flags);
spin_lock(&iocg->waitq.lock);
action = iocg_handle_over_budget(rqos, iocg, bio, &now, &wait,
use_debt, ioc_locked, abs_cost,
cost);
spin_unlock(&iocg->waitq.lock);
spin_unlock_irqrestore(&iocg->ioc->lock, flags);
} else {
spin_lock_irqsave(&iocg->waitq.lock, flags);
action = iocg_handle_over_budget(rqos, iocg, bio, &now, &wait,
use_debt, ioc_locked, abs_cost,
cost);
spin_unlock_irqrestore(&iocg->waitq.lock, flags);
}
switch (action) {
case action_retry:
ioc_locked = true;
goto retry_lock;
case action_commit:
iocg_commit_bio(iocg, bio, abs_cost, cost);
return;
}
/*
* We're over budget. If @bio has to be issued regardless, remember
* the abs_cost instead of advancing vtime. iocg_kick_waitq() will pay
* off the debt before waking more IOs.
*
* This way, the debt is continuously paid off each period with the
* actual budget available to the cgroup. If we just wound vtime, we
* would incorrectly use the current hw_inuse for the entire amount
* which, for example, can lead to the cgroup staying blocked for a
* long time even with substantially raised hw_inuse.
*
* An iocg with vdebt should stay online so that the timer can keep
* deducting its vdebt and [de]activate use_delay mechanism
* accordingly. We don't want to race against the timer trying to
* clear them and leave @iocg inactive w/ dangling use_delay heavily
* penalizing the cgroup and its descendants.
*/
if (use_debt) {
iocg_incur_debt(iocg, abs_cost, &now);
if (iocg_kick_delay(iocg, &now))
blkcg_schedule_throttle(rqos->disk,
(bio->bi_opf & REQ_SWAP) == REQ_SWAP);
iocg_unlock(iocg, ioc_locked, &flags);
case action_return:
return;
case action_wait:
break;
}
/* guarantee that iocgs w/ waiters have maximum inuse */
if (!iocg->abs_vdebt && iocg->inuse != iocg->active) {
if (!ioc_locked) {
iocg_unlock(iocg, false, &flags);
ioc_locked = true;
goto retry_lock;
}
propagate_weights(iocg, iocg->active, iocg->active, true,
&now);
}
/*
* Append self to the waitq and schedule the wakeup timer if we're
* the first waiter. The timer duration is calculated based on the
* current vrate. vtime and hweight changes can make it too short
* or too long. Each wait entry records the absolute cost it's
* waiting for to allow re-evaluation using a custom wait entry.
*
* If too short, the timer simply reschedules itself. If too long,
* the period timer will notice and trigger wakeups.
*
* All waiters are on iocg->waitq and the wait states are
* synchronized using waitq.lock.
*/
init_wait_func(&wait.wait, iocg_wake_fn);
wait.bio = bio;
wait.abs_cost = abs_cost;
wait.committed = false; /* will be set true by waker */
__add_wait_queue_entry_tail(&iocg->waitq, &wait.wait);
iocg_kick_waitq(iocg, ioc_locked, &now);
iocg_unlock(iocg, ioc_locked, &flags);
while (true) {
set_current_state(TASK_UNINTERRUPTIBLE);
if (wait.committed)
@@ -3140,19 +3164,25 @@ static ssize_t ioc_weight_write(struct kernfs_open_file *of, char *buf,
blkg_conf_init(&ctx, buf);
ret = blkg_conf_open_bdev(&ctx);
if (ret)
return ret;
ret = blkg_conf_prep(blkcg, &blkcg_policy_iocost, &ctx);
if (ret)
goto err;
goto close_bdev;
iocg = blkg_to_iocg(ctx.blkg);
ret = -EINVAL;
if (!strncmp(ctx.body, "default", 7)) {
v = 0;
} else {
if (!sscanf(ctx.body, "%u", &v))
goto einval;
goto unprep;
if (v < CGROUP_WEIGHT_MIN || v > CGROUP_WEIGHT_MAX)
goto einval;
goto unprep;
}
spin_lock(&iocg->ioc->lock);
@@ -3161,14 +3191,15 @@ static ssize_t ioc_weight_write(struct kernfs_open_file *of, char *buf,
weight_updated(iocg, &now);
spin_unlock(&iocg->ioc->lock);
blkg_conf_exit(&ctx);
return nbytes;
ret = 0;
einval:
ret = -EINVAL;
err:
blkg_conf_exit(&ctx);
return ret;
unprep:
blkg_conf_unprep(&ctx);
close_bdev:
blkg_conf_close_bdev(&ctx);
return ret ?: nbytes;
}
static u64 ioc_qos_prfill(struct seq_file *sf, struct blkg_policy_data *pd,
@@ -3226,34 +3257,43 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
size_t nbytes, loff_t off)
{
struct blkg_conf_ctx ctx;
struct request_queue *q;
struct gendisk *disk;
struct ioc *ioc;
u32 qos[NR_QOS_PARAMS];
bool enable, user;
char *body, *p;
unsigned long memflags;
unsigned int memflags;
int ret;
blkg_conf_init(&ctx, input);
memflags = blkg_conf_open_bdev_frozen(&ctx);
if (IS_ERR_VALUE(memflags)) {
ret = memflags;
goto err;
}
ret = blkg_conf_open_bdev(&ctx);
if (ret)
return ret;
/*
* At this point, we haven’t started protecting anything related to QoS,
* so we release q->rq_qos_mutex here, which was first acquired in blkg_
* conf_open_bdev. Later, we re-acquire q->rq_qos_mutex after freezing
* the queue to maintain the correct locking order.
*/
mutex_unlock(&ctx.bdev->bd_queue->rq_qos_mutex);
memflags = blk_mq_freeze_queue(ctx.bdev->bd_queue);
mutex_lock(&ctx.bdev->bd_queue->rq_qos_mutex);
body = ctx.body;
disk = ctx.bdev->bd_disk;
if (!queue_is_mq(disk->queue)) {
ret = -EOPNOTSUPP;
goto err;
goto close_bdev;
}
ioc = q_to_ioc(disk->queue);
if (!ioc) {
ret = blk_iocost_init(disk);
if (ret)
goto err;
goto close_bdev;
ioc = q_to_ioc(disk->queue);
}
@@ -3357,15 +3397,17 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
blk_mq_unquiesce_queue(disk->queue);
blkg_conf_exit_frozen(&ctx, memflags);
return nbytes;
close_bdev:
q = ctx.bdev->bd_queue;
blkg_conf_close_bdev(&ctx);
blk_mq_unfreeze_queue(q, memflags);
return ret ?: nbytes;
einval:
spin_unlock_irq(&ioc->lock);
blk_mq_unquiesce_queue(disk->queue);
ret = -EINVAL;
err:
blkg_conf_exit_frozen(&ctx, memflags);
return ret;
goto close_bdev;
}
static u64 ioc_cost_model_prfill(struct seq_file *sf,
@@ -3430,20 +3472,20 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
ret = blkg_conf_open_bdev(&ctx);
if (ret)
goto err;
return ret;
body = ctx.body;
q = bdev_get_queue(ctx.bdev);
if (!queue_is_mq(q)) {
ret = -EOPNOTSUPP;
goto err;
goto close_bdev;
}
ioc = q_to_ioc(q);
if (!ioc) {
ret = blk_iocost_init(ctx.bdev->bd_disk);
if (ret)
goto err;
goto close_bdev;
ioc = q_to_ioc(q);
}
@@ -3454,6 +3496,8 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
memcpy(u, ioc->params.i_lcoefs, sizeof(u));
user = ioc->user_cost_model;
ret = -EINVAL;
while ((p = strsep(&body, " \t\n"))) {
substring_t args[MAX_OPT_ARGS];
char buf[32];
@@ -3471,20 +3515,20 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
else if (!strcmp(buf, "user"))
user = true;
else
goto einval;
goto unlock;
continue;
case COST_MODEL:
match_strlcpy(buf, &args[0], sizeof(buf));
if (strcmp(buf, "linear"))
goto einval;
goto unlock;
continue;
}
tok = match_token(p, i_lcoef_tokens, args);
if (tok == NR_I_LCOEFS)
goto einval;
goto unlock;
if (match_u64(&args[0], &v))
goto einval;
goto unlock;
u[tok] = v;
user = true;
}
@@ -3496,24 +3540,18 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
ioc->user_cost_model = false;
}
ioc_refresh_params(ioc, true);
ret = 0;
unlock:
spin_unlock_irq(&ioc->lock);
blk_mq_unquiesce_queue(q);
blk_mq_unfreeze_queue(q, memflags);
blkg_conf_exit(&ctx);
return nbytes;
einval:
spin_unlock_irq(&ioc->lock);
blk_mq_unquiesce_queue(q);
blk_mq_unfreeze_queue(q, memflags);
ret = -EINVAL;
err:
blkg_conf_exit(&ctx);
return ret;
close_bdev:
blkg_conf_close_bdev(&ctx);
return ret ?: nbytes;
}
static struct cftype ioc_files[] = {

19
block/blk-iolatency.c
View File

@@ -840,7 +840,7 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf,
ret = blkg_conf_open_bdev(&ctx);
if (ret)
goto out;
return ret;
/*
* blk_iolatency_init() may fail after rq_qos_add() succeeds which can
@@ -850,11 +850,11 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf,
if (!iolat_rq_qos(ctx.bdev->bd_queue))
ret = blk_iolatency_init(ctx.bdev->bd_disk);
if (ret)
goto out;
goto close_bdev;
ret = blkg_conf_prep(blkcg, &blkcg_policy_iolatency, &ctx);
if (ret)
goto out;
goto close_bdev;
iolat = blkg_to_lat(ctx.blkg);
p = ctx.body;
@@ -865,7 +865,7 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf,
char val[21]; /* 18446744073709551616 */
if (sscanf(tok, "%15[^=]=%20s", key, val) != 2)
goto out;
goto unprep;
if (!strcmp(key, "target")) {
u64 v;
@@ -875,9 +875,9 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf,
else if (sscanf(val, "%llu", &v) == 1)
lat_val = v * NSEC_PER_USEC;
else
goto out;
goto unprep;
} else {
goto out;
goto unprep;
}
}
@@ -889,8 +889,11 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf,
if (oldval != iolat->min_lat_nsec)
iolatency_clear_scaling(blkg);
ret = 0;
out:
blkg_conf_exit(&ctx);
unprep:
blkg_conf_unprep(&ctx);
close_bdev:
blkg_conf_close_bdev(&ctx);
return ret ?: nbytes;
}

17
block/blk-merge.c
View File

@@ -122,8 +122,7 @@ struct bio *bio_submit_split_bioset(struct bio *bio, unsigned int split_sectors,
struct bio *split = bio_split(bio, split_sectors, GFP_NOIO, bs);
if (IS_ERR(split)) {
bio->bi_status = errno_to_blk_status(PTR_ERR(split));
bio_endio(bio);
bio_endio_status(bio, errno_to_blk_status(PTR_ERR(split)));
return NULL;
}
@@ -143,8 +142,7 @@ EXPORT_SYMBOL_GPL(bio_submit_split_bioset);
static struct bio *bio_submit_split(struct bio *bio, int split_sectors)
{
if (unlikely(split_sectors < 0)) {
bio->bi_status = errno_to_blk_status(split_sectors);
bio_endio(bio);
bio_endio_status(bio, errno_to_blk_status(split_sectors));
return NULL;
}
@@ -547,7 +545,7 @@ static inline int ll_new_hw_segment(struct request *req, struct bio *bio,
if (!blk_cgroup_mergeable(req, bio))
goto no_merge;
if (blk_integrity_merge_bio(req->q, req, bio) == false)
if (unlikely(!blk_integrity_merge_bio(req->q, req, bio)))
goto no_merge;
/* discard request merge won't add new segment */
@@ -649,7 +647,7 @@ static int ll_merge_requests_fn(struct request_queue *q, struct request *req,
if (!blk_cgroup_mergeable(req, next->bio))
return 0;
if (blk_integrity_merge_rq(q, req, next) == false)
if (unlikely(!blk_integrity_merge_rq(q, req, next)))
return 0;
if (!bio_crypt_ctx_merge_rq(req, next))
@@ -723,8 +721,7 @@ static void blk_account_io_merge_request(struct request *req)
if (req->rq_flags & RQF_IO_STAT) {
part_stat_lock();
part_stat_inc(req->part, merges[op_stat_group(req_op(req))]);
part_stat_local_dec(req->part,
in_flight[op_is_write(req_op(req))]);
bdev_dec_in_flight(req->part, req_op(req));
part_stat_unlock();
}
}
@@ -905,7 +902,7 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio)
if (!blk_cgroup_mergeable(rq, bio))
return false;
if (blk_integrity_merge_bio(rq->q, rq, bio) == false)
if (unlikely(!blk_integrity_merge_bio(rq->q, rq, bio)))
return false;
if (!bio_crypt_rq_ctx_compatible(rq, bio))
return false;
@@ -915,7 +912,7 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio)
return false;
if (rq->bio->bi_ioprio != bio->bi_ioprio)
return false;
if (blk_atomic_write_mergeable_rq_bio(rq, bio) == false)
if (unlikely(!blk_atomic_write_mergeable_rq_bio(rq, bio)))
return false;
return true;

24
block/blk-mq-debugfs.c
View File

@@ -19,8 +19,10 @@ static int queue_poll_stat_show(void *data, struct seq_file *m)
return 0;
}
#define TO_REQUEST_QUEUE(m) ((struct request_queue *)m->private)
static void *queue_requeue_list_start(struct seq_file *m, loff_t *pos)
__acquires(&q->requeue_lock)
__acquires(&TO_REQUEST_QUEUE(m)->requeue_lock)
{
struct request_queue *q = m->private;
@@ -36,13 +38,15 @@ static void *queue_requeue_list_next(struct seq_file *m, void *v, loff_t *pos)
}
static void queue_requeue_list_stop(struct seq_file *m, void *v)
__releases(&q->requeue_lock)
__releases(&TO_REQUEST_QUEUE(m)->requeue_lock)
{
struct request_queue *q = m->private;
spin_unlock_irq(&q->requeue_lock);
}
#undef TO_REQUEST_QUEUE
static const struct seq_operations queue_requeue_list_seq_ops = {
.start = queue_requeue_list_start,
.next = queue_requeue_list_next,
@@ -297,8 +301,10 @@ int blk_mq_debugfs_rq_show(struct seq_file *m, void *v)
}
EXPORT_SYMBOL_GPL(blk_mq_debugfs_rq_show);
#define TO_HCTX(m) ((struct blk_mq_hw_ctx *)m->private)
static void *hctx_dispatch_start(struct seq_file *m, loff_t *pos)
__acquires(&hctx->lock)
__acquires(&TO_HCTX(m)->lock)
{
struct blk_mq_hw_ctx *hctx = m->private;
@@ -314,13 +320,15 @@ static void *hctx_dispatch_next(struct seq_file *m, void *v, loff_t *pos)
}
static void hctx_dispatch_stop(struct seq_file *m, void *v)
__releases(&hctx->lock)
__releases(&TO_HCTX(m)->lock)
{
struct blk_mq_hw_ctx *hctx = m->private;
spin_unlock(&hctx->lock);
}
#undef TO_HCTX
static const struct seq_operations hctx_dispatch_seq_ops = {
.start = hctx_dispatch_start,
.next = hctx_dispatch_next,
@@ -484,9 +492,11 @@ static int hctx_dispatch_busy_show(void *data, struct seq_file *m)
return 0;
}
#define TO_CTX(m) ((struct blk_mq_ctx *)m->private)
#define CTX_RQ_SEQ_OPS(name, type) \
static void *ctx_##name##_rq_list_start(struct seq_file *m, loff_t *pos) \
__acquires(&ctx->lock) \
__acquires(&TO_CTX(m)->lock) \
{ \
struct blk_mq_ctx *ctx = m->private; \
\
@@ -503,7 +513,7 @@ static void *ctx_##name##_rq_list_next(struct seq_file *m, void *v, \
} \
\
static void ctx_##name##_rq_list_stop(struct seq_file *m, void *v) \
__releases(&ctx->lock) \
__releases(&TO_CTX(m)->lock) \
{ \
struct blk_mq_ctx *ctx = m->private; \
\
@@ -521,6 +531,8 @@ CTX_RQ_SEQ_OPS(default, HCTX_TYPE_DEFAULT);
CTX_RQ_SEQ_OPS(read, HCTX_TYPE_READ);
CTX_RQ_SEQ_OPS(poll, HCTX_TYPE_POLL);
#undef TO_CTX
static int blk_mq_debugfs_show(struct seq_file *m, void *v)
{
const struct blk_mq_debugfs_attr *attr = m->private;

6
block/blk-mq-tag.c
View File

@@ -13,6 +13,7 @@
#include <linux/kmemleak.h>
#include <linux/delay.h>
#include <trace/events/block.h>
#include "blk.h"
#include "blk-mq.h"
#include "blk-mq-sched.h"
@@ -181,6 +182,11 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data)
if (tag != BLK_MQ_NO_TAG)
break;
/* Log the starvation event before altering task state */
trace_block_rq_tag_wait(data->q, data->hctx,
data->rq_flags & RQF_SCHED_TAGS,
data->flags);
sbitmap_prepare_to_wait(bt, ws, &wait, TASK_UNINTERRUPTIBLE);
tag = __blk_mq_get_tag(data, bt);

43
block/blk-mq.c
View File

@@ -1082,49 +1082,18 @@ static inline void blk_account_io_done(struct request *req, u64 now)
update_io_ticks(req->part, jiffies, true);
part_stat_inc(req->part, ios[sgrp]);
part_stat_add(req->part, nsecs[sgrp], now - req->start_time_ns);
part_stat_local_dec(req->part,
in_flight[op_is_write(req_op(req))]);
bdev_dec_in_flight(req->part, req_op(req));
part_stat_unlock();
}
}
static inline bool blk_rq_passthrough_stats(struct request *req)
{
struct bio *bio = req->bio;
if (!blk_queue_passthrough_stat(req->q))
return false;
/* Requests without a bio do not transfer data. */
if (!bio)
return false;
/*
* Stats are accumulated in the bdev, so must have one attached to a
* bio to track stats. Most drivers do not set the bdev for passthrough
* requests, but nvme is one that will set it.
*/
if (!bio->bi_bdev)
return false;
/*
* We don't know what a passthrough command does, but we know the
* payload size and data direction. Ensuring the size is aligned to the
* block size filters out most commands with payloads that don't
* represent sector access.
*/
if (blk_rq_bytes(req) & (bdev_logical_block_size(bio->bi_bdev) - 1))
return false;
return true;
}
static inline void blk_account_io_start(struct request *req)
{
trace_block_io_start(req);
if (!blk_queue_io_stat(req->q))
return;
if (blk_rq_is_passthrough(req) && !blk_rq_passthrough_stats(req))
if (blk_rq_is_passthrough(req) && !blk_rq_passthrough_stats(req, req->q))
return;
req->rq_flags |= RQF_IO_STAT;
@@ -1143,7 +1112,7 @@ static inline void blk_account_io_start(struct request *req)
part_stat_lock();
update_io_ticks(req->part, jiffies, false);
part_stat_local_inc(req->part, in_flight[op_is_write(req_op(req))]);
bdev_inc_in_flight(req->part, req_op(req));
part_stat_unlock();
}
@@ -3170,8 +3139,7 @@ void blk_mq_submit_bio(struct bio *bio)
}
if ((bio->bi_opf & REQ_POLLED) && !blk_mq_can_poll(q)) {
bio->bi_status = BLK_STS_NOTSUPP;
bio_endio(bio);
bio_endio_status(bio, BLK_STS_NOTSUPP);
goto queue_exit;
}
@@ -3215,8 +3183,7 @@ void blk_mq_submit_bio(struct bio *bio)
ret = blk_crypto_rq_get_keyslot(rq);
if (ret != BLK_STS_OK) {
bio->bi_status = ret;
bio_endio(bio);
bio_endio_status(bio, ret);
blk_mq_free_request(rq);
return;
}

2
block/blk-settings.c
View File

@@ -795,6 +795,8 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b,
t->features &= ~BLK_FEAT_NOWAIT;
if (!(b->features & BLK_FEAT_POLL))
t->features &= ~BLK_FEAT_POLL;
if (!(b->features & BLK_FEAT_PCI_P2PDMA))
t->features &= ~BLK_FEAT_PCI_P2PDMA;
t->flags |= (b->flags & BLK_FLAG_MISALIGNED);

5
block/blk-sysfs.c
View File

@@ -19,6 +19,7 @@
#include "blk-wbt.h"
#include "blk-cgroup.h"
#include "blk-throttle.h"
#include "error-injection.h"
struct queue_sysfs_entry {
struct attribute attr;
@@ -933,6 +934,8 @@ static void blk_debugfs_remove(struct gendisk *disk)
blk_debugfs_lock_nomemsave(q);
blk_trace_shutdown(q);
if (IS_ENABLED(CONFIG_BLK_ERROR_INJECTION))
blk_error_injection_exit(disk);
debugfs_remove_recursive(q->debugfs_dir);
q->debugfs_dir = NULL;
q->sched_debugfs_dir = NULL;
@@ -963,6 +966,8 @@ int blk_register_queue(struct gendisk *disk)
memflags = blk_debugfs_lock(q);
q->debugfs_dir = debugfs_create_dir(disk->disk_name, blk_debugfs_root);
if (IS_ENABLED(CONFIG_BLK_ERROR_INJECTION))
blk_error_injection_init(disk);
if (queue_is_mq(q))
blk_mq_debugfs_register(q);
blk_debugfs_unlock(q, memflags);

85
block/blk-throttle.c
View File

@@ -1353,21 +1353,21 @@ static ssize_t tg_set_conf(struct kernfs_open_file *of,
ret = blkg_conf_open_bdev(&ctx);
if (ret)
goto out_finish;
return ret;
if (!blk_throtl_activated(ctx.bdev->bd_queue)) {
ret = blk_throtl_init(ctx.bdev->bd_disk);
if (ret)
goto out_finish;
goto close_bdev;
}
ret = blkg_conf_prep(blkcg, &blkcg_policy_throtl, &ctx);
if (ret)
goto out_finish;
goto close_bdev;
ret = -EINVAL;
if (sscanf(ctx.body, "%llu", &v) != 1)
goto out_finish;
goto unprep;
if (!v)
v = U64_MAX;
@@ -1381,8 +1381,12 @@ static ssize_t tg_set_conf(struct kernfs_open_file *of,
tg_conf_updated(tg, false);
ret = 0;
out_finish:
blkg_conf_exit(&ctx);
unprep:
blkg_conf_unprep(&ctx);
close_bdev:
blkg_conf_close_bdev(&ctx);
return ret ?: nbytes;
}
@@ -1537,17 +1541,17 @@ static ssize_t tg_set_limit(struct kernfs_open_file *of,
ret = blkg_conf_open_bdev(&ctx);
if (ret)
goto out_finish;
return ret;
if (!blk_throtl_activated(ctx.bdev->bd_queue)) {
ret = blk_throtl_init(ctx.bdev->bd_disk);
if (ret)
goto out_finish;
goto close_bdev;
}
ret = blkg_conf_prep(blkcg, &blkcg_policy_throtl, &ctx);
if (ret)
goto out_finish;
goto close_bdev;
tg = blkg_to_tg(ctx.blkg);
tg_update_carryover(tg);
@@ -1573,11 +1577,11 @@ static ssize_t tg_set_limit(struct kernfs_open_file *of,
p = tok;
strsep(&p, "=");
if (!p || (sscanf(p, "%llu", &val) != 1 && strcmp(p, "max")))
goto out_finish;
goto unprep;
ret = -ERANGE;
if (!val)
goto out_finish;
goto unprep;
ret = -EINVAL;
if (!strcmp(tok, "rbps"))
@@ -1589,7 +1593,7 @@ static ssize_t tg_set_limit(struct kernfs_open_file *of,
else if (!strcmp(tok, "wiops"))
v[3] = min_t(u64, val, UINT_MAX);
else
goto out_finish;
goto unprep;
}
tg->bps[READ] = v[0];
@@ -1599,8 +1603,10 @@ static ssize_t tg_set_limit(struct kernfs_open_file *of,
tg_conf_updated(tg, false);
ret = 0;
out_finish:
blkg_conf_exit(&ctx);
unprep:
blkg_conf_unprep(&ctx);
close_bdev:
blkg_conf_close_bdev(&ctx);
return ret ?: nbytes;
}
@@ -1649,7 +1655,7 @@ static void tg_flush_bios(struct throtl_grp *tg)
*/
tg_update_disptime(tg);
throtl_schedule_pending_timer(sq, jiffies + 1);
throtl_schedule_next_dispatch(sq->parent_sq, true);
}
static void throtl_pd_offline(struct blkg_policy_data *pd)
@@ -1668,11 +1674,52 @@ struct blkcg_policy blkcg_policy_throtl = {
.pd_free_fn = throtl_pd_free,
};
static void tg_cancel_writeback_bios(struct throtl_grp *tg,
struct bio_list *cancel_bios)
{
struct throtl_service_queue *sq = &tg->service_queue;
struct throtl_data *td = sq_to_td(sq);
int rw;
if (tg->flags & THROTL_TG_CANCELING)
return;
tg->flags |= THROTL_TG_CANCELING;
for (rw = READ; rw <= WRITE; rw++) {
struct throtl_qnode *qn, *tmp;
unsigned int nr_bios = 0;
list_for_each_entry_safe(qn, tmp, &sq->queued[rw], node) {
struct bio *bio;
while ((bio = bio_list_pop(&qn->bios_iops))) {
sq->nr_queued_iops[rw]--;
bio_list_add(&cancel_bios[rw], bio);
nr_bios++;
}
while ((bio = bio_list_pop(&qn->bios_bps))) {
sq->nr_queued_bps[rw]--;
bio_list_add(&cancel_bios[rw], bio);
nr_bios++;
}
list_del_init(&qn->node);
blkg_put(tg_to_blkg(qn->tg));
}
td->nr_queued[rw] -= nr_bios;
}
throtl_dequeue_tg(tg);
}
void blk_throtl_cancel_bios(struct gendisk *disk)
{
struct request_queue *q = disk->queue;
struct cgroup_subsys_state *pos_css;
struct blkcg_gq *blkg;
struct bio_list cancel_bios[2] = { };
int rw;
if (!blk_throtl_activated(q))
return;
@@ -1693,10 +1740,16 @@ void blk_throtl_cancel_bios(struct gendisk *disk)
* Cancel bios here to ensure no bios are inflight after
* del_gendisk.
*/
tg_flush_bios(blkg_to_tg(blkg));
tg_cancel_writeback_bios(blkg_to_tg(blkg), cancel_bios);
}
rcu_read_unlock();
spin_unlock_irq(&q->queue_lock);
for (rw = READ; rw <= WRITE; rw++) {
struct bio *bio;
while ((bio = bio_list_pop(&cancel_bios[rw])))
bio_io_error(bio);
}
}
static bool tg_within_limit(struct throtl_grp *tg, struct bio *bio, bool rw)

2
block/blk-zoned.c
View File

@@ -1924,7 +1924,7 @@ static int disk_alloc_zone_resources(struct gendisk *disk,
goto free_hash;
disk->zone_wplugs_wq =
alloc_workqueue("%s_zwplugs", WQ_MEM_RECLAIM | WQ_HIGHPRI,
alloc_workqueue("%s_zwplugs", WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_PERCPU,
pool_size, disk->disk_name);
if (!disk->zone_wplugs_wq)
goto destroy_pool;

32
block/blk.h
View File

@@ -4,6 +4,7 @@
#include <linux/bio-integrity.h>
#include <linux/blk-crypto.h>
#include <linux/part_stat.h>
#include <linux/lockdep.h>
#include <linux/memblock.h> /* for max_pfn/max_low_pfn */
#include <linux/sched/sysctl.h>
@@ -49,6 +50,11 @@ struct blk_flush_queue *blk_alloc_flush_queue(int node, int cmd_size,
gfp_t flags);
void blk_free_flush_queue(struct blk_flush_queue *q);
const char *blk_status_to_str(blk_status_t status);
const char *blk_status_to_tag(blk_status_t status);
blk_status_t tag_to_blk_status(const char *tag);
enum req_op str_to_blk_op(const char *op);
bool __blk_mq_unfreeze_queue(struct request_queue *q, bool force_atomic);
bool blk_queue_start_drain(struct request_queue *q);
bool __blk_freeze_queue_start(struct request_queue *q,
@@ -402,6 +408,8 @@ static inline bool bio_may_need_split(struct bio *bio,
bv = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter);
if (bio->bi_iter.bi_size > bv->bv_len - bio->bi_iter.bi_bvec_done)
return true;
if ((bv->bv_offset | bv->bv_len) & lim->dma_alignment)
return true;
return bv->bv_len + bv->bv_offset > lim->max_fast_segment_size;
}
@@ -485,6 +493,26 @@ static inline void req_set_nomerge(struct request_queue *q, struct request *req)
q->last_merge = NULL;
}
static inline void bdev_inc_in_flight(struct block_device *bdev,
enum req_op op)
{
bool rw = op_is_write(op);
part_stat_local_inc(bdev, in_flight[rw]);
if (bdev_is_partition(bdev))
part_stat_local_inc(bdev_whole(bdev), in_flight[rw]);
}
static inline void bdev_dec_in_flight(struct block_device *bdev,
enum req_op op)
{
bool rw = op_is_write(op);
part_stat_local_dec(bdev, in_flight[rw]);
if (bdev_is_partition(bdev))
part_stat_local_dec(bdev_whole(bdev), in_flight[rw]);
}
/*
* Internal io_context interface
*/
@@ -754,16 +782,19 @@ static inline void blk_unfreeze_release_lock(struct request_queue *q)
* reclaim from triggering block I/O.
*/
static inline void blk_debugfs_lock_nomemsave(struct request_queue *q)
__acquires(&q->debugfs_mutex)
{
mutex_lock(&q->debugfs_mutex);
}
static inline void blk_debugfs_unlock_nomemrestore(struct request_queue *q)
__releases(&q->debugfs_mutex)
{
mutex_unlock(&q->debugfs_mutex);
}
static inline unsigned int __must_check blk_debugfs_lock(struct request_queue *q)
__acquires(&q->debugfs_mutex)
{
unsigned int memflags = memalloc_noio_save();
@@ -773,6 +804,7 @@ static inline unsigned int __must_check blk_debugfs_lock(struct request_queue *q
static inline void blk_debugfs_unlock(struct request_queue *q,
unsigned int memflags)
__releases(&q->debugfs_mutex)
{
blk_debugfs_unlock_nomemrestore(q);
memalloc_noio_restore(memflags);

2
block/bsg-lib.c
View File

@@ -299,7 +299,7 @@ static blk_status_t bsg_queue_rq(struct blk_mq_hw_ctx *hctx,
/* called right after the request is allocated for the request_queue */
static int bsg_init_rq(struct blk_mq_tag_set *set, struct request *req,
unsigned int hctx_idx, unsigned int numa_node)
unsigned int hctx_idx, int numa_node)
{
struct bsg_job *job = blk_mq_rq_to_pdu(req);

315
block/error-injection.c Normal file
View File

@@ -0,0 +1,315 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (c) 2026 Christoph Hellwig.
*/
#include <linux/debugfs.h>
#include <linux/blkdev.h>
#include <linux/parser.h>
#include <linux/seq_file.h>
#include "blk.h"
#include "error-injection.h"
struct blk_error_inject {
struct list_head entry;
sector_t start;
sector_t end;
enum req_op op;
blk_status_t status;
/* only inject every 1 / chance times */
unsigned int chance;
};
DEFINE_STATIC_KEY_FALSE(blk_error_injection_enabled);
bool __blk_error_inject(struct bio *bio)
{
struct gendisk *disk = bio->bi_bdev->bd_disk;
struct blk_error_inject *inj;
rcu_read_lock();
list_for_each_entry_rcu(inj, &disk->error_injection_list, entry) {
if (bio_op(bio) != inj->op)
continue;
/*
* This never matches 0-sized bios like empty WRITEs with
* REQ_PREFLUSH or ZONE_RESET_ALL. While adding a special case
* for them would be trivial, that means any WRITE rule would
* trigger for flushes. So before we can make this work
* properly, we'll need to start using REQ_OP_FLUSH for pure
* flushes at the bio level like we already do in blk-mq.
*/
if (bio->bi_iter.bi_sector > inj->end ||
bio_end_sector(bio) <= inj->start)
continue;
if (inj->chance > 1 && (get_random_u32() % inj->chance) != 0)
continue;
pr_info_ratelimited("%pg: injecting %s error for %s at sector %llu:%u\n",
disk->part0, blk_status_to_str(inj->status),
blk_op_str(inj->op), bio->bi_iter.bi_sector,
bio_sectors(bio));
bio->bi_status = inj->status;
rcu_read_unlock();
bio_endio(bio);
return true;
}
rcu_read_unlock();
return false;
}
static int error_inject_add(struct gendisk *disk, enum req_op op,
sector_t start, u64 nr_sectors, blk_status_t status,
unsigned int chance)
{
struct blk_error_inject *inj;
int error = -EINVAL;
if (op == REQ_OP_LAST)
return -EINVAL;
if (status == BLK_STS_OK)
return -EINVAL;
inj = kzalloc_obj(*inj);
if (!inj)
return -ENOMEM;
if (nr_sectors) {
if (U64_MAX - nr_sectors < start)
goto out_free_inj;
inj->end = start + nr_sectors - 1;
} else {
inj->end = U64_MAX;
}
inj->op = op;
inj->start = start;
inj->status = status;
inj->chance = chance;
pr_debug_ratelimited("%pg: adding %s injection for %s at sector %llu:%llu\n",
disk->part0, blk_status_to_str(status),
blk_op_str(op),
start, nr_sectors);
/*
* Add to the front of the list so that newer entries can partially
* override other entries. This also intentionally allows duplicate
* entries as there is no real reason to reject them.
*/
mutex_lock(&disk->error_injection_lock);
if (!disk_live(disk)) {
mutex_unlock(&disk->error_injection_lock);
error = -ENODEV;
goto out_free_inj;
}
if (list_empty(&disk->error_injection_list))
static_branch_inc(&blk_error_injection_enabled);
list_add_rcu(&inj->entry, &disk->error_injection_list);
set_bit(GD_ERROR_INJECT, &disk->state);
mutex_unlock(&disk->error_injection_lock);
return 0;
out_free_inj:
kfree(inj);
return error;
}
static void error_inject_removeall(struct gendisk *disk)
{
struct blk_error_inject *inj;
mutex_lock(&disk->error_injection_lock);
clear_bit(GD_ERROR_INJECT, &disk->state);
while ((inj = list_first_entry_or_null(&disk->error_injection_list,
struct blk_error_inject, entry))) {
list_del_rcu(&inj->entry);
kfree_rcu_mightsleep(inj);
}
static_branch_dec(&blk_error_injection_enabled);
mutex_unlock(&disk->error_injection_lock);
}
enum options {
Opt_add = (1u << 0),
Opt_removeall = (1u << 1),
Opt_op = (1u << 16),
Opt_start = (1u << 17),
Opt_nr_sectors = (1u << 18),
Opt_status = (1u << 19),
Opt_chance = (1u << 20),
Opt_invalid,
};
static const match_table_t opt_tokens = {
{ Opt_add, "add", },
{ Opt_removeall, "removeall", },
{ Opt_op, "op=%s", },
{ Opt_start, "start=%u" },
{ Opt_nr_sectors, "nr_sectors=%u" },
{ Opt_status, "status=%s" },
{ Opt_chance, "chance=%u" },
{ Opt_invalid, NULL, },
};
static int match_op(substring_t *args, enum req_op *op)
{
const char *tag;
tag = match_strdup(args);
if (!tag)
return -ENOMEM;
*op = str_to_blk_op(tag);
if (*op == REQ_OP_LAST)
pr_warn("invalid op '%s'\n", tag);
kfree(tag);
return 0;
}
static int match_status(substring_t *args, blk_status_t *status)
{
const char *tag;
tag = match_strdup(args);
if (!tag)
return -ENOMEM;
*status = tag_to_blk_status(tag);
if (!*status)
pr_warn("invalid status '%s'\n", tag);
kfree(tag);
return 0;
}
static ssize_t blk_error_injection_parse_options(struct gendisk *disk,
char *options)
{
enum { Unset, Add, Removeall } action = Unset;
unsigned int option_mask = 0, chance = 1;
enum req_op op = REQ_OP_LAST;
u64 start = 0, nr_sectors = 0;
blk_status_t status = BLK_STS_OK;
substring_t args[MAX_OPT_ARGS];
char *p;
while ((p = strsep(&options, ",\n")) != NULL) {
int error = 0;
ssize_t token;
if (!*p)
continue;
token = match_token(p, opt_tokens, args);
option_mask |= token;
switch (token) {
case Opt_add:
if (action != Unset)
return -EINVAL;
action = Add;
break;
case Opt_removeall:
if (action != Unset)
return -EINVAL;
action = Removeall;
break;
case Opt_op:
error = match_op(args, &op);
break;
case Opt_start:
error = match_u64(args, &start);
break;
case Opt_nr_sectors:
error = match_u64(args, &nr_sectors);
break;
case Opt_status:
error = match_status(args, &status);
break;
case Opt_chance:
error = match_uint(args, &chance);
if (!error && chance == 0)
error = -EINVAL;
break;
default:
pr_warn("unknown parameter or missing value '%s'\n", p);
error = -EINVAL;
}
if (error)
return error;
}
switch (action) {
case Add:
return error_inject_add(disk, op, start, nr_sectors, status,
chance);
case Removeall:
if (option_mask & ~Opt_removeall)
return -EINVAL;
error_inject_removeall(disk);
return 0;
default:
return -EINVAL;
}
}
static ssize_t blk_error_injection_write(struct file *file,
const char __user *ubuf, size_t count, loff_t *pos)
{
struct gendisk *disk = file_inode(file)->i_private;
char *options;
int error;
options = memdup_user_nul(ubuf, count);
if (IS_ERR(options))
return PTR_ERR(options);
error = blk_error_injection_parse_options(disk, options);
kfree(options);
if (error)
return error;
return count;
}
static int blk_error_injection_show(struct seq_file *s, void *private)
{
struct gendisk *disk = s->private;
struct blk_error_inject *inj;
rcu_read_lock();
list_for_each_entry_rcu(inj, &disk->error_injection_list, entry) {
seq_printf(s, "%llu:%llu status=%s,chance=%u",
inj->start, inj->end,
blk_status_to_tag(inj->status), inj->chance);
seq_putc(s, '\n');
}
rcu_read_unlock();
return 0;
}
static int blk_error_injection_open(struct inode *inode, struct file *file)
{
return single_open(file, blk_error_injection_show, inode->i_private);
}
static int blk_error_injection_release(struct inode *inode, struct file *file)
{
return single_release(inode, file);
}
static const struct file_operations blk_error_injection_fops = {
.owner = THIS_MODULE,
.write = blk_error_injection_write,
.read = seq_read,
.open = blk_error_injection_open,
.release = blk_error_injection_release,
};
void blk_error_injection_init(struct gendisk *disk)
{
debugfs_create_file("error_injection", 0600, disk->queue->debugfs_dir,
disk, &blk_error_injection_fops);
}
void blk_error_injection_exit(struct gendisk *disk)
{
error_inject_removeall(disk);
}

21
block/error-injection.h Normal file
View File

@@ -0,0 +1,21 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _BLK_ERROR_INJECTION_H
#define _BLK_ERROR_INJECTION_H 1
#include <linux/jump_label.h>
DECLARE_STATIC_KEY_FALSE(blk_error_injection_enabled);
void blk_error_injection_init(struct gendisk *disk);
void blk_error_injection_exit(struct gendisk *disk);
bool __blk_error_inject(struct bio *bio);
static inline bool blk_error_inject(struct bio *bio)
{
if (IS_ENABLED(CONFIG_BLK_ERROR_INJECTION) &&
static_branch_unlikely(&blk_error_injection_enabled) &&
test_bit(GD_ERROR_INJECT, &bio->bi_bdev->bd_disk->state))
return __blk_error_inject(bio);
return false;
}
#endif /* _BLK_ERROR_INJECTION_H */

27
block/fops.c
View File

@@ -218,8 +218,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
ret = blkdev_iov_iter_get_pages(bio, iter, bdev);
if (unlikely(ret)) {
bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
bio_endio_status(bio, BLK_STS_IOERR);
break;
}
if (iocb->ki_flags & IOCB_NOWAIT) {
@@ -499,36 +498,12 @@ static void blkdev_readahead(struct readahead_control *rac)
mpage_readahead(rac, blkdev_get_block);
}
static int blkdev_write_begin(const struct kiocb *iocb,
struct address_space *mapping, loff_t pos,
unsigned len, struct folio **foliop,
void **fsdata)
{
return block_write_begin(mapping, pos, len, foliop, blkdev_get_block);
}
static int blkdev_write_end(const struct kiocb *iocb,
struct address_space *mapping,
loff_t pos, unsigned len, unsigned copied,
struct folio *folio, void *fsdata)
{
int ret;
ret = block_write_end(pos, len, copied, folio);
folio_unlock(folio);
folio_put(folio);
return ret;
}
const struct address_space_operations def_blk_aops = {
.dirty_folio = block_dirty_folio,
.invalidate_folio = block_invalidate_folio,
.read_folio = blkdev_read_folio,
.readahead = blkdev_readahead,
.writepages = blkdev_writepages,
.write_begin = blkdev_write_begin,
.write_end = blkdev_write_end,
.migrate_folio = buffer_migrate_folio_norefs,
.is_dirty_writeback = buffer_check_dirty_writeback,
};

4
block/genhd.c
View File

@@ -1485,6 +1485,10 @@ struct gendisk *__alloc_disk_node(struct request_queue *q, int node_id,
lockdep_init_map(&disk->lockdep_map, "(bio completion)", lkclass, 0);
#ifdef CONFIG_BLOCK_HOLDER_DEPRECATED
INIT_LIST_HEAD(&disk->slave_bdevs);
#endif
#ifdef CONFIG_BLK_ERROR_INJECTION
mutex_init(&disk->error_injection_lock);
INIT_LIST_HEAD(&disk->error_injection_list);
#endif
mutex_init(&disk->rqos_state_mutex);
kobject_init(&disk->queue_kobj, &blk_queue_ktype);

7
block/kyber-iosched.c
View File

@@ -882,6 +882,9 @@ static const struct elv_fs_entry kyber_sched_attrs[] = {
};
#undef KYBER_LAT_ATTR
#define HCTX_FROM_SEQ_FILE(m) ((struct blk_mq_hw_ctx *)(m)->private)
#define KYBER_HCTX_DATA(hctx) ((struct kyber_hctx_data *)(hctx)->sched_data)
#ifdef CONFIG_BLK_DEBUG_FS
#define KYBER_DEBUGFS_DOMAIN_ATTRS(domain, name) \
static int kyber_##name##_tokens_show(void *data, struct seq_file *m) \
@@ -894,7 +897,7 @@ static int kyber_##name##_tokens_show(void *data, struct seq_file *m) \
} \
\
static void *kyber_##name##_rqs_start(struct seq_file *m, loff_t *pos) \
__acquires(&khd->lock) \
__acquires(&KYBER_HCTX_DATA(HCTX_FROM_SEQ_FILE(m))->lock) \
{ \
struct blk_mq_hw_ctx *hctx = m->private; \
struct kyber_hctx_data *khd = hctx->sched_data; \
@@ -913,7 +916,7 @@ static void *kyber_##name##_rqs_next(struct seq_file *m, void *v, \
} \
\
static void kyber_##name##_rqs_stop(struct seq_file *m, void *v) \
__releases(&khd->lock) \
__releases(&KYBER_HCTX_DATA(HCTX_FROM_SEQ_FILE(m))->lock) \
{ \
struct blk_mq_hw_ctx *hctx = m->private; \
struct kyber_hctx_data *khd = hctx->sched_data; \

12
block/mq-deadline.c
View File

@@ -794,11 +794,15 @@ static const struct elv_fs_entry deadline_attrs[] = {
__ATTR_NULL
};
#define RQ_FROM_SEQ_FILE(m) ((struct request_queue *)(m)->private)
#define DD_DATA_FROM_RQ(rq) \
((struct deadline_data *)(rq)->elevator->elevator_data)
#ifdef CONFIG_BLK_DEBUG_FS
#define DEADLINE_DEBUGFS_DDIR_ATTRS(prio, data_dir, name) \
static void *deadline_##name##_fifo_start(struct seq_file *m, \
loff_t *pos) \
__acquires(&dd->lock) \
__acquires(&DD_DATA_FROM_RQ(RQ_FROM_SEQ_FILE(m))->lock) \
{ \
struct request_queue *q = m->private; \
struct deadline_data *dd = q->elevator->elevator_data; \
@@ -819,7 +823,7 @@ static void *deadline_##name##_fifo_next(struct seq_file *m, void *v, \
} \
\
static void deadline_##name##_fifo_stop(struct seq_file *m, void *v) \
__releases(&dd->lock) \
__releases(&DD_DATA_FROM_RQ(RQ_FROM_SEQ_FILE(m))->lock) \
{ \
struct request_queue *q = m->private; \
struct deadline_data *dd = q->elevator->elevator_data; \
@@ -921,7 +925,7 @@ static int dd_owned_by_driver_show(void *data, struct seq_file *m)
}
static void *deadline_dispatch_start(struct seq_file *m, loff_t *pos)
__acquires(&dd->lock)
__acquires(&DD_DATA_FROM_RQ(RQ_FROM_SEQ_FILE(m))->lock)
{
struct request_queue *q = m->private;
struct deadline_data *dd = q->elevator->elevator_data;
@@ -939,7 +943,7 @@ static void *deadline_dispatch_next(struct seq_file *m, void *v, loff_t *pos)
}
static void deadline_dispatch_stop(struct seq_file *m, void *v)
__releases(&dd->lock)
__releases(&DD_DATA_FROM_RQ(RQ_FROM_SEQ_FILE(m))->lock)
{
struct request_queue *q = m->private;
struct deadline_data *dd = q->elevator->elevator_data;

5
block/partitions/acorn.c
View File

@@ -9,6 +9,7 @@
*/
#include <linux/buffer_head.h>
#include <linux/adfs_fs.h>
#include <linux/minmax.h>
#include "check.h"
@@ -80,7 +81,7 @@ static int riscix_partition(struct parsed_partitions *state,
if (rr->magic == RISCIX_MAGIC) {
unsigned long size = nr_sects > 2 ? 2 : nr_sects;
unsigned long size = min(nr_sects, 2);
int part;
seq_buf_puts(&state->pp_buf, " <");
@@ -124,7 +125,7 @@ static int linux_partition(struct parsed_partitions *state,
{
Sector sect;
struct linux_part *linuxp;
unsigned long size = nr_sects > 2 ? 2 : nr_sects;
unsigned long size = min(nr_sects, 2);
seq_buf_puts(&state->pp_buf, " [Linux]");

9
block/partitions/aix.c
View File

@@ -226,6 +226,15 @@ int aix_partition(struct parsed_partitions *state)
int next_lp_ix = 1;
int lp_ix;
/*
* pvd was read into a fixed-size struct pvd whose ppe[] array
* holds ARRAY_SIZE(pvd->ppe) entries. pp_count is an
* unvalidated on-disk __be16, so clamp the scan to the array
* size to avoid walking past the allocation.
*/
if (numpps > ARRAY_SIZE(pvd->ppe))
numpps = ARRAY_SIZE(pvd->ppe);
for (i = 0; i < numpps; i += 1) {
struct ppe *p = pvd->ppe + i;
unsigned int lv_ix;

6
block/partitions/core.c
View File

@@ -124,7 +124,7 @@ static struct parsed_partitions *check_partition(struct gendisk *hd)
state = allocate_partitions(hd);
if (!state)
return NULL;
state->pp_buf.buffer = (char *)__get_free_page(GFP_KERNEL);
state->pp_buf.buffer = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!state->pp_buf.buffer) {
free_partitions(state);
return NULL;
@@ -154,7 +154,7 @@ static struct parsed_partitions *check_partition(struct gendisk *hd)
if (res > 0) {
printk(KERN_INFO "%s", seq_buf_str(&state->pp_buf));
free_page((unsigned long)state->pp_buf.buffer);
kfree(state->pp_buf.buffer);
return state;
}
if (state->access_beyond_eod)
@@ -170,7 +170,7 @@ static struct parsed_partitions *check_partition(struct gendisk *hd)
printk(KERN_INFO "%s", seq_buf_str(&state->pp_buf));
}
free_page((unsigned long)state->pp_buf.buffer);
kfree(state->pp_buf.buffer);
free_partitions(state);
return ERR_PTR(res);
}

5
block/partitions/of.c
View File

@@ -74,8 +74,10 @@ int of_partition(struct parsed_partitions *state)
struct device_node *partitions_np = of_node_get(ddev->of_node);
if (!partitions_np ||
!of_device_is_compatible(partitions_np, "fixed-partitions"))
!of_device_is_compatible(partitions_np, "fixed-partitions")) {
of_node_put(partitions_np);
return 0;
}
slot = 1;
/* Validate parition offset and size */
@@ -104,5 +106,6 @@ int of_partition(struct parsed_partitions *state)
seq_buf_puts(&state->pp_buf, "\n");
of_node_put(partitions_np);
return 1;
}

1
drivers/block/drbd/Makefile
View File

@@ -3,6 +3,7 @@ drbd-y := drbd_buildtag.o drbd_bitmap.o drbd_proc.o
drbd-y += drbd_worker.o drbd_receiver.o drbd_req.o drbd_actlog.o
drbd-y += drbd_main.o drbd_strings.o drbd_nl.o
drbd-y += drbd_interval.o drbd_state.o
drbd-y += drbd_nl_gen.o
drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o
obj-$(CONFIG_BLK_DEV_DRBD) += drbd.o

2
drivers/block/drbd/drbd_buildtag.c
View File

@@ -1,5 +1,5 @@
// SPDX-License-Identifier: GPL-2.0-only
#include <linux/drbd_config.h>
#include "drbd_config.h"
#include <linux/module.h>
const char *drbd_buildtag(void)

0
include/linux/drbd_config.h → drivers/block/drbd/drbd_config.h
View File

2
drivers/block/drbd/drbd_debugfs.c
View File

@@ -844,7 +844,7 @@ static int drbd_version_show(struct seq_file *m, void *ignored)
{
seq_printf(m, "# %s\n", drbd_buildtag());
seq_printf(m, "VERSION=%s\n", REL_VERSION);
seq_printf(m, "API_VERSION=%u\n", GENL_MAGIC_VERSION);
seq_printf(m, "API_VERSION=%u\n", DRBD_FAMILY_VERSION);
seq_printf(m, "PRO_VERSION_MIN=%u\n", PRO_VERSION_MIN);
seq_printf(m, "PRO_VERSION_MAX=%u\n", PRO_VERSION_MAX);
return 0;

6
drivers/block/drbd/drbd_int.h
View File

@@ -32,14 +32,16 @@
#include <net/tcp.h>
#include <linux/lru_cache.h>
#include <linux/prefetch.h>
#include <linux/drbd_genl_api.h>
#include <linux/drbd.h>
#include <linux/drbd_config.h>
#include "drbd_config.h"
#include "drbd_nl_gen.h"
#include "drbd_strings.h"
#include "drbd_state.h"
#include "drbd_protocol.h"
#include "drbd_polymorph_printk.h"
extern struct genl_family drbd_nl_family;
/* shared module parameters, defined in drbd_main.c */
#ifdef CONFIG_DRBD_FAULT_INJECTION
extern int drbd_enable_faults;

6
drivers/block/drbd/drbd_main.c
View File

@@ -2324,7 +2324,7 @@ static void drbd_cleanup(void)
if (retry.wq)
destroy_workqueue(retry.wq);
drbd_genl_unregister();
genl_unregister_family(&drbd_nl_family);
idr_for_each_entry(&drbd_devices, device, i)
drbd_delete_device(device);
@@ -2846,7 +2846,7 @@ static int __init drbd_init(void)
mutex_init(&resources_mutex);
INIT_LIST_HEAD(&drbd_resources);
err = drbd_genl_register();
err = genl_register_family(&drbd_nl_family);
if (err) {
pr_err("unable to register generic netlink family\n");
goto fail;
@@ -2876,7 +2876,7 @@ static int __init drbd_init(void)
pr_info("initialized. "
"Version: " REL_VERSION " (api:%d/proto:%d-%d)\n",
GENL_MAGIC_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX);
DRBD_FAMILY_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX);
pr_info("%s\n", drbd_buildtag());
pr_info("registered as block device major %d\n", DRBD_MAJOR);
return 0; /* Success! */

414
drivers/block/drbd/drbd_nl.c
View File

@@ -31,59 +31,13 @@
#include <net/genetlink.h>
/* .doit */
// int drbd_adm_create_resource(struct sk_buff *skb, struct genl_info *info);
// int drbd_adm_delete_resource(struct sk_buff *skb, struct genl_info *info);
#include "drbd_nl_gen.h"
int drbd_adm_new_minor(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_del_minor(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_new_resource(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_del_resource(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_down(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_set_role(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_attach(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_disk_opts(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_detach(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_connect(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_net_opts(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_resize(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_start_ov(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_new_c_uuid(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_disconnect(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_invalidate(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_invalidate_peer(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_pause_sync(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_resume_sync(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_suspend_io(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_resume_io(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_outdate(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_resource_opts(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_get_status(struct sk_buff *skb, struct genl_info *info);
int drbd_adm_get_timeout_type(struct sk_buff *skb, struct genl_info *info);
/* .dumpit */
int drbd_adm_get_status_all(struct sk_buff *skb, struct netlink_callback *cb);
int drbd_adm_dump_resources(struct sk_buff *skb, struct netlink_callback *cb);
int drbd_adm_dump_devices(struct sk_buff *skb, struct netlink_callback *cb);
int drbd_adm_dump_devices_done(struct netlink_callback *cb);
int drbd_adm_dump_connections(struct sk_buff *skb, struct netlink_callback *cb);
int drbd_adm_dump_connections_done(struct netlink_callback *cb);
int drbd_adm_dump_peer_devices(struct sk_buff *skb, struct netlink_callback *cb);
int drbd_adm_dump_peer_devices_done(struct netlink_callback *cb);
int drbd_adm_get_initial_state(struct sk_buff *skb, struct netlink_callback *cb);
#include <linux/drbd_genl_api.h>
static int drbd_pre_doit(const struct genl_split_ops *ops,
struct sk_buff *skb, struct genl_info *info);
static void drbd_post_doit(const struct genl_split_ops *ops,
struct sk_buff *skb, struct genl_info *info);
#define GENL_MAGIC_FAMILY_PRE_DOIT drbd_pre_doit
#define GENL_MAGIC_FAMILY_POST_DOIT drbd_post_doit
#include <linux/genl_magic_func.h>
static int drbd_genl_multicast_events(struct sk_buff *skb, gfp_t flags)
{
return genlmsg_multicast(&drbd_nl_family, skb, 0,
DRBD_NLGRP_EVENTS, flags);
}
static atomic_t drbd_genl_seq = ATOMIC_INIT(2); /* two. */
static atomic_t notify_genl_seq = ATOMIC_INIT(2); /* two. */
@@ -114,7 +68,7 @@ static int drbd_msg_put_info(struct sk_buff *skb, const char *info)
if (!nla)
return err;
err = nla_put_string(skb, T_info_text, info);
err = nla_put_string(skb, DRBD_A_DRBD_CFG_REPLY_INFO_TEXT, info);
if (err) {
nla_nest_cancel(skb, nla);
return err;
@@ -135,7 +89,7 @@ static int drbd_msg_sprintf_info(struct sk_buff *skb, const char *fmt, ...)
if (!nla)
return err;
txt = nla_reserve(skb, T_info_text, 256);
txt = nla_reserve(skb, DRBD_A_DRBD_CFG_REPLY_INFO_TEXT, 256);
if (!txt) {
nla_nest_cancel(skb, nla);
return err;
@@ -187,6 +141,15 @@ static const unsigned int drbd_genl_cmd_flags[] = {
[DRBD_ADM_DOWN] = DRBD_ADM_NEED_RESOURCE,
};
/* Detect attempts to change invariant attributes in a _change_ handler. */
#define has_invariant(ntb, attr) \
({ \
bool __found = !!(ntb)[attr]; \
if (__found) \
pr_info("must not change invariant attr: %s\n", #attr); \
__found; \
})
/*
* At this point, we still rely on the global genl_lock().
* If we want to avoid that, and allow "genl_family.parallel_ops", we may need
@@ -210,7 +173,7 @@ static int drbd_adm_prepare(struct drbd_config_context *adm_ctx,
}
adm_ctx->reply_dh = genlmsg_put_reply(adm_ctx->reply_skb,
info, &drbd_genl_family, 0, cmd);
info, &drbd_nl_family, 0, cmd);
/* put of a few bytes into a fresh skb of >= 4k will always succeed.
* but anyways */
if (!adm_ctx->reply_dh) {
@@ -223,9 +186,11 @@ static int drbd_adm_prepare(struct drbd_config_context *adm_ctx,
adm_ctx->volume = VOLUME_UNSPECIFIED;
if (info->attrs[DRBD_NLA_CFG_CONTEXT]) {
struct nlattr **ntb;
struct nlattr *nla;
/* parse and validate only */
err = drbd_cfg_context_from_attrs(NULL, info);
/* parse and validate, get nested attribute table */
err = drbd_cfg_context_ntb_from_attrs(&ntb, info);
if (err)
goto fail;
@@ -234,18 +199,21 @@ static int drbd_adm_prepare(struct drbd_config_context *adm_ctx,
err = nla_put_nohdr(adm_ctx->reply_skb,
info->attrs[DRBD_NLA_CFG_CONTEXT]->nla_len,
info->attrs[DRBD_NLA_CFG_CONTEXT]);
if (err)
if (err) {
kfree(ntb);
goto fail;
}
/* and assign stuff to the adm_ctx */
nla = nested_attr_tb[T_ctx_volume];
nla = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME];
if (nla)
adm_ctx->volume = nla_get_u32(nla);
nla = nested_attr_tb[T_ctx_resource_name];
nla = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME];
if (nla)
adm_ctx->resource_name = nla_data(nla);
adm_ctx->my_addr = nested_attr_tb[T_ctx_my_addr];
adm_ctx->peer_addr = nested_attr_tb[T_ctx_peer_addr];
adm_ctx->my_addr = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR];
adm_ctx->peer_addr = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR];
kfree(ntb);
if ((adm_ctx->my_addr &&
nla_len(adm_ctx->my_addr) > sizeof(adm_ctx->connection->my_addr)) ||
(adm_ctx->peer_addr &&
@@ -259,7 +227,7 @@ static int drbd_adm_prepare(struct drbd_config_context *adm_ctx,
adm_ctx->device = minor_to_device(d_in->minor);
/* We are protected by the global genl_lock().
* But we may explicitly drop it/retake it in drbd_adm_set_role(),
* But we may explicitly drop it/retake it in drbd_nl_set_role(),
* so make sure this object stays around. */
if (adm_ctx->device)
kref_get(&adm_ctx->device->kref);
@@ -334,8 +302,8 @@ static int drbd_adm_prepare(struct drbd_config_context *adm_ctx,
return err;
}
static int drbd_pre_doit(const struct genl_split_ops *ops,
struct sk_buff *skb, struct genl_info *info)
int drbd_pre_doit(const struct genl_split_ops *ops,
struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx;
u8 cmd = info->genlhdr->cmd;
@@ -362,8 +330,8 @@ static int drbd_pre_doit(const struct genl_split_ops *ops,
return 0;
}
static void drbd_post_doit(const struct genl_split_ops *ops,
struct sk_buff *skb, struct genl_info *info)
void drbd_post_doit(const struct genl_split_ops *ops,
struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
@@ -828,7 +796,7 @@ static const char *from_attrs_err_to_txt(int err)
"invalid attribute value";
}
int drbd_adm_set_role(struct sk_buff *skb, struct genl_info *info)
static int drbd_nl_set_role(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct set_role_parms parms;
@@ -868,6 +836,16 @@ int drbd_adm_set_role(struct sk_buff *skb, struct genl_info *info)
return 0;
}
int drbd_nl_primary_doit(struct sk_buff *skb, struct genl_info *info)
{
return drbd_nl_set_role(skb, info);
}
int drbd_nl_secondary_doit(struct sk_buff *skb, struct genl_info *info)
{
return drbd_nl_set_role(skb, info);
}
/* Initializes the md.*_offset members, so we are able to find
* the on disk meta data.
*
@@ -962,7 +940,7 @@ char *ppsize(char *buf, unsigned long long size)
* peer may not initiate a resize.
*/
/* Note these are not to be confused with
* drbd_adm_suspend_io/drbd_adm_resume_io,
* drbd_nl_suspend_io_doit/drbd_nl_resume_io_doit,
* which are (sub) state changes triggered by admin (drbdsetup),
* and can be long lived.
* This changes an device->flag, is triggered by drbd internals,
@@ -1574,13 +1552,14 @@ static int disk_opts_check_al_size(struct drbd_device *device, struct disk_conf
return err;
}
int drbd_adm_disk_opts(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_chg_disk_opts_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
enum drbd_ret_code retcode;
struct drbd_device *device;
struct disk_conf *new_disk_conf, *old_disk_conf;
struct fifo_buffer *old_plan = NULL, *new_plan = NULL;
struct nlattr **ntb;
int err;
unsigned int fifo_size;
@@ -1612,13 +1591,29 @@ int drbd_adm_disk_opts(struct sk_buff *skb, struct genl_info *info)
if (should_set_defaults(info))
set_disk_conf_defaults(new_disk_conf);
err = disk_conf_from_attrs_for_change(new_disk_conf, info);
err = disk_conf_from_attrs(new_disk_conf, info);
if (err && err != -ENOMSG) {
retcode = ERR_MANDATORY_TAG;
drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
goto fail_unlock;
}
err = disk_conf_ntb_from_attrs(&ntb, info);
if (!err) {
if (has_invariant(ntb, DRBD_A_DISK_CONF_BACKING_DEV) ||
has_invariant(ntb, DRBD_A_DISK_CONF_META_DEV) ||
has_invariant(ntb, DRBD_A_DISK_CONF_META_DEV_IDX) ||
has_invariant(ntb, DRBD_A_DISK_CONF_DISK_SIZE) ||
has_invariant(ntb, DRBD_A_DISK_CONF_MAX_BIO_BVECS)) {
retcode = ERR_MANDATORY_TAG;
drbd_msg_put_info(adm_ctx->reply_skb,
"cannot change invariant setting");
kfree(ntb);
goto fail_unlock;
}
kfree(ntb);
}
if (!expect(device, new_disk_conf->resync_rate >= 1))
new_disk_conf->resync_rate = 1;
@@ -1796,7 +1791,7 @@ void drbd_backing_dev_free(struct drbd_device *device, struct drbd_backing_dev *
kfree(ldev);
}
int drbd_adm_attach(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_attach_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_device *device;
@@ -2236,7 +2231,7 @@ static int adm_detach(struct drbd_device *device, int force)
* Then we transition to D_DISKLESS, and wait for put_ldev() to return all
* internal references as well.
* Only then we have finally detached. */
int drbd_adm_detach(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_detach_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
enum drbd_ret_code retcode;
@@ -2434,12 +2429,13 @@ static void free_crypto(struct crypto *crypto)
crypto_free_shash(crypto->verify_tfm);
}
int drbd_adm_net_opts(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_chg_net_opts_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
enum drbd_ret_code retcode;
struct drbd_connection *connection;
struct net_conf *old_net_conf, *new_net_conf = NULL;
struct nlattr **ntb;
int err;
int ovr; /* online verify running */
int rsr; /* re-sync running */
@@ -2476,13 +2472,26 @@ int drbd_adm_net_opts(struct sk_buff *skb, struct genl_info *info)
if (should_set_defaults(info))
set_net_conf_defaults(new_net_conf);
err = net_conf_from_attrs_for_change(new_net_conf, info);
err = net_conf_from_attrs(new_net_conf, info);
if (err && err != -ENOMSG) {
retcode = ERR_MANDATORY_TAG;
drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err));
goto fail;
}
err = net_conf_ntb_from_attrs(&ntb, info);
if (!err) {
if (has_invariant(ntb, DRBD_A_NET_CONF_DISCARD_MY_DATA) ||
has_invariant(ntb, DRBD_A_NET_CONF_TENTATIVE)) {
retcode = ERR_MANDATORY_TAG;
drbd_msg_put_info(adm_ctx->reply_skb,
"cannot change invariant setting");
kfree(ntb);
goto fail;
}
kfree(ntb);
}
retcode = check_net_options(connection, new_net_conf);
if (retcode != NO_ERROR)
goto fail;
@@ -2575,7 +2584,7 @@ static void peer_device_to_info(struct peer_device_info *info,
info->peer_resync_susp_dependency = device->state.aftr_isp;
}
int drbd_adm_connect(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_connect_doit(struct sk_buff *skb, struct genl_info *info)
{
struct connection_info connection_info;
enum drbd_notification_type flags;
@@ -2790,7 +2799,7 @@ static enum drbd_state_rv conn_try_disconnect(struct drbd_connection *connection
return rv;
}
int drbd_adm_disconnect(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_disconnect_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct disconnect_parms parms;
@@ -2845,7 +2854,7 @@ void resync_after_online_grow(struct drbd_device *device)
_drbd_request_state(device, NS(conn, C_WF_SYNC_UUID), CS_VERBOSE + CS_SERIALIZE);
}
int drbd_adm_resize(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_resize_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct disk_conf *old_disk_conf, *new_disk_conf = NULL;
@@ -2981,7 +2990,7 @@ int drbd_adm_resize(struct sk_buff *skb, struct genl_info *info)
goto fail;
}
int drbd_adm_resource_opts(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_resource_opts_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
enum drbd_ret_code retcode;
@@ -3019,7 +3028,7 @@ int drbd_adm_resource_opts(struct sk_buff *skb, struct genl_info *info)
return 0;
}
int drbd_adm_invalidate(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_invalidate_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_device *device;
@@ -3097,7 +3106,7 @@ static int drbd_bmio_set_susp_al(struct drbd_device *device,
return rv;
}
int drbd_adm_invalidate_peer(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_inval_peer_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
int retcode; /* drbd_ret_code, drbd_state_rv */
@@ -3148,7 +3157,7 @@ int drbd_adm_invalidate_peer(struct sk_buff *skb, struct genl_info *info)
return 0;
}
int drbd_adm_pause_sync(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_pause_sync_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
enum drbd_ret_code retcode;
@@ -3168,7 +3177,7 @@ int drbd_adm_pause_sync(struct sk_buff *skb, struct genl_info *info)
return 0;
}
int drbd_adm_resume_sync(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_resume_sync_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
union drbd_dev_state s;
@@ -3196,12 +3205,12 @@ int drbd_adm_resume_sync(struct sk_buff *skb, struct genl_info *info)
return 0;
}
int drbd_adm_suspend_io(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_suspend_io_doit(struct sk_buff *skb, struct genl_info *info)
{
return drbd_adm_simple_request_state(skb, info, NS(susp, 1));
}
int drbd_adm_resume_io(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_resume_io_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_device *device;
@@ -3257,7 +3266,7 @@ int drbd_adm_resume_io(struct sk_buff *skb, struct genl_info *info)
return 0;
}
int drbd_adm_outdate(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_outdate_doit(struct sk_buff *skb, struct genl_info *info)
{
return drbd_adm_simple_request_state(skb, info, NS(disk, D_OUTDATED));
}
@@ -3272,16 +3281,20 @@ static int nla_put_drbd_cfg_context(struct sk_buff *skb,
if (!nla)
goto nla_put_failure;
if (device &&
nla_put_u32(skb, T_ctx_volume, device->vnr))
nla_put_u32(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME, device->vnr))
goto nla_put_failure;
if (nla_put_string(skb, T_ctx_resource_name, resource->name))
if (nla_put_string(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME, resource->name))
goto nla_put_failure;
if (connection) {
if (connection->my_addr_len &&
nla_put(skb, T_ctx_my_addr, connection->my_addr_len, &connection->my_addr))
nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR,
connection->my_addr_len,
&connection->my_addr))
goto nla_put_failure;
if (connection->peer_addr_len &&
nla_put(skb, T_ctx_peer_addr, connection->peer_addr_len, &connection->peer_addr))
nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR,
connection->peer_addr_len,
&connection->peer_addr))
goto nla_put_failure;
}
nla_nest_end(skb, nla);
@@ -3300,7 +3313,7 @@ static int nla_put_drbd_cfg_context(struct sk_buff *skb,
*/
static struct nlattr *find_cfg_context_attr(const struct nlmsghdr *nlh, int attr)
{
const unsigned hdrlen = GENL_HDRLEN + GENL_MAGIC_FAMILY_HDRSZ;
const unsigned int hdrlen = GENL_HDRLEN + sizeof(struct drbd_genlmsghdr);
struct nlattr *nla;
nla = nla_find(nlmsg_attrdata(nlh, hdrlen), nlmsg_attrlen(nlh, hdrlen),
@@ -3312,7 +3325,7 @@ static struct nlattr *find_cfg_context_attr(const struct nlmsghdr *nlh, int attr
static void resource_to_info(struct resource_info *, struct drbd_resource *);
int drbd_adm_dump_resources(struct sk_buff *skb, struct netlink_callback *cb)
int drbd_nl_get_resources_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
{
struct drbd_genlmsghdr *dh;
struct drbd_resource *resource;
@@ -3340,7 +3353,7 @@ int drbd_adm_dump_resources(struct sk_buff *skb, struct netlink_callback *cb)
put_result:
dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
cb->nlh->nlmsg_seq, &drbd_genl_family,
cb->nlh->nlmsg_seq, &drbd_nl_family,
NLM_F_MULTI, DRBD_ADM_GET_RESOURCES);
err = -ENOMEM;
if (!dh)
@@ -3350,15 +3363,15 @@ int drbd_adm_dump_resources(struct sk_buff *skb, struct netlink_callback *cb)
err = nla_put_drbd_cfg_context(skb, resource, NULL, NULL);
if (err)
goto out;
err = res_opts_to_skb(skb, &resource->res_opts, !capable(CAP_SYS_ADMIN));
err = res_opts_to_skb(skb, &resource->res_opts);
if (err)
goto out;
resource_to_info(&resource_info, resource);
err = resource_info_to_skb(skb, &resource_info, !capable(CAP_SYS_ADMIN));
err = resource_info_to_skb(skb, &resource_info);
if (err)
goto out;
resource_statistics.res_stat_write_ordering = resource->write_ordering;
err = resource_statistics_to_skb(skb, &resource_statistics, !capable(CAP_SYS_ADMIN));
err = resource_statistics_to_skb(skb, &resource_statistics);
if (err)
goto out;
cb->args[0] = (long)resource;
@@ -3423,7 +3436,7 @@ int drbd_adm_dump_devices_done(struct netlink_callback *cb) {
static void device_to_info(struct device_info *, struct drbd_device *);
int drbd_adm_dump_devices(struct sk_buff *skb, struct netlink_callback *cb)
int drbd_nl_get_devices_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
{
struct nlattr *resource_filter;
struct drbd_resource *resource;
@@ -3436,7 +3449,8 @@ int drbd_adm_dump_devices(struct sk_buff *skb, struct netlink_callback *cb)
resource = (struct drbd_resource *)cb->args[0];
if (!cb->args[0] && !cb->args[1]) {
resource_filter = find_cfg_context_attr(cb->nlh, T_ctx_resource_name);
resource_filter = find_cfg_context_attr(cb->nlh,
DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME);
if (resource_filter) {
retcode = ERR_RES_NOT_KNOWN;
resource = drbd_find_resource(nla_data(resource_filter));
@@ -3465,7 +3479,7 @@ int drbd_adm_dump_devices(struct sk_buff *skb, struct netlink_callback *cb)
put_result:
dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
cb->nlh->nlmsg_seq, &drbd_genl_family,
cb->nlh->nlmsg_seq, &drbd_nl_family,
NLM_F_MULTI, DRBD_ADM_GET_DEVICES);
err = -ENOMEM;
if (!dh)
@@ -3481,18 +3495,18 @@ int drbd_adm_dump_devices(struct sk_buff *skb, struct netlink_callback *cb)
struct disk_conf *disk_conf =
rcu_dereference(device->ldev->disk_conf);
err = disk_conf_to_skb(skb, disk_conf, !capable(CAP_SYS_ADMIN));
err = disk_conf_to_skb(skb, disk_conf);
put_ldev(device);
if (err)
goto out;
}
device_to_info(&device_info, device);
err = device_info_to_skb(skb, &device_info, !capable(CAP_SYS_ADMIN));
err = device_info_to_skb(skb, &device_info);
if (err)
goto out;
device_to_statistics(&device_statistics, device);
err = device_statistics_to_skb(skb, &device_statistics, !capable(CAP_SYS_ADMIN));
err = device_statistics_to_skb(skb, &device_statistics);
if (err)
goto out;
cb->args[1] = minor + 1;
@@ -3514,7 +3528,7 @@ int drbd_adm_dump_connections_done(struct netlink_callback *cb)
enum { SINGLE_RESOURCE, ITERATE_RESOURCES };
int drbd_adm_dump_connections(struct sk_buff *skb, struct netlink_callback *cb)
int drbd_nl_get_connections_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
{
struct nlattr *resource_filter;
struct drbd_resource *resource = NULL, *next_resource;
@@ -3527,7 +3541,8 @@ int drbd_adm_dump_connections(struct sk_buff *skb, struct netlink_callback *cb)
rcu_read_lock();
resource = (struct drbd_resource *)cb->args[0];
if (!cb->args[0]) {
resource_filter = find_cfg_context_attr(cb->nlh, T_ctx_resource_name);
resource_filter = find_cfg_context_attr(cb->nlh,
DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME);
if (resource_filter) {
retcode = ERR_RES_NOT_KNOWN;
resource = drbd_find_resource(nla_data(resource_filter));
@@ -3591,7 +3606,7 @@ int drbd_adm_dump_connections(struct sk_buff *skb, struct netlink_callback *cb)
put_result:
dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
cb->nlh->nlmsg_seq, &drbd_genl_family,
cb->nlh->nlmsg_seq, &drbd_nl_family,
NLM_F_MULTI, DRBD_ADM_GET_CONNECTIONS);
err = -ENOMEM;
if (!dh)
@@ -3606,16 +3621,16 @@ int drbd_adm_dump_connections(struct sk_buff *skb, struct netlink_callback *cb)
goto out;
net_conf = rcu_dereference(connection->net_conf);
if (net_conf) {
err = net_conf_to_skb(skb, net_conf, !capable(CAP_SYS_ADMIN));
err = net_conf_to_skb(skb, net_conf);
if (err)
goto out;
}
connection_to_info(&connection_info, connection);
err = connection_info_to_skb(skb, &connection_info, !capable(CAP_SYS_ADMIN));
err = connection_info_to_skb(skb, &connection_info);
if (err)
goto out;
connection_statistics.conn_congested = test_bit(NET_CONGESTED, &connection->flags);
err = connection_statistics_to_skb(skb, &connection_statistics, !capable(CAP_SYS_ADMIN));
err = connection_statistics_to_skb(skb, &connection_statistics);
if (err)
goto out;
cb->args[2] = (long)connection;
@@ -3676,7 +3691,7 @@ int drbd_adm_dump_peer_devices_done(struct netlink_callback *cb)
return put_resource_in_arg0(cb, 9);
}
int drbd_adm_dump_peer_devices(struct sk_buff *skb, struct netlink_callback *cb)
int drbd_nl_get_peer_devices_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
{
struct nlattr *resource_filter;
struct drbd_resource *resource;
@@ -3688,7 +3703,8 @@ int drbd_adm_dump_peer_devices(struct sk_buff *skb, struct netlink_callback *cb)
resource = (struct drbd_resource *)cb->args[0];
if (!cb->args[0] && !cb->args[1]) {
resource_filter = find_cfg_context_attr(cb->nlh, T_ctx_resource_name);
resource_filter = find_cfg_context_attr(cb->nlh,
DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME);
if (resource_filter) {
retcode = ERR_RES_NOT_KNOWN;
resource = drbd_find_resource(nla_data(resource_filter));
@@ -3735,7 +3751,7 @@ int drbd_adm_dump_peer_devices(struct sk_buff *skb, struct netlink_callback *cb)
put_result:
dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
cb->nlh->nlmsg_seq, &drbd_genl_family,
cb->nlh->nlmsg_seq, &drbd_nl_family,
NLM_F_MULTI, DRBD_ADM_GET_PEER_DEVICES);
err = -ENOMEM;
if (!dh)
@@ -3751,11 +3767,11 @@ int drbd_adm_dump_peer_devices(struct sk_buff *skb, struct netlink_callback *cb)
if (err)
goto out;
peer_device_to_info(&peer_device_info, peer_device);
err = peer_device_info_to_skb(skb, &peer_device_info, !capable(CAP_SYS_ADMIN));
err = peer_device_info_to_skb(skb, &peer_device_info);
if (err)
goto out;
peer_device_to_statistics(&peer_device_statistics, peer_device);
err = peer_device_statistics_to_skb(skb, &peer_device_statistics, !capable(CAP_SYS_ADMIN));
err = peer_device_statistics_to_skb(skb, &peer_device_statistics);
if (err)
goto out;
cb->args[1] = minor;
@@ -3795,11 +3811,11 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device,
/* If sib != NULL, this is drbd_bcast_event, which anyone can listen
* to. So we better exclude_sensitive information.
*
* If sib == NULL, this is drbd_adm_get_status, executed synchronously
* If sib == NULL, this is drbd_nl_get_status_doit, executed synchronously
* in the context of the requesting user process. Exclude sensitive
* information, unless current has superuser.
*
* NOTE: for drbd_adm_get_status_all(), this is a netlink dump, and
* NOTE: for drbd_nl_get_status_dumpit(), this is a netlink dump, and
* relies on the current implementation of netlink_dump(), which
* executes the dump callback successively from netlink_recvmsg(),
* always in the context of the receiving process */
@@ -3812,7 +3828,7 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device,
if (nla_put_drbd_cfg_context(skb, resource, the_only_connection(resource), device))
goto nla_put_failure;
if (res_opts_to_skb(skb, &device->resource->res_opts, exclude_sensitive))
if (res_opts_to_skb(skb, &device->resource->res_opts))
goto nla_put_failure;
rcu_read_lock();
@@ -3820,14 +3836,24 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device,
struct disk_conf *disk_conf;
disk_conf = rcu_dereference(device->ldev->disk_conf);
err = disk_conf_to_skb(skb, disk_conf, exclude_sensitive);
err = disk_conf_to_skb(skb, disk_conf);
}
if (!err) {
struct net_conf *nc;
nc = rcu_dereference(first_peer_device(device)->connection->net_conf);
if (nc)
err = net_conf_to_skb(skb, nc, exclude_sensitive);
if (nc) {
if (exclude_sensitive) {
struct net_conf nc_clean = *nc;
memset(nc_clean.shared_secret, 0,
sizeof(nc_clean.shared_secret));
nc_clean.shared_secret_len = 0;
err = net_conf_to_skb(skb, &nc_clean);
} else {
err = net_conf_to_skb(skb, nc);
}
}
}
rcu_read_unlock();
if (err)
@@ -3836,42 +3862,57 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device,
nla = nla_nest_start_noflag(skb, DRBD_NLA_STATE_INFO);
if (!nla)
goto nla_put_failure;
if (nla_put_u32(skb, T_sib_reason, sib ? sib->sib_reason : SIB_GET_STATUS_REPLY) ||
nla_put_u32(skb, T_current_state, device->state.i) ||
nla_put_u64_0pad(skb, T_ed_uuid, device->ed_uuid) ||
nla_put_u64_0pad(skb, T_capacity, get_capacity(device->vdisk)) ||
nla_put_u64_0pad(skb, T_send_cnt, device->send_cnt) ||
nla_put_u64_0pad(skb, T_recv_cnt, device->recv_cnt) ||
nla_put_u64_0pad(skb, T_read_cnt, device->read_cnt) ||
nla_put_u64_0pad(skb, T_writ_cnt, device->writ_cnt) ||
nla_put_u64_0pad(skb, T_al_writ_cnt, device->al_writ_cnt) ||
nla_put_u64_0pad(skb, T_bm_writ_cnt, device->bm_writ_cnt) ||
nla_put_u32(skb, T_ap_bio_cnt, atomic_read(&device->ap_bio_cnt)) ||
nla_put_u32(skb, T_ap_pending_cnt, atomic_read(&device->ap_pending_cnt)) ||
nla_put_u32(skb, T_rs_pending_cnt, atomic_read(&device->rs_pending_cnt)))
if (nla_put_u32(skb, DRBD_A_STATE_INFO_SIB_REASON,
sib ? sib->sib_reason : SIB_GET_STATUS_REPLY) ||
nla_put_u32(skb, DRBD_A_STATE_INFO_CURRENT_STATE,
device->state.i) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_ED_UUID,
device->ed_uuid, 0) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_CAPACITY,
get_capacity(device->vdisk), 0) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_SEND_CNT,
device->send_cnt, 0) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_RECV_CNT,
device->recv_cnt, 0) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_READ_CNT,
device->read_cnt, 0) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_WRIT_CNT,
device->writ_cnt, 0) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_AL_WRIT_CNT,
device->al_writ_cnt, 0) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BM_WRIT_CNT,
device->bm_writ_cnt, 0) ||
nla_put_u32(skb, DRBD_A_STATE_INFO_AP_BIO_CNT,
atomic_read(&device->ap_bio_cnt)) ||
nla_put_u32(skb, DRBD_A_STATE_INFO_AP_PENDING_CNT,
atomic_read(&device->ap_pending_cnt)) ||
nla_put_u32(skb, DRBD_A_STATE_INFO_RS_PENDING_CNT,
atomic_read(&device->rs_pending_cnt)))
goto nla_put_failure;
if (got_ldev) {
int err;
spin_lock_irq(&device->ldev->md.uuid_lock);
err = nla_put(skb, T_uuids, sizeof(si->uuids), device->ldev->md.uuid);
err = nla_put(skb, DRBD_A_STATE_INFO_UUIDS,
sizeof(si->uuids),
device->ldev->md.uuid);
spin_unlock_irq(&device->ldev->md.uuid_lock);
if (err)
goto nla_put_failure;
if (nla_put_u32(skb, T_disk_flags, device->ldev->md.flags) ||
nla_put_u64_0pad(skb, T_bits_total, drbd_bm_bits(device)) ||
nla_put_u64_0pad(skb, T_bits_oos,
drbd_bm_total_weight(device)))
if (nla_put_u32(skb, DRBD_A_STATE_INFO_DISK_FLAGS, device->ldev->md.flags) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_TOTAL, drbd_bm_bits(device), 0) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_OOS,
drbd_bm_total_weight(device), 0))
goto nla_put_failure;
if (C_SYNC_SOURCE <= device->state.conn &&
C_PAUSED_SYNC_T >= device->state.conn) {
if (nla_put_u64_0pad(skb, T_bits_rs_total,
device->rs_total) ||
nla_put_u64_0pad(skb, T_bits_rs_failed,
device->rs_failed))
if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_RS_TOTAL,
device->rs_total, 0) ||
nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_RS_FAILED,
device->rs_failed, 0))
goto nla_put_failure;
}
}
@@ -3882,17 +3923,17 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device,
case SIB_GET_STATUS_REPLY:
break;
case SIB_STATE_CHANGE:
if (nla_put_u32(skb, T_prev_state, sib->os.i) ||
nla_put_u32(skb, T_new_state, sib->ns.i))
if (nla_put_u32(skb, DRBD_A_STATE_INFO_PREV_STATE, sib->os.i) ||
nla_put_u32(skb, DRBD_A_STATE_INFO_NEW_STATE, sib->ns.i))
goto nla_put_failure;
break;
case SIB_HELPER_POST:
if (nla_put_u32(skb, T_helper_exit_code,
if (nla_put_u32(skb, DRBD_A_STATE_INFO_HELPER_EXIT_CODE,
sib->helper_exit_code))
goto nla_put_failure;
fallthrough;
case SIB_HELPER_PRE:
if (nla_put_string(skb, T_helper, sib->helper_name))
if (nla_put_string(skb, DRBD_A_STATE_INFO_HELPER, sib->helper_name))
goto nla_put_failure;
break;
}
@@ -3907,7 +3948,7 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device,
return err;
}
int drbd_adm_get_status(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_get_status_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
enum drbd_ret_code retcode;
@@ -3997,7 +4038,7 @@ static int get_one_status(struct sk_buff *skb, struct netlink_callback *cb)
}
dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
cb->nlh->nlmsg_seq, &drbd_genl_family,
cb->nlh->nlmsg_seq, &drbd_nl_family,
NLM_F_MULTI, DRBD_ADM_GET_STATUS);
if (!dh)
goto out;
@@ -4017,7 +4058,7 @@ static int get_one_status(struct sk_buff *skb, struct netlink_callback *cb)
struct net_conf *nc;
nc = rcu_dereference(connection->net_conf);
if (nc && net_conf_to_skb(skb, nc, 1) != 0)
if (nc && net_conf_to_skb(skb, nc) != 0)
goto cancel;
}
goto done;
@@ -4059,9 +4100,9 @@ static int get_one_status(struct sk_buff *skb, struct netlink_callback *cb)
*
* Once things are setup properly, we call into get_one_status().
*/
int drbd_adm_get_status_all(struct sk_buff *skb, struct netlink_callback *cb)
int drbd_nl_get_status_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
{
const unsigned hdrlen = GENL_HDRLEN + GENL_MAGIC_FAMILY_HDRSZ;
const unsigned int hdrlen = GENL_HDRLEN + sizeof(struct drbd_genlmsghdr);
struct nlattr *nla;
const char *resource_name;
struct drbd_resource *resource;
@@ -4084,7 +4125,7 @@ int drbd_adm_get_status_all(struct sk_buff *skb, struct netlink_callback *cb)
/* No explicit context given. Dump all. */
if (!nla)
goto dump;
nla = nla_find_nested(nla, T_ctx_resource_name);
nla = nla_find_nested(nla, DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME);
/* context given, but no name present? */
if (!nla)
return -EINVAL;
@@ -4107,7 +4148,7 @@ int drbd_adm_get_status_all(struct sk_buff *skb, struct netlink_callback *cb)
return get_one_status(skb, cb);
}
int drbd_adm_get_timeout_type(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_get_timeout_type_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
enum drbd_ret_code retcode;
@@ -4125,7 +4166,7 @@ int drbd_adm_get_timeout_type(struct sk_buff *skb, struct genl_info *info)
test_bit(USE_DEGR_WFC_T, &adm_ctx->device->flags) ? UT_DEGRADED :
UT_DEFAULT;
err = timeout_parms_to_priv_skb(adm_ctx->reply_skb, &tp);
err = timeout_parms_to_skb(adm_ctx->reply_skb, &tp);
if (err) {
nlmsg_free(adm_ctx->reply_skb);
adm_ctx->reply_skb = NULL;
@@ -4136,7 +4177,7 @@ int drbd_adm_get_timeout_type(struct sk_buff *skb, struct genl_info *info)
return 0;
}
int drbd_adm_start_ov(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_start_ov_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_device *device;
@@ -4182,7 +4223,7 @@ int drbd_adm_start_ov(struct sk_buff *skb, struct genl_info *info)
}
int drbd_adm_new_c_uuid(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_new_c_uuid_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_device *device;
@@ -4285,7 +4326,7 @@ static void resource_to_info(struct resource_info *info,
info->res_susp_fen = resource->susp_fen;
}
int drbd_adm_new_resource(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_new_resource_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_connection *connection;
struct drbd_config_context *adm_ctx = info->user_ptr[0];
@@ -4348,7 +4389,7 @@ static void device_to_info(struct device_info *info,
}
int drbd_adm_new_minor(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_new_minor_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_genlmsghdr *dh = genl_info_userhdr(info);
@@ -4455,7 +4496,7 @@ static enum drbd_ret_code adm_del_minor(struct drbd_device *device)
return ERR_MINOR_CONFIGURED;
}
int drbd_adm_del_minor(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_del_minor_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
enum drbd_ret_code retcode;
@@ -4504,7 +4545,7 @@ static int adm_del_resource(struct drbd_resource *resource)
return NO_ERROR;
}
int drbd_adm_down(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_down_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_resource *resource;
@@ -4567,7 +4608,7 @@ int drbd_adm_down(struct sk_buff *skb, struct genl_info *info)
return 0;
}
int drbd_adm_del_resource(struct sk_buff *skb, struct genl_info *info)
int drbd_nl_del_resource_doit(struct sk_buff *skb, struct genl_info *info)
{
struct drbd_config_context *adm_ctx = info->user_ptr[0];
struct drbd_resource *resource;
@@ -4601,7 +4642,7 @@ void drbd_bcast_event(struct drbd_device *device, const struct sib_info *sib)
goto failed;
err = -EMSGSIZE;
d_out = genlmsg_put(msg, 0, seq, &drbd_genl_family, 0, DRBD_EVENT);
d_out = genlmsg_put(msg, 0, seq, &drbd_nl_family, 0, DRBD_ADM_EVENT);
if (!d_out) /* cannot happen, but anyways. */
goto nla_put_failure;
d_out->minor = device_to_minor(device);
@@ -4632,7 +4673,7 @@ static int nla_put_notification_header(struct sk_buff *msg,
.nh_type = type,
};
return drbd_notification_header_to_skb(msg, &nh, true);
return drbd_notification_header_to_skb(msg, &nh);
}
int notify_resource_state(struct sk_buff *skb,
@@ -4656,7 +4697,7 @@ int notify_resource_state(struct sk_buff *skb,
}
err = -EMSGSIZE;
dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_RESOURCE_STATE);
dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_RESOURCE_STATE);
if (!dh)
goto nla_put_failure;
dh->minor = -1U;
@@ -4664,10 +4705,10 @@ int notify_resource_state(struct sk_buff *skb,
if (nla_put_drbd_cfg_context(skb, resource, NULL, NULL) ||
nla_put_notification_header(skb, type) ||
((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY &&
resource_info_to_skb(skb, resource_info, true)))
resource_info_to_skb(skb, resource_info)))
goto nla_put_failure;
resource_statistics.res_stat_write_ordering = resource->write_ordering;
err = resource_statistics_to_skb(skb, &resource_statistics, !capable(CAP_SYS_ADMIN));
err = resource_statistics_to_skb(skb, &resource_statistics);
if (err)
goto nla_put_failure;
genlmsg_end(skb, dh);
@@ -4708,7 +4749,7 @@ int notify_device_state(struct sk_buff *skb,
}
err = -EMSGSIZE;
dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_DEVICE_STATE);
dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_DEVICE_STATE);
if (!dh)
goto nla_put_failure;
dh->minor = device->minor;
@@ -4716,10 +4757,10 @@ int notify_device_state(struct sk_buff *skb,
if (nla_put_drbd_cfg_context(skb, device->resource, NULL, device) ||
nla_put_notification_header(skb, type) ||
((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY &&
device_info_to_skb(skb, device_info, true)))
device_info_to_skb(skb, device_info)))
goto nla_put_failure;
device_to_statistics(&device_statistics, device);
device_statistics_to_skb(skb, &device_statistics, !capable(CAP_SYS_ADMIN));
device_statistics_to_skb(skb, &device_statistics);
genlmsg_end(skb, dh);
if (multicast) {
err = drbd_genl_multicast_events(skb, GFP_NOWAIT);
@@ -4758,7 +4799,7 @@ int notify_connection_state(struct sk_buff *skb,
}
err = -EMSGSIZE;
dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_CONNECTION_STATE);
dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_CONNECTION_STATE);
if (!dh)
goto nla_put_failure;
dh->minor = -1U;
@@ -4766,10 +4807,10 @@ int notify_connection_state(struct sk_buff *skb,
if (nla_put_drbd_cfg_context(skb, connection->resource, connection, NULL) ||
nla_put_notification_header(skb, type) ||
((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY &&
connection_info_to_skb(skb, connection_info, true)))
connection_info_to_skb(skb, connection_info)))
goto nla_put_failure;
connection_statistics.conn_congested = test_bit(NET_CONGESTED, &connection->flags);
connection_statistics_to_skb(skb, &connection_statistics, !capable(CAP_SYS_ADMIN));
connection_statistics_to_skb(skb, &connection_statistics);
genlmsg_end(skb, dh);
if (multicast) {
err = drbd_genl_multicast_events(skb, GFP_NOWAIT);
@@ -4809,7 +4850,7 @@ int notify_peer_device_state(struct sk_buff *skb,
}
err = -EMSGSIZE;
dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_PEER_DEVICE_STATE);
dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_PEER_DEVICE_STATE);
if (!dh)
goto nla_put_failure;
dh->minor = -1U;
@@ -4817,10 +4858,10 @@ int notify_peer_device_state(struct sk_buff *skb,
if (nla_put_drbd_cfg_context(skb, resource, peer_device->connection, peer_device->device) ||
nla_put_notification_header(skb, type) ||
((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY &&
peer_device_info_to_skb(skb, peer_device_info, true)))
peer_device_info_to_skb(skb, peer_device_info)))
goto nla_put_failure;
peer_device_to_statistics(&peer_device_statistics, peer_device);
peer_device_statistics_to_skb(skb, &peer_device_statistics, !capable(CAP_SYS_ADMIN));
peer_device_statistics_to_skb(skb, &peer_device_statistics);
genlmsg_end(skb, dh);
if (multicast) {
err = drbd_genl_multicast_events(skb, GFP_NOWAIT);
@@ -4859,7 +4900,7 @@ void notify_helper(enum drbd_notification_type type,
goto fail;
err = -EMSGSIZE;
dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_HELPER);
dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_HELPER);
if (!dh)
goto fail;
dh->minor = device ? device->minor : -1;
@@ -4867,7 +4908,7 @@ void notify_helper(enum drbd_notification_type type,
mutex_lock(&notification_mutex);
if (nla_put_drbd_cfg_context(skb, resource, connection, device) ||
nla_put_notification_header(skb, type) ||
drbd_helper_info_to_skb(skb, &helper_info, true))
drbd_helper_info_to_skb(skb, &helper_info))
goto unlock_fail;
genlmsg_end(skb, dh);
err = drbd_genl_multicast_events(skb, GFP_NOWAIT);
@@ -4892,7 +4933,7 @@ static int notify_initial_state_done(struct sk_buff *skb, unsigned int seq)
int err;
err = -EMSGSIZE;
dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_INITIAL_STATE_DONE);
dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_INITIAL_STATE_DONE);
if (!dh)
goto nla_put_failure;
dh->minor = -1U;
@@ -4987,7 +5028,7 @@ static int get_initial_state(struct sk_buff *skb, struct netlink_callback *cb)
return skb->len;
}
int drbd_adm_get_initial_state(struct sk_buff *skb, struct netlink_callback *cb)
int drbd_nl_get_initial_state_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
{
struct drbd_resource *resource;
LIST_HEAD(head);
@@ -5035,3 +5076,20 @@ int drbd_adm_get_initial_state(struct sk_buff *skb, struct netlink_callback *cb)
cb->args[2] = cb->nlh->nlmsg_seq;
return get_initial_state(skb, cb);
}
static const struct genl_multicast_group drbd_nl_mcgrps[] = {
[DRBD_NLGRP_EVENTS] = { .name = "events", },
};
struct genl_family drbd_nl_family __ro_after_init = {
.name = "drbd",
.version = DRBD_FAMILY_VERSION,
.hdrsize = NLA_ALIGN(sizeof(struct drbd_genlmsghdr)),
.split_ops = drbd_nl_ops,
.n_split_ops = ARRAY_SIZE(drbd_nl_ops),
.mcgrps = drbd_nl_mcgrps,
.n_mcgrps = ARRAY_SIZE(drbd_nl_mcgrps),
.resv_start_op = 42,
.module = THIS_MODULE,
.netnsok = true,
};

2606
drivers/block/drbd/drbd_nl_gen.c Normal file
View File

File diff suppressed because it is too large Load Diff

395
drivers/block/drbd/drbd_nl_gen.h Normal file
View File

@@ -0,0 +1,395 @@
/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
#ifndef _LINUX_DRBD_GEN_H
#define _LINUX_DRBD_GEN_H
#include <net/netlink.h>
#include <net/genetlink.h>
#include <uapi/linux/drbd_genl.h>
#include <linux/drbd.h>
#include <linux/drbd_limits.h>
/* Common nested types */
extern const struct nla_policy drbd_connection_info_nl_policy[DRBD_A_CONNECTION_INFO_CONN_ROLE + 1];
extern const struct nla_policy drbd_connection_statistics_nl_policy[DRBD_A_CONNECTION_STATISTICS_CONN_CONGESTED + 1];
extern const struct nla_policy drbd_detach_parms_nl_policy[DRBD_A_DETACH_PARMS_FORCE_DETACH + 1];
extern const struct nla_policy drbd_device_info_nl_policy[DRBD_A_DEVICE_INFO_DEV_DISK_STATE + 1];
extern const struct nla_policy drbd_device_statistics_nl_policy[DRBD_A_DEVICE_STATISTICS_HISTORY_UUIDS + 1];
extern const struct nla_policy drbd_disconnect_parms_nl_policy[DRBD_A_DISCONNECT_PARMS_FORCE_DISCONNECT + 1];
extern const struct nla_policy drbd_disk_conf_nl_policy[DRBD_A_DISK_CONF_DISABLE_WRITE_SAME + 1];
extern const struct nla_policy drbd_drbd_cfg_context_nl_policy[DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR + 1];
extern const struct nla_policy drbd_net_conf_nl_policy[DRBD_A_NET_CONF_SOCK_CHECK_TIMEO + 1];
extern const struct nla_policy drbd_new_c_uuid_parms_nl_policy[DRBD_A_NEW_C_UUID_PARMS_CLEAR_BM + 1];
extern const struct nla_policy drbd_peer_device_info_nl_policy[DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_DEPENDENCY + 1];
extern const struct nla_policy drbd_peer_device_statistics_nl_policy[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_FLAGS + 1];
extern const struct nla_policy drbd_res_opts_nl_policy[DRBD_A_RES_OPTS_ON_NO_DATA + 1];
extern const struct nla_policy drbd_resize_parms_nl_policy[DRBD_A_RESIZE_PARMS_AL_STRIPE_SIZE + 1];
extern const struct nla_policy drbd_resource_info_nl_policy[DRBD_A_RESOURCE_INFO_RES_SUSP_FEN + 1];
extern const struct nla_policy drbd_resource_statistics_nl_policy[DRBD_A_RESOURCE_STATISTICS_RES_STAT_WRITE_ORDERING + 1];
extern const struct nla_policy drbd_set_role_parms_nl_policy[DRBD_A_SET_ROLE_PARMS_ASSUME_UPTODATE + 1];
extern const struct nla_policy drbd_start_ov_parms_nl_policy[DRBD_A_START_OV_PARMS_OV_STOP_SECTOR + 1];
/* Ops table for drbd */
extern const struct genl_split_ops drbd_nl_ops[32];
int drbd_pre_doit(const struct genl_split_ops *ops, struct sk_buff *skb,
struct genl_info *info);
void
drbd_post_doit(const struct genl_split_ops *ops, struct sk_buff *skb,
struct genl_info *info);
int drbd_adm_dump_devices_done(struct netlink_callback *cb);
int drbd_adm_dump_connections_done(struct netlink_callback *cb);
int drbd_adm_dump_peer_devices_done(struct netlink_callback *cb);
int drbd_nl_get_status_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_get_status_dumpit(struct sk_buff *skb, struct netlink_callback *cb);
int drbd_nl_new_minor_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_del_minor_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_new_resource_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_del_resource_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_resource_opts_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_connect_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_disconnect_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_attach_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_resize_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_primary_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_secondary_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_new_c_uuid_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_start_ov_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_detach_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_invalidate_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_inval_peer_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_pause_sync_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_resume_sync_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_suspend_io_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_resume_io_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_outdate_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_get_timeout_type_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_down_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_chg_disk_opts_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_chg_net_opts_doit(struct sk_buff *skb, struct genl_info *info);
int drbd_nl_get_resources_dumpit(struct sk_buff *skb,
struct netlink_callback *cb);
int drbd_nl_get_devices_dumpit(struct sk_buff *skb,
struct netlink_callback *cb);
int drbd_nl_get_connections_dumpit(struct sk_buff *skb,
struct netlink_callback *cb);
int drbd_nl_get_peer_devices_dumpit(struct sk_buff *skb,
struct netlink_callback *cb);
int drbd_nl_get_initial_state_dumpit(struct sk_buff *skb,
struct netlink_callback *cb);
enum {
DRBD_NLGRP_EVENTS,
};
struct drbd_cfg_reply {
char info_text[0];
__u32 info_text_len;
};
struct drbd_cfg_context {
__u32 ctx_volume;
char ctx_resource_name[128];
__u32 ctx_resource_name_len;
char ctx_my_addr[128];
__u32 ctx_my_addr_len;
char ctx_peer_addr[128];
__u32 ctx_peer_addr_len;
};
struct disk_conf {
char backing_dev[128];
__u32 backing_dev_len;
char meta_dev[128];
__u32 meta_dev_len;
__s32 meta_dev_idx;
__u64 disk_size;
__u32 max_bio_bvecs;
__u32 on_io_error;
__u32 fencing;
__u32 resync_rate;
__s32 resync_after;
__u32 al_extents;
__u32 c_plan_ahead;
__u32 c_delay_target;
__u32 c_fill_target;
__u32 c_max_rate;
__u32 c_min_rate;
unsigned char disk_barrier;
unsigned char disk_flushes;
unsigned char disk_drain;
unsigned char md_flushes;
__u32 disk_timeout;
__u32 read_balancing;
unsigned char al_updates;
unsigned char discard_zeroes_if_aligned;
__u32 rs_discard_granularity;
unsigned char disable_write_same;
};
struct res_opts {
char cpu_mask[DRBD_CPU_MASK_SIZE];
__u32 cpu_mask_len;
__u32 on_no_data;
};
struct net_conf {
char shared_secret[SHARED_SECRET_MAX];
__u32 shared_secret_len;
char cram_hmac_alg[SHARED_SECRET_MAX];
__u32 cram_hmac_alg_len;
char integrity_alg[SHARED_SECRET_MAX];
__u32 integrity_alg_len;
char verify_alg[SHARED_SECRET_MAX];
__u32 verify_alg_len;
char csums_alg[SHARED_SECRET_MAX];
__u32 csums_alg_len;
__u32 wire_protocol;
__u32 connect_int;
__u32 timeout;
__u32 ping_int;
__u32 ping_timeo;
__u32 sndbuf_size;
__u32 rcvbuf_size;
__u32 ko_count;
__u32 max_buffers;
__u32 max_epoch_size;
__u32 unplug_watermark;
__u32 after_sb_0p;
__u32 after_sb_1p;
__u32 after_sb_2p;
__u32 rr_conflict;
__u32 on_congestion;
__u32 cong_fill;
__u32 cong_extents;
unsigned char two_primaries;
unsigned char discard_my_data;
unsigned char tcp_cork;
unsigned char always_asbp;
unsigned char tentative;
unsigned char use_rle;
unsigned char csums_after_crash_only;
__u32 sock_check_timeo;
};
struct set_role_parms {
unsigned char assume_uptodate;
};
struct resize_parms {
__u64 resize_size;
unsigned char resize_force;
unsigned char no_resync;
__u32 al_stripes;
__u32 al_stripe_size;
};
struct state_info {
__u32 sib_reason;
__u32 current_state;
__u64 capacity;
__u64 ed_uuid;
__u32 prev_state;
__u32 new_state;
char uuids[DRBD_NL_UUIDS_SIZE];
__u32 uuids_len;
__u32 disk_flags;
__u64 bits_total;
__u64 bits_oos;
__u64 bits_rs_total;
__u64 bits_rs_failed;
char helper[32];
__u32 helper_len;
__u32 helper_exit_code;
__u64 send_cnt;
__u64 recv_cnt;
__u64 read_cnt;
__u64 writ_cnt;
__u64 al_writ_cnt;
__u64 bm_writ_cnt;
__u32 ap_bio_cnt;
__u32 ap_pending_cnt;
__u32 rs_pending_cnt;
};
struct start_ov_parms {
__u64 ov_start_sector;
__u64 ov_stop_sector;
};
struct new_c_uuid_parms {
unsigned char clear_bm;
};
struct timeout_parms {
__u32 timeout_type;
};
struct disconnect_parms {
unsigned char force_disconnect;
};
struct detach_parms {
unsigned char force_detach;
};
struct resource_info {
__u32 res_role;
unsigned char res_susp;
unsigned char res_susp_nod;
unsigned char res_susp_fen;
};
struct device_info {
__u32 dev_disk_state;
};
struct connection_info {
__u32 conn_connection_state;
__u32 conn_role;
};
struct peer_device_info {
__u32 peer_repl_state;
__u32 peer_disk_state;
__u32 peer_resync_susp_user;
__u32 peer_resync_susp_peer;
__u32 peer_resync_susp_dependency;
};
struct resource_statistics {
__u32 res_stat_write_ordering;
};
struct device_statistics {
__u64 dev_size;
__u64 dev_read;
__u64 dev_write;
__u64 dev_al_writes;
__u64 dev_bm_writes;
__u32 dev_upper_pending;
__u32 dev_lower_pending;
unsigned char dev_upper_blocked;
unsigned char dev_lower_blocked;
unsigned char dev_al_suspended;
__u64 dev_exposed_data_uuid;
__u64 dev_current_uuid;
__u32 dev_disk_flags;
char history_uuids[DRBD_NL_HISTORY_UUIDS_SIZE];
__u32 history_uuids_len;
};
struct connection_statistics {
unsigned char conn_congested;
};
struct peer_device_statistics {
__u64 peer_dev_received;
__u64 peer_dev_sent;
__u32 peer_dev_pending;
__u32 peer_dev_unacked;
__u64 peer_dev_out_of_sync;
__u64 peer_dev_resync_failed;
__u64 peer_dev_bitmap_uuid;
__u32 peer_dev_flags;
};
struct drbd_notification_header {
__u32 nh_type;
};
struct drbd_helper_info {
char helper_name[32];
__u32 helper_name_len;
__u32 helper_status;
};
int drbd_cfg_reply_to_skb(struct sk_buff *skb, struct drbd_cfg_reply *s);
int drbd_cfg_context_from_attrs(struct drbd_cfg_context *s, struct genl_info *info);
int drbd_cfg_context_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int drbd_cfg_context_to_skb(struct sk_buff *skb, struct drbd_cfg_context *s);
int disk_conf_from_attrs(struct disk_conf *s, struct genl_info *info);
int disk_conf_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int disk_conf_to_skb(struct sk_buff *skb, struct disk_conf *s);
void set_disk_conf_defaults(struct disk_conf *x);
int res_opts_from_attrs(struct res_opts *s, struct genl_info *info);
int res_opts_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int res_opts_to_skb(struct sk_buff *skb, struct res_opts *s);
void set_res_opts_defaults(struct res_opts *x);
int net_conf_from_attrs(struct net_conf *s, struct genl_info *info);
int net_conf_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int net_conf_to_skb(struct sk_buff *skb, struct net_conf *s);
void set_net_conf_defaults(struct net_conf *x);
int set_role_parms_from_attrs(struct set_role_parms *s, struct genl_info *info);
int set_role_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int set_role_parms_to_skb(struct sk_buff *skb, struct set_role_parms *s);
int resize_parms_from_attrs(struct resize_parms *s, struct genl_info *info);
int resize_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int resize_parms_to_skb(struct sk_buff *skb, struct resize_parms *s);
void set_resize_parms_defaults(struct resize_parms *x);
int state_info_to_skb(struct sk_buff *skb, struct state_info *s);
int start_ov_parms_from_attrs(struct start_ov_parms *s, struct genl_info *info);
int start_ov_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int start_ov_parms_to_skb(struct sk_buff *skb, struct start_ov_parms *s);
int new_c_uuid_parms_from_attrs(struct new_c_uuid_parms *s, struct genl_info *info);
int new_c_uuid_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int new_c_uuid_parms_to_skb(struct sk_buff *skb, struct new_c_uuid_parms *s);
int timeout_parms_to_skb(struct sk_buff *skb, struct timeout_parms *s);
int disconnect_parms_from_attrs(struct disconnect_parms *s, struct genl_info *info);
int disconnect_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int disconnect_parms_to_skb(struct sk_buff *skb, struct disconnect_parms *s);
int detach_parms_from_attrs(struct detach_parms *s, struct genl_info *info);
int detach_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int detach_parms_to_skb(struct sk_buff *skb, struct detach_parms *s);
int resource_info_from_attrs(struct resource_info *s, struct genl_info *info);
int resource_info_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int resource_info_to_skb(struct sk_buff *skb, struct resource_info *s);
int device_info_from_attrs(struct device_info *s, struct genl_info *info);
int device_info_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int device_info_to_skb(struct sk_buff *skb, struct device_info *s);
int connection_info_from_attrs(struct connection_info *s, struct genl_info *info);
int connection_info_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int connection_info_to_skb(struct sk_buff *skb, struct connection_info *s);
int peer_device_info_from_attrs(struct peer_device_info *s, struct genl_info *info);
int peer_device_info_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int peer_device_info_to_skb(struct sk_buff *skb, struct peer_device_info *s);
int resource_statistics_from_attrs(struct resource_statistics *s, struct genl_info *info);
int resource_statistics_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int resource_statistics_to_skb(struct sk_buff *skb, struct resource_statistics *s);
int device_statistics_from_attrs(struct device_statistics *s, struct genl_info *info);
int device_statistics_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int device_statistics_to_skb(struct sk_buff *skb, struct device_statistics *s);
int connection_statistics_from_attrs(struct connection_statistics *s, struct genl_info *info);
int connection_statistics_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int connection_statistics_to_skb(struct sk_buff *skb, struct connection_statistics *s);
int peer_device_statistics_from_attrs(struct peer_device_statistics *s, struct genl_info *info);
int peer_device_statistics_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info);
int peer_device_statistics_to_skb(struct sk_buff *skb, struct peer_device_statistics *s);
int drbd_notification_header_to_skb(struct sk_buff *skb, struct drbd_notification_header *s);
int drbd_helper_info_to_skb(struct sk_buff *skb, struct drbd_helper_info *s);
#endif /* _LINUX_DRBD_GEN_H */

2
drivers/block/drbd/drbd_proc.c
View File

@@ -228,7 +228,7 @@ int drbd_seq_show(struct seq_file *seq, void *v)
};
seq_printf(seq, "version: " REL_VERSION " (api:%d/proto:%d-%d)\n%s\n",
GENL_MAGIC_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX, drbd_buildtag());
DRBD_FAMILY_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX, drbd_buildtag());
/*
cs .. connection state

4
drivers/block/floppy.c
View File

@@ -5012,8 +5012,8 @@ MODULE_LICENSE("GPL");
/* This doesn't actually get used other than for module information */
static const struct pnp_device_id floppy_pnpids[] = {
{"PNP0700", 0},
{}
{ .id = "PNP0700" },
{ }
};
MODULE_DEVICE_TABLE(pnp, floppy_pnpids);

24
drivers/block/loop.c
View File

@@ -342,23 +342,19 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd,
{
struct iov_iter iter;
struct req_iterator rq_iter;
struct bio_vec *bvec;
struct request *rq = blk_mq_rq_from_pdu(cmd);
struct bio *bio = rq->bio;
struct file *file = lo->lo_backing_file;
struct bio_vec tmp;
unsigned int offset;
unsigned int nr_bvec;
int ret;
nr_bvec = blk_rq_nr_bvec(rq);
if (rq->bio != rq->biotail) {
struct bio_vec tmp, *bvec;
bvec = kmalloc_objs(struct bio_vec, nr_bvec, GFP_NOIO);
if (!bvec)
cmd->bvec = kmalloc_objs(*cmd->bvec, nr_bvec, GFP_NOIO);
if (!cmd->bvec)
return -EIO;
cmd->bvec = bvec;
/*
* The bios of the request may be started from the middle of
@@ -366,26 +362,26 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd,
* copy bio->bi_iov_vec to new bvec. The rq_for_each_bvec
* API will take care of all details for us.
*/
bvec = cmd->bvec;
rq_for_each_bvec(tmp, rq, rq_iter) {
*bvec = tmp;
bvec++;
}
bvec = cmd->bvec;
offset = 0;
iov_iter_bvec(&iter, rw, cmd->bvec, nr_bvec, blk_rq_bytes(rq));
iter.iov_offset = 0;
} else {
/*
* Same here, this bio may be started from the middle of the
* 'bvec' because of bio splitting, so offset from the bvec
* must be passed to iov iterator
*/
offset = bio->bi_iter.bi_bvec_done;
bvec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter);
iov_iter_bvec(&iter, rw,
__bvec_iter_bvec(rq->bio->bi_io_vec, rq->bio->bi_iter),
nr_bvec, blk_rq_bytes(rq));
iter.iov_offset = rq->bio->bi_iter.bi_bvec_done;
}
atomic_set(&cmd->ref, 2);
iov_iter_bvec(&iter, rw, bvec, nr_bvec, blk_rq_bytes(rq));
iter.iov_offset = offset;
cmd->iocb.ki_pos = pos;
cmd->iocb.ki_filp = file;
cmd->iocb.ki_ioprio = req_get_ioprio(rq);

19
drivers/block/mtip32xx/mtip32xx.c
View File

@@ -3340,7 +3340,7 @@ static void mtip_free_cmd(struct blk_mq_tag_set *set, struct request *rq,
}
static int mtip_init_cmd(struct blk_mq_tag_set *set, struct request *rq,
unsigned int hctx_idx, unsigned int numa_node)
unsigned int hctx_idx, int numa_node)
{
struct driver_data *dd = set->driver_data;
struct mtip_cmd *cmd = blk_mq_rq_to_pdu(rq);
@@ -3405,6 +3405,7 @@ static int mtip_block_initialize(struct driver_data *dd)
.max_segment_size = 0x400000,
};
int rv = 0, wait_for_rebuild = 0;
bool disk_added = false;
sector_t capacity;
unsigned int index = 0;
@@ -3438,6 +3439,7 @@ static int mtip_block_initialize(struct driver_data *dd)
dev_err(&dd->pdev->dev,
"Unable to allocate request queue\n");
rv = -ENOMEM;
dd->disk = NULL;
goto block_queue_alloc_init_error;
}
dd->queue = dd->disk->queue;
@@ -3496,6 +3498,7 @@ static int mtip_block_initialize(struct driver_data *dd)
rv = device_add_disk(&dd->pdev->dev, dd->disk, mtip_disk_attr_groups);
if (rv)
goto read_capacity_error;
disk_added = true;
if (dd->mtip_svc_handler) {
set_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag);
@@ -3511,7 +3514,9 @@ static int mtip_block_initialize(struct driver_data *dd)
dev_err(&dd->pdev->dev, "service thread failed to start\n");
dd->mtip_svc_handler = NULL;
rv = -EFAULT;
goto kthread_run_error;
if (disk_added)
goto kthread_run_error;
goto read_capacity_error;
}
wake_up_process(dd->mtip_svc_handler);
if (wait_for_rebuild == MTIP_FTL_REBUILD_MAGIC)
@@ -3522,6 +3527,10 @@ static int mtip_block_initialize(struct driver_data *dd)
kthread_run_error:
/* Delete our gendisk. This also removes the device from /dev */
del_gendisk(dd->disk);
mtip_hw_debugfs_exit(dd);
blk_mq_free_tag_set(&dd->tags);
mtip_hw_exit(dd);
return rv;
read_capacity_error:
init_hw_cmds_error:
mtip_hw_debugfs_exit(dd);
@@ -3529,6 +3538,7 @@ static int mtip_block_initialize(struct driver_data *dd)
ida_free(&rssd_index_ida, index);
ida_get_error:
put_disk(dd->disk);
dd->disk = NULL;
block_queue_alloc_init_error:
blk_mq_free_tag_set(&dd->tags);
block_queue_alloc_tag_error:
@@ -3839,7 +3849,10 @@ static int mtip_pci_probe(struct pci_dev *pdev,
}
iomap_err:
kfree(dd);
if (dd->disk)
put_disk(dd->disk);
else
kfree(dd);
pci_set_drvdata(pdev, NULL);
return rv;
done:

3
drivers/block/n64cart.c
View File

@@ -12,6 +12,7 @@
#include <linux/init.h>
#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/string.h>
enum {
PI_DRAM_REG = 0,
@@ -145,7 +146,7 @@ static int __init n64cart_probe(struct platform_device *pdev)
disk->flags = GENHD_FL_NO_PART;
disk->fops = &n64cart_fops;
disk->private_data = &pdev->dev;
strcpy(disk->disk_name, "n64cart");
strscpy(disk->disk_name, "n64cart");
set_capacity(disk, size >> SECTOR_SHIFT);
set_disk_ro(disk, 1);

39
drivers/block/nbd.c
View File

@@ -1238,6 +1238,42 @@ static struct socket *nbd_get_socket(struct nbd_device *nbd, unsigned long fd,
return sock;
}
#ifdef CONFIG_DEBUG_LOCK_ALLOC
static struct lock_class_key nbd_key[3];
static struct lock_class_key nbd_slock_key[3];
static void nbd_reclassify_socket(struct socket *sock)
{
struct sock *sk = sock->sk;
if (WARN_ON_ONCE(!sock_allow_reclassification(sk)))
return;
switch (sk->sk_family) {
case AF_INET:
sock_lock_init_class_and_name(sk, "slock-AF_INET-NBD",
&nbd_slock_key[0],
"sk_lock-AF_INET-NBD",
&nbd_key[0]);
break;
case AF_INET6:
sock_lock_init_class_and_name(sk, "slock-AF_INET6-NBD",
&nbd_slock_key[1],
"sk_lock-AF_INET6-NBD",
&nbd_key[1]);
break;
case AF_UNIX:
sock_lock_init_class_and_name(sk, "slock-AF_UNIX-NBD",
&nbd_slock_key[2],
"sk_lock-AF_UNIX-NBD",
&nbd_key[2]);
break;
}
}
#else
static inline void nbd_reclassify_socket(struct socket *sock) {}
#endif
static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
bool netlink)
{
@@ -1254,6 +1290,7 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
sock = nbd_get_socket(nbd, arg, &err);
if (!sock)
return err;
nbd_reclassify_socket(sock);
/*
* We need to make sure we don't get any errant requests while we're
@@ -1888,7 +1925,7 @@ static void nbd_dbg_close(void)
#endif
static int nbd_init_request(struct blk_mq_tag_set *set, struct request *rq,
unsigned int hctx_idx, unsigned int numa_node)
unsigned int hctx_idx, int numa_node)
{
struct nbd_cmd *cmd = blk_mq_rq_to_pdu(rq);
cmd->nbd = set->driver_data;

9
drivers/block/rbd.c
View File

@@ -3672,7 +3672,7 @@ static void __rbd_lock(struct rbd_device *rbd_dev, const char *cookie)
struct rbd_client_id cid = rbd_get_cid(rbd_dev);
rbd_dev->lock_state = RBD_LOCK_STATE_LOCKED;
strcpy(rbd_dev->lock_cookie, cookie);
strscpy(rbd_dev->lock_cookie, cookie);
rbd_set_owner_cid(rbd_dev, &cid);
queue_work(rbd_dev->task_wq, &rbd_dev->acquired_lock_work);
}
@@ -6082,12 +6082,9 @@ static int rbd_dev_v2_snap_context(struct rbd_device *rbd_dev,
/*
* Make sure the reported number of snapshot ids wouldn't go
* beyond the end of our buffer. But before checking that,
* make sure the computed size of the snapshot context we
* allocate is representable in a size_t.
* beyond the end of our buffer.
*/
if (snap_count > (SIZE_MAX - sizeof (struct ceph_snap_context))
/ sizeof (u64)) {
if (snap_count > RBD_MAX_SNAP_COUNT) {
ret = -EINVAL;
goto out;
}

174
drivers/block/ublk_drv.c
View File

@@ -359,7 +359,6 @@ static void ublk_buf_cleanup(struct ublk_device *ub);
static void ublk_abort_queue(struct ublk_device *ub, struct ublk_queue *ubq);
static inline struct request *__ublk_check_and_get_req(struct ublk_device *ub,
u16 q_id, u16 tag, struct ublk_io *io);
static inline unsigned int ublk_req_build_flags(struct request *req);
static void ublk_batch_dispatch(struct ublk_queue *ubq,
const struct ublk_batch_io_data *data,
struct ublk_batch_fetch_cmd *fcmd);
@@ -471,6 +470,62 @@ static inline bool ublk_dev_support_integrity(const struct ublk_device *ub)
return ub->dev_info.flags & UBLK_F_INTEGRITY;
}
static inline unsigned int ublk_req_build_flags(struct request *req)
{
unsigned flags = 0;
if (req->cmd_flags & REQ_FAILFAST_DEV)
flags |= UBLK_IO_F_FAILFAST_DEV;
if (req->cmd_flags & REQ_FAILFAST_TRANSPORT)
flags |= UBLK_IO_F_FAILFAST_TRANSPORT;
if (req->cmd_flags & REQ_FAILFAST_DRIVER)
flags |= UBLK_IO_F_FAILFAST_DRIVER;
if (req->cmd_flags & REQ_META)
flags |= UBLK_IO_F_META;
if (req->cmd_flags & REQ_FUA)
flags |= UBLK_IO_F_FUA;
if (req->cmd_flags & REQ_NOUNMAP)
flags |= UBLK_IO_F_NOUNMAP;
if (req->cmd_flags & REQ_SWAP)
flags |= UBLK_IO_F_SWAP;
if (blk_integrity_rq(req))
flags |= UBLK_IO_F_INTEGRITY;
return flags;
}
static void ublk_init_iod(struct ublk_queue *ubq, struct request *req,
uint8_t ublk_op, uint32_t nr_sectors,
uint64_t start_sector)
{
struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag);
struct ublk_io *io = &ubq->ios[req->tag];
iod->op_flags = ublk_op | ublk_req_build_flags(req);
iod->nr_sectors = nr_sectors;
iod->start_sector = start_sector;
/* Try shmem zero-copy match before setting addr */
if (ublk_support_shmem_zc(ubq) && blk_rq_has_data(req)) {
u32 buf_idx, buf_off;
if (ublk_try_buf_match(ubq->dev, req, &buf_idx, &buf_off)) {
iod->op_flags |= UBLK_IO_F_SHMEM_ZC;
iod->addr = ublk_shmem_zc_addr(buf_idx, buf_off);
return;
}
}
iod->addr = io->buf.addr;
}
#ifdef CONFIG_BLK_DEV_ZONED
struct ublk_zoned_report_desc {
@@ -652,8 +707,6 @@ static int ublk_report_zones(struct gendisk *disk, sector_t sector,
static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq,
struct request *req)
{
struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag);
struct ublk_io *io = &ubq->ios[req->tag];
struct ublk_zoned_report_desc *desc;
u32 ublk_op;
@@ -683,9 +736,8 @@ static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq,
ublk_op = desc->operation;
switch (ublk_op) {
case UBLK_IO_OP_REPORT_ZONES:
iod->op_flags = ublk_op | ublk_req_build_flags(req);
iod->nr_zones = desc->nr_zones;
iod->start_sector = desc->sector;
ublk_init_iod(ubq, req, ublk_op, desc->nr_zones,
desc->sector);
return BLK_STS_OK;
default:
return BLK_STS_IOERR;
@@ -697,11 +749,7 @@ static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq,
return BLK_STS_IOERR;
}
iod->op_flags = ublk_op | ublk_req_build_flags(req);
iod->nr_sectors = blk_rq_sectors(req);
iod->start_sector = blk_rq_pos(req);
iod->addr = io->buf.addr;
ublk_init_iod(ubq, req, ublk_op, blk_rq_sectors(req), blk_rq_pos(req));
return BLK_STS_OK;
}
@@ -1189,11 +1237,6 @@ static inline struct ublk_queue *ublk_get_queue(struct ublk_device *dev,
return dev->queues[qid];
}
static inline bool ublk_rq_has_data(const struct request *rq)
{
return bio_has_data(rq->bio);
}
static inline struct ublksrv_io_desc *
ublk_queue_cmd_buf(struct ublk_device *ub, int q_id)
{
@@ -1406,12 +1449,12 @@ static size_t ublk_copy_user_integrity(const struct request *req,
static inline bool ublk_need_map_req(const struct request *req)
{
return ublk_rq_has_data(req) && req_op(req) == REQ_OP_WRITE;
return blk_rq_has_data(req) && req_op(req) == REQ_OP_WRITE;
}
static inline bool ublk_need_unmap_req(const struct request *req)
{
return ublk_rq_has_data(req) &&
return blk_rq_has_data(req) &&
(req_op(req) == REQ_OP_READ || req_op(req) == REQ_OP_DRV_IN);
}
@@ -1460,41 +1503,8 @@ static unsigned int ublk_unmap_io(bool need_map,
return rq_bytes;
}
static inline unsigned int ublk_req_build_flags(struct request *req)
{
unsigned flags = 0;
if (req->cmd_flags & REQ_FAILFAST_DEV)
flags |= UBLK_IO_F_FAILFAST_DEV;
if (req->cmd_flags & REQ_FAILFAST_TRANSPORT)
flags |= UBLK_IO_F_FAILFAST_TRANSPORT;
if (req->cmd_flags & REQ_FAILFAST_DRIVER)
flags |= UBLK_IO_F_FAILFAST_DRIVER;
if (req->cmd_flags & REQ_META)
flags |= UBLK_IO_F_META;
if (req->cmd_flags & REQ_FUA)
flags |= UBLK_IO_F_FUA;
if (req->cmd_flags & REQ_NOUNMAP)
flags |= UBLK_IO_F_NOUNMAP;
if (req->cmd_flags & REQ_SWAP)
flags |= UBLK_IO_F_SWAP;
if (blk_integrity_rq(req))
flags |= UBLK_IO_F_INTEGRITY;
return flags;
}
static blk_status_t ublk_setup_iod(struct ublk_queue *ubq, struct request *req)
{
struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag);
struct ublk_io *io = &ubq->ios[req->tag];
u32 ublk_op;
switch (req_op(req)) {
@@ -1519,25 +1529,7 @@ static blk_status_t ublk_setup_iod(struct ublk_queue *ubq, struct request *req)
return BLK_STS_IOERR;
}
/* need to translate since kernel may change */
iod->op_flags = ublk_op | ublk_req_build_flags(req);
iod->nr_sectors = blk_rq_sectors(req);
iod->start_sector = blk_rq_pos(req);
/* Try shmem zero-copy match before setting addr */
if (ublk_support_shmem_zc(ubq) && ublk_rq_has_data(req)) {
u32 buf_idx, buf_off;
if (ublk_try_buf_match(ubq->dev, req,
&buf_idx, &buf_off)) {
iod->op_flags |= UBLK_IO_F_SHMEM_ZC;
iod->addr = ublk_shmem_zc_addr(buf_idx, buf_off);
return BLK_STS_OK;
}
}
iod->addr = io->buf.addr;
ublk_init_iod(ubq, req, ublk_op, blk_rq_sectors(req), blk_rq_pos(req));
return BLK_STS_OK;
}
@@ -1815,7 +1807,7 @@ static void ublk_dispatch_req(struct ublk_queue *ubq, struct request *req)
if (!ublk_start_io(ubq, req, io))
return;
if (ublk_support_auto_buf_reg(ubq) && ublk_rq_has_data(req)) {
if (ublk_support_auto_buf_reg(ubq) && blk_rq_has_data(req)) {
ublk_auto_buf_dispatch(ubq, req, io, io->cmd, issue_flags);
} else {
ublk_init_req_ref(ubq, io);
@@ -1836,7 +1828,7 @@ static bool __ublk_batch_prep_dispatch(struct ublk_queue *ubq,
if (!ublk_start_io(ubq, req, io))
return false;
if (ublk_support_auto_buf_reg(ubq) && ublk_rq_has_data(req)) {
if (ublk_support_auto_buf_reg(ubq) && blk_rq_has_data(req)) {
res = ublk_auto_buf_register(ubq, req, io, cmd,
data->issue_flags);
@@ -2735,23 +2727,27 @@ static void ublk_start_cancel(struct ublk_device *ub)
{
struct gendisk *disk = ublk_get_disk(ub);
/* Our disk has been dead */
if (!disk)
return;
mutex_lock(&ub->cancel_mutex);
if (ub->canceling)
goto out;
/*
* Now we are serialized with ublk_queue_rq()
*
* Make sure that ubq->canceling is set when queue is frozen,
* because ublk_queue_rq() has to rely on this flag for avoiding to
* touch completed uring_cmd
*/
blk_mq_quiesce_queue(disk->queue);
ublk_set_canceling(ub, true);
blk_mq_unquiesce_queue(disk->queue);
if (disk) {
/*
* Quiesce to serialize with ublk_queue_rq(), ensuring
* ubq->canceling is visible when the queue resumes.
*/
blk_mq_quiesce_queue(disk->queue);
ublk_set_canceling(ub, true);
blk_mq_unquiesce_queue(disk->queue);
} else {
/*
* Disk not yet allocated by ublk_ctrl_start_dev(), so
* there is no request queue and ublk_queue_rq() cannot
* be running. Just set the flag; if start_dev proceeds
* later, new I/O will see canceling and be aborted.
*/
ublk_set_canceling(ub, true);
}
out:
mutex_unlock(&ub->cancel_mutex);
ublk_put_disk(disk);
@@ -3227,7 +3223,7 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd,
return ublk_register_io_buf(cmd, ub, q_id, tag, io, index,
issue_flags);
if (!ublk_dev_support_zero_copy(ub) || !ublk_rq_has_data(req))
if (!ublk_dev_support_zero_copy(ub) || !blk_rq_has_data(req))
return -EINVAL;
ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index,
@@ -3510,7 +3506,7 @@ static inline struct request *__ublk_check_and_get_req(struct ublk_device *ub,
if (unlikely(!blk_mq_request_started(req) || req->tag != tag))
goto fail_put;
if (!ublk_rq_has_data(req))
if (!blk_rq_has_data(req))
goto fail_put;
return req;
@@ -4085,7 +4081,7 @@ ublk_user_copy(struct kiocb *iocb, struct iov_iter *iter, int dir)
return -EINVAL;
req = io->req;
if (!ublk_rq_has_data(req))
if (!blk_rq_has_data(req))
return -EINVAL;
} else {
req = __ublk_check_and_get_req(ub, q_id, tag, io);

2
drivers/block/virtio_blk.c
View File

@@ -689,6 +689,8 @@ static int virtblk_report_zones(struct gendisk *disk, sector_t sector,
nz = min_t(u64, virtio64_to_cpu(vblk->vdev, report->nr_zones),
nr_zones);
nz = min_t(u64, nz,
(buflen - sizeof(*report)) / sizeof(report->zones[0]));
if (!nz)
break;

6
drivers/md/dm-raid.c
View File

@@ -3831,6 +3831,7 @@ static void raid_presuspend(struct dm_target *ti)
* resume, raid_postsuspend() is too late.
*/
set_bit(RT_FLAG_RS_FROZEN, &rs->runtime_flags);
set_bit(MD_DM_SUSPENDING, &mddev->flags);
if (!reshape_interrupted(mddev))
return;
@@ -3847,13 +3848,16 @@ static void raid_presuspend(struct dm_target *ti)
static void raid_presuspend_undo(struct dm_target *ti)
{
struct raid_set *rs = ti->private;
struct mddev *mddev = &rs->md;
clear_bit(MD_DM_SUSPENDING, &mddev->flags);
clear_bit(RT_FLAG_RS_FROZEN, &rs->runtime_flags);
}
static void raid_postsuspend(struct dm_target *ti)
{
struct raid_set *rs = ti->private;
struct mddev *mddev = &rs->md;
if (!test_and_set_bit(RT_FLAG_RS_SUSPENDED, &rs->runtime_flags)) {
/*
@@ -3864,6 +3868,8 @@ static void raid_postsuspend(struct dm_target *ti)
mddev_suspend(&rs->md, false);
rs->md.ro = MD_RDONLY;
}
clear_bit(MD_DM_SUSPENDING, &mddev->flags);
}
static void attempt_restore_of_faulty_devices(struct raid_set *rs)

2
drivers/md/dm-rq.c
View File

@@ -462,7 +462,7 @@ static void dm_start_request(struct mapped_device *md, struct request *orig)
}
static int dm_mq_init_request(struct blk_mq_tag_set *set, struct request *rq,
unsigned int hctx_idx, unsigned int numa_node)
unsigned int hctx_idx, int numa_node)
{
struct mapped_device *md = set->driver_data;
struct dm_rq_target_io *tio = blk_mq_rq_to_pdu(rq);

32
drivers/md/md.c
View File

@@ -395,17 +395,24 @@ static bool is_suspended(struct mddev *mddev, struct bio *bio)
bool md_handle_request(struct mddev *mddev, struct bio *bio)
{
check_suspended:
if (is_suspended(mddev, bio)) {
/* Bail out if REQ_NOWAIT is set for the bio */
if (bio->bi_opf & REQ_NOWAIT) {
bio_wouldblock_error(bio);
return true;
if (unlikely(md_cloned_bio(mddev, bio))) {
/*
* This bio is an MD cloned bio and already holds an
* active_io reference, so percpu_ref_get() is safe here.
*/
percpu_ref_get(&mddev->active_io);
} else {
if (is_suspended(mddev, bio)) {
/* Bail out if REQ_NOWAIT is set for the bio */
if (bio->bi_opf & REQ_NOWAIT) {
bio_wouldblock_error(bio);
return true;
}
wait_event(mddev->sb_wait, !is_suspended(mddev, bio));
}
wait_event(mddev->sb_wait, !is_suspended(mddev, bio));
if (!percpu_ref_tryget_live(&mddev->active_io))
goto check_suspended;
}
if (!percpu_ref_tryget_live(&mddev->active_io))
goto check_suspended;
if (!mddev->pers->make_request(mddev, bio)) {
percpu_ref_put(&mddev->active_io);
if (mddev_is_dm(mddev) && mddev->pers->prepare_suspend)
@@ -4414,9 +4421,10 @@ raid_disks_store(struct mddev *mddev, const char *buf, size_t len)
err = mddev_suspend_and_lock(mddev);
if (err)
return err;
if (mddev->pers)
err = update_raid_disks(mddev, n);
else if (mddev->reshape_position != MaxSector) {
if (mddev->pers) {
if (n != mddev->raid_disks)
err = update_raid_disks(mddev, n);
} else if (mddev->reshape_position != MaxSector) {
struct md_rdev *rdev;
int olddisks = mddev->raid_disks - mddev->delta_disks;

7
drivers/md/md.h
View File

@@ -346,6 +346,7 @@ struct md_cluster_operations;
* @MD_HAS_SUPERBLOCK: There is persistence sb in member disks.
* @MD_FAILLAST_DEV: Allow last rdev to be removed.
* @MD_SERIALIZE_POLICY: Enforce write IO is not reordered, just used by raid1.
* @MD_DM_SUSPENDING: This DM raid device is suspending.
*
* change UNSUPPORTED_MDDEV_FLAGS for each array type if new flag is added
*/
@@ -365,6 +366,7 @@ enum mddev_flags {
MD_HAS_SUPERBLOCK,
MD_FAILLAST_DEV,
MD_SERIALIZE_POLICY,
MD_DM_SUSPENDING,
};
enum mddev_sb_flags {
@@ -1042,6 +1044,11 @@ void mddev_update_io_opt(struct mddev *mddev, unsigned int nr_stripes);
extern const struct block_device_operations md_fops;
static inline bool md_cloned_bio(struct mddev *mddev, struct bio *bio)
{
return bio->bi_pool == &mddev->io_clone_set;
}
/*
* MD devices can be used undeneath by DM, in which case ->gendisk is NULL.
*/

4
drivers/md/raid0.c
View File

@@ -14,6 +14,7 @@
#include <linux/seq_file.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/string_choices.h>
#include <trace/events/block.h>
#include "md.h"
#include "raid0.h"
@@ -43,7 +44,7 @@ static void dump_zones(struct mddev *mddev)
int raid_disks = conf->strip_zone[0].nb_dev;
pr_debug("md: RAID0 configuration for %s - %d zone%s\n",
mdname(mddev),
conf->nr_strip_zones, conf->nr_strip_zones==1?"":"s");
conf->nr_strip_zones, str_plural(conf->nr_strip_zones));
for (j = 0; j < conf->nr_strip_zones; j++) {
char line[200];
int len = 0;
@@ -392,6 +393,7 @@ static int raid0_set_limits(struct mddev *mddev)
lim.io_opt = lim.io_min * mddev->raid_disks;
lim.chunk_sectors = mddev->chunk_sectors;
lim.features |= BLK_FEAT_ATOMIC_WRITES;
lim.features |= BLK_FEAT_PCI_P2PDMA;
err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
if (err)
return err;

64
drivers/md/raid1.c
View File

@@ -1343,11 +1343,18 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
bool r1bio_existed = !!r1_bio;
/*
* If r1_bio is set, we are blocking the raid1d thread
* so there is a tiny risk of deadlock. So ask for
* An md cloned bio indicates we are in the error path.
* This is more reliable than checking r1_bio, which might
* be NULL even in the error path if a failed bio was split.
*/
bool err_path = md_cloned_bio(mddev, bio);
/*
* If we are in the error path, we are blocking the raid1d
* thread so there is a tiny risk of deadlock. So ask for
* emergency memory if needed.
*/
gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO;
gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO;
/*
* Still need barrier for READ in case that whole
@@ -1411,7 +1418,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
}
r1_bio->read_disk = rdisk;
if (!r1bio_existed) {
if (likely(!md_cloned_bio(mddev, bio))) {
md_account_bio(mddev, &bio);
r1_bio->master_bio = bio;
}
@@ -1596,8 +1603,10 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
* complexity of supporting that is not worth
* the benefit.
*/
if (bio->bi_opf & REQ_ATOMIC)
if (bio->bi_opf & REQ_ATOMIC) {
rdev_dec_pending(rdev, mddev);
goto err_handle;
}
good_sectors = first_bad - r1_bio->sector;
if (good_sectors < max_sectors)
@@ -2411,11 +2420,6 @@ static void fix_read_error(struct r1conf *conf, struct r1bio *r1_bio)
struct mddev *mddev = conf->mddev;
struct md_rdev *rdev = conf->mirrors[read_disk].rdev;
if (exceed_read_errors(mddev, rdev)) {
r1_bio->bios[r1_bio->read_disk] = IO_BLOCKED;
return;
}
while(sectors) {
int s = sectors;
int d = read_disk;
@@ -2627,35 +2631,36 @@ static void handle_write_finished(struct r1conf *conf, struct r1bio *r1_bio)
static void handle_read_error(struct r1conf *conf, struct r1bio *r1_bio)
{
struct md_rdev *rdev = conf->mirrors[r1_bio->read_disk].rdev;
struct bio *bio = r1_bio->bios[r1_bio->read_disk];
struct mddev *mddev = conf->mddev;
struct bio *bio;
struct md_rdev *rdev;
sector_t sector;
clear_bit(R1BIO_ReadError, &r1_bio->state);
/* we got a read error. Maybe the drive is bad. Maybe just
* the block and we can fix it.
* We freeze all other IO, and try reading the block from
* other devices. When we find one, we re-write
* and check it that fixes the read error.
* This is all done synchronously while the array is
* frozen
*/
bio = r1_bio->bios[r1_bio->read_disk];
bio_put(bio);
r1_bio->bios[r1_bio->read_disk] = NULL;
rdev = conf->mirrors[r1_bio->read_disk].rdev;
if (mddev->ro == 0
&& !test_bit(FailFast, &rdev->flags)) {
freeze_array(conf, 1);
fix_read_error(conf, r1_bio);
unfreeze_array(conf);
} else if (mddev->ro == 0 && test_bit(FailFast, &rdev->flags)) {
/*
* We got a read error. Maybe the drive is bad. Maybe just the block
* and we can fix it.
*
* If allowed, freeze all other IO, and try reading the block from other
* devices. If we find one, we re-write and check it that fixes the
* read error. This is all done synchronously while the array is
* frozen.
*/
if (mddev->ro) {
r1_bio->bios[r1_bio->read_disk] = IO_BLOCKED;
} else if (test_bit(FailFast, &rdev->flags)) {
md_error(mddev, rdev);
} else {
r1_bio->bios[r1_bio->read_disk] = IO_BLOCKED;
freeze_array(conf, 1);
if (exceed_read_errors(mddev, rdev))
r1_bio->bios[r1_bio->read_disk] = IO_BLOCKED;
else
fix_read_error(conf, r1_bio);
unfreeze_array(conf);
}
rdev_dec_pending(rdev, conf->mddev);
@@ -3208,6 +3213,7 @@ static int raid1_set_limits(struct mddev *mddev)
lim.max_hw_wzeroes_unmap_sectors = 0;
lim.logical_block_size = mddev->logical_block_size;
lim.features |= BLK_FEAT_ATOMIC_WRITES;
lim.features |= BLK_FEAT_PCI_P2PDMA;
err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
if (err)
return err;

30
drivers/md/raid10.c
View File

@@ -1146,7 +1146,7 @@ static bool regular_request_wait(struct mddev *mddev, struct r10conf *conf,
}
static void raid10_read_request(struct mddev *mddev, struct bio *bio,
struct r10bio *r10_bio, bool io_accounting)
struct r10bio *r10_bio)
{
struct r10conf *conf = mddev->private;
struct bio *read_bio;
@@ -1155,7 +1155,20 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
char b[BDEVNAME_SIZE];
int slot = r10_bio->read_slot;
struct md_rdev *err_rdev = NULL;
gfp_t gfp = GFP_NOIO;
/*
* An md cloned bio indicates we are in the error path.
* This is more reliable than checking slot, which might
* be -1 even in the error path if a failed bio was split.
*/
bool err_path = md_cloned_bio(mddev, bio);
/*
* If we are in the error path, we are blocking the raid10d
* thread so there is a tiny risk of deadlock. So ask for
* emergency memory if needed.
*/
gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO;
if (slot >= 0 && r10_bio->devs[slot].rdev) {
/*
@@ -1166,11 +1179,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
* we lose the device name in error messages.
*/
int disk;
/*
* As we are blocking raid10, it is a little safer to
* use __GFP_HIGH.
*/
gfp = GFP_NOIO | __GFP_HIGH;
disk = r10_bio->devs[slot].devnum;
err_rdev = conf->mirrors[disk].rdev;
@@ -1218,7 +1226,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
}
slot = r10_bio->read_slot;
if (io_accounting) {
if (likely(!md_cloned_bio(mddev, bio))) {
md_account_bio(mddev, &bio);
r10_bio->master_bio = bio;
}
@@ -1544,7 +1552,7 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
conf->geo.raid_disks);
if (bio_data_dir(bio) == READ)
raid10_read_request(mddev, bio, r10_bio, true);
raid10_read_request(mddev, bio, r10_bio);
else
raid10_write_request(mddev, bio, r10_bio);
}
@@ -1727,6 +1735,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
r10_bio->mddev = mddev;
r10_bio->state = 0;
r10_bio->sectors = 0;
r10_bio->read_slot = -1;
memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) * geo->raid_disks);
wait_blocked_dev(mddev, r10_bio);
@@ -2858,7 +2867,7 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio)
rdev_dec_pending(rdev, mddev);
r10_bio->state = 0;
raid10_read_request(mddev, r10_bio->master_bio, r10_bio, false);
raid10_read_request(mddev, r10_bio->master_bio, r10_bio);
/*
* allow_barrier after re-submit to ensure no sync io
* can be issued while regular io pending.
@@ -3941,6 +3950,7 @@ static int raid10_set_queue_limits(struct mddev *mddev)
lim.chunk_sectors = mddev->chunk_sectors;
lim.io_opt = lim.io_min * raid10_nr_stripes(conf);
lim.features |= BLK_FEAT_ATOMIC_WRITES;
lim.features |= BLK_FEAT_PCI_P2PDMA;
err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
if (err)
return err;

7
drivers/md/raid5.c
View File

@@ -6042,8 +6042,11 @@ static enum stripe_result make_stripe_request(struct mddev *mddev,
raid5_release_stripe(sh);
out:
if (ret == STRIPE_SCHEDULE_AND_RETRY && reshape_interrupted(mddev)) {
bi->bi_status = BLK_STS_RESOURCE;
ret = STRIPE_WAIT_RESHAPE;
if (!mddev_is_dm(mddev) ||
test_bit(MD_DM_SUSPENDING, &mddev->flags)) {
bi->bi_status = BLK_STS_RESOURCE;
ret = STRIPE_WAIT_RESHAPE;
}
pr_err_ratelimited("dm-raid456: io across reshape position while reshape can't make progress");
}
return ret;

2
drivers/mmc/core/queue.c
View File

@@ -208,7 +208,7 @@ static unsigned short mmc_get_max_segments(struct mmc_host *host)
}
static int mmc_mq_init_request(struct blk_mq_tag_set *set, struct request *req,
unsigned int hctx_idx, unsigned int numa_node)
unsigned int hctx_idx, int numa_node)
{
struct mmc_queue_req *mq_rq = req_to_mmc_queue_req(req);
struct mmc_queue *mq = set->driver_data;

2
drivers/mtd/ubi/block.c
View File

@@ -312,7 +312,7 @@ static blk_status_t ubiblock_queue_rq(struct blk_mq_hw_ctx *hctx,
static int ubiblock_init_request(struct blk_mq_tag_set *set,
struct request *req, unsigned int hctx_idx,
unsigned int numa_node)
int numa_node)
{
struct ubiblock_pdu *pdu = blk_mq_rq_to_pdu(req);

4
drivers/nvme/host/apple.c
View File

@@ -819,7 +819,7 @@ static int apple_nvme_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
static int apple_nvme_init_request(struct blk_mq_tag_set *set,
struct request *req, unsigned int hctx_idx,
unsigned int numa_node)
int numa_node)
{
struct apple_nvme_queue *q = set->driver_data;
struct apple_nvme *anv = queue_to_apple_nvme(q);
@@ -858,7 +858,7 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown)
* doing a safe shutdown.
*/
if (!dead && shutdown && freeze)
nvme_wait_freeze_timeout(&anv->ctrl, NVME_IO_TIMEOUT);
nvme_wait_freeze_timeout(&anv->ctrl);
nvme_quiesce_io_queues(&anv->ctrl);

74
drivers/nvme/host/core.c
View File

@@ -323,6 +323,7 @@ static void nvme_retry_req(struct request *req)
{
unsigned long delay = 0;
u16 crd;
struct nvme_ns *ns = req->q->queuedata;
/* The mask and shift result must be <= 3 */
crd = (nvme_req(req)->status & NVME_STATUS_CRD) >> 11;
@@ -330,6 +331,9 @@ static void nvme_retry_req(struct request *req)
delay = nvme_req(req)->ctrl->crdt[crd - 1] * 100;
nvme_req(req)->retries++;
if (ns)
atomic_long_inc(&ns->retries);
blk_mq_requeue_request(req, false);
blk_mq_delay_kick_requeue_list(req->q, delay);
}
@@ -434,11 +438,19 @@ static inline void nvme_end_req_zoned(struct request *req)
static inline void __nvme_end_req(struct request *req)
{
if (unlikely(nvme_req(req)->status && !(req->rq_flags & RQF_QUIET))) {
struct nvme_ns *ns = req->q->queuedata;
struct nvme_request *nr = nvme_req(req);
if (unlikely(nr->status && !(req->rq_flags & RQF_QUIET))) {
if (blk_rq_is_passthrough(req))
nvme_log_err_passthru(req);
else
nvme_log_error(req);
if (ns)
atomic_long_inc(&ns->errors);
else
atomic_long_inc(&nr->ctrl->errors);
}
nvme_end_req_zoned(req);
nvme_trace_bio_complete(req);
@@ -584,6 +596,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
case NVME_CTRL_NEW:
case NVME_CTRL_LIVE:
changed = true;
atomic_long_inc(&ctrl->nr_reset);
fallthrough;
default:
break;
@@ -729,10 +742,8 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)
struct nvme_ns *ns = req->q->disk->private_data;
logging_enabled = ns->head->passthru_err_log_enabled;
req->timeout = NVME_IO_TIMEOUT;
} else { /* no queuedata implies admin queue */
logging_enabled = nr->ctrl->passthru_err_log_enabled;
req->timeout = NVME_ADMIN_TIMEOUT;
}
if (!logging_enabled)
@@ -2263,7 +2274,7 @@ static int nvme_query_fdp_granularity(struct nvme_ctrl *ctrl,
}
n = le16_to_cpu(h->numfdpc) + 1;
if (fdp_idx > n) {
if (fdp_idx >= n) {
dev_warn(ctrl->device, "FDP index:%d out of range:%d\n",
fdp_idx, n);
/* Proceed without registering FDP streams */
@@ -2275,14 +2286,16 @@ static int nvme_query_fdp_granularity(struct nvme_ctrl *ctrl,
desc = log;
end = log + size - sizeof(*h);
for (i = 0; i < fdp_idx; i++) {
log += le16_to_cpu(desc->dsze);
desc = log;
if (log >= end) {
u16 dsze = le16_to_cpu(desc->dsze);
if (!dsze || log + dsze > end) {
dev_warn(ctrl->device,
"FDP invalid config descriptor list\n");
"FDP invalid config descriptor at index %d\n", i);
ret = 0;
goto out;
}
log += dsze;
desc = log;
}
if (le32_to_cpu(desc->nrg) > 1) {
@@ -2409,12 +2422,22 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
goto out;
}
if (id->lbaf[lbaf].ds < SECTOR_SHIFT ||
check_shl_overflow(le64_to_cpu(id->nsze),
id->lbaf[lbaf].ds - SECTOR_SHIFT,
&capacity)) {
dev_warn_once(ns->ctrl->device,
"invalid LBA data size %u, skipping namespace\n",
id->lbaf[lbaf].ds);
ret = -ENODEV;
goto out;
}
lim = queue_limits_start_update(ns->disk->queue);
memflags = blk_mq_freeze_queue(ns->disk->queue);
ns->head->lba_shift = id->lbaf[lbaf].ds;
ns->head->nuse = le64_to_cpu(id->nuse);
capacity = nvme_lba_to_sect(ns->head, le64_to_cpu(id->nsze));
nvme_set_ctrl_limits(ns->ctrl, &lim, false);
nvme_configure_metadata(ns->ctrl, ns->head, id, nvm, info);
nvme_set_chunk_sectors(ns, id, &lim);
@@ -2483,6 +2506,14 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
return ret;
}
static void nvme_stack_zone_resources(struct queue_limits *t,
const struct queue_limits *b)
{
t->max_open_zones = min_not_zero(t->max_open_zones, b->max_open_zones);
t->max_active_zones =
min_not_zero(t->max_active_zones, b->max_active_zones);
}
static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info)
{
bool unsupported = false;
@@ -2549,6 +2580,8 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info)
lim.io_opt = ns_lim->io_opt;
queue_limits_stack_bdev(&lim, ns->disk->part0, 0,
ns->head->disk->disk_name);
if (lim.features & BLK_FEAT_ZONED)
nvme_stack_zone_resources(&lim, ns_lim);
if (unsupported)
ns->head->disk->flags |= GENHD_FL_HIDDEN;
else
@@ -2559,7 +2592,7 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info)
set_capacity_and_notify(ns->head->disk, get_capacity(ns->disk));
set_disk_ro(ns->head->disk, nvme_ns_is_readonly(ns, info));
nvme_mpath_revalidate_paths(ns);
nvme_mpath_revalidate_paths(ns->head);
blk_mq_unfreeze_queue(ns->head->disk->queue, memflags);
}
@@ -3926,7 +3959,7 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
int ret = -ENOMEM;
#ifdef CONFIG_NVME_MULTIPATH
size += num_possible_nodes() * sizeof(struct nvme_ns *);
size += nr_node_ids * sizeof(struct nvme_ns *);
#endif
head = kzalloc(size, GFP_KERNEL);
@@ -4209,6 +4242,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
mutex_unlock(&ctrl->namespaces_lock);
goto out_unlink_ns;
}
blk_queue_rq_timeout(ns->queue, ctrl->io_timeout);
nvme_ns_add_to_ctrl_list(ns);
mutex_unlock(&ctrl->namespaces_lock);
synchronize_srcu(&ctrl->srcu);
@@ -4894,12 +4928,7 @@ int nvme_alloc_admin_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set,
if (ret)
return ret;
/*
* If a previous admin queue exists (e.g., from before a reset),
* put it now before allocating a new one to avoid orphaning it.
*/
if (ctrl->admin_q)
blk_put_queue(ctrl->admin_q);
WARN_ON_ONCE(ctrl->admin_q);
ctrl->admin_q = blk_mq_alloc_queue(set, NULL, NULL);
if (IS_ERR(ctrl->admin_q)) {
@@ -4937,10 +4966,8 @@ void nvme_remove_admin_tag_set(struct nvme_ctrl *ctrl)
*/
nvme_stop_keep_alive(ctrl);
blk_mq_destroy_queue(ctrl->admin_q);
if (ctrl->ops->flags & NVME_F_FABRICS) {
if (ctrl->fabrics_q)
blk_mq_destroy_queue(ctrl->fabrics_q);
blk_put_queue(ctrl->fabrics_q);
}
blk_mq_free_tag_set(ctrl->admin_tagset);
}
EXPORT_SYMBOL_GPL(nvme_remove_admin_tag_set);
@@ -5082,6 +5109,8 @@ static void nvme_free_ctrl(struct device *dev)
if (ctrl->admin_q)
blk_put_queue(ctrl->admin_q);
if (ctrl->fabrics_q)
blk_put_queue(ctrl->fabrics_q);
if (!subsys || ctrl->instance != subsys->instance)
ida_free(&nvme_instance_ida, ctrl->instance);
nvme_free_cels(ctrl);
@@ -5146,6 +5175,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
memset(&ctrl->ka_cmd, 0, sizeof(ctrl->ka_cmd));
ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive;
ctrl->ka_last_check_time = jiffies;
ctrl->admin_timeout = NVME_ADMIN_TIMEOUT;
ctrl->io_timeout = NVME_IO_TIMEOUT;
BUILD_BUG_ON(NVME_DSM_MAX_RANGES * sizeof(struct nvme_dsm_range) >
PAGE_SIZE);
@@ -5252,8 +5283,9 @@ void nvme_unfreeze(struct nvme_ctrl *ctrl)
}
EXPORT_SYMBOL_GPL(nvme_unfreeze);
int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl, long timeout)
int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl)
{
long timeout = ctrl->io_timeout;
struct nvme_ns *ns;
int srcu_idx;

5
drivers/nvme/host/fc.c
View File

@@ -2109,7 +2109,7 @@ __nvme_fc_init_request(struct nvme_fc_ctrl *ctrl,
static int
nvme_fc_init_request(struct blk_mq_tag_set *set, struct request *rq,
unsigned int hctx_idx, unsigned int numa_node)
unsigned int hctx_idx, int numa_node)
{
struct nvme_fc_ctrl *ctrl = to_fc_ctrl(set->driver_data);
struct nvme_fcp_op_w_sgl *op = blk_mq_rq_to_pdu(rq);
@@ -3148,6 +3148,8 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
goto out_term_aen_ops;
}
/* accumulate reconnect attempts before resetting it to zero */
atomic_long_add(ctrl->ctrl.nr_reconnects, &ctrl->ctrl.acc_reconnects);
ctrl->ctrl.nr_reconnects = 0;
nvme_start_ctrl(&ctrl->ctrl);
@@ -3470,6 +3472,7 @@ nvme_fc_alloc_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
ctrl->ctrl.opts = opts;
ctrl->ctrl.nr_reconnects = 0;
atomic_long_set(&ctrl->ctrl.acc_reconnects, 0);
INIT_LIST_HEAD(&ctrl->ctrl_list);
ctrl->lport = lport;
ctrl->rport = rport;

9
drivers/nvme/host/ioctl.c
View File

@@ -102,8 +102,17 @@ static struct request *nvme_alloc_user_request(struct request_queue *q,
struct nvme_command *cmd, blk_opf_t rq_flags,
blk_mq_req_flags_t blk_flags)
{
struct nvme_ns *ns = q->queuedata;
struct request *req;
/*
* The NVME_MPATH flag is set only for IO commands sent to a namespace
* with a multipath enabled head. The request is not eligible for
* failover as passthrough requests also append REQ_FAILFAST_DRIVER.
*/
if (ns && nvme_ns_head_multipath(ns->head))
rq_flags |= REQ_NVME_MPATH;
req = blk_mq_alloc_request(q, nvme_req_op(cmd) | rq_flags, blk_flags);
if (IS_ERR(req))
return req;

144
drivers/nvme/host/multipath.c
View File

@@ -73,19 +73,29 @@ static const char *nvme_iopolicy_names[] = {
static int iopolicy = NVME_IOPOLICY_NUMA;
static int nvme_iopolicy_parse(const char *str)
{
int i;
for (i = 0; i < ARRAY_SIZE(nvme_iopolicy_names); i++) {
if (sysfs_streq(str, nvme_iopolicy_names[i]))
return i;
}
return -EINVAL;
}
static int nvme_set_iopolicy(const char *val, const struct kernel_param *kp)
{
int policy;
if (!val)
return -EINVAL;
if (!strncmp(val, "numa", 4))
iopolicy = NVME_IOPOLICY_NUMA;
else if (!strncmp(val, "round-robin", 11))
iopolicy = NVME_IOPOLICY_RR;
else if (!strncmp(val, "queue-depth", 11))
iopolicy = NVME_IOPOLICY_QD;
else
return -EINVAL;
policy = nvme_iopolicy_parse(val);
if (policy < 0)
return policy;
iopolicy = policy;
return 0;
}
@@ -142,6 +152,7 @@ void nvme_failover_req(struct request *req)
struct bio *bio;
nvme_mpath_clear_current_path(ns);
atomic_long_inc(&ns->failover);
/*
* If we got back an ANA error, we know the controller is alive but not
@@ -175,9 +186,12 @@ void nvme_mpath_start_request(struct request *rq)
nvme_req(rq)->flags |= NVME_MPATH_CNT_ACTIVE;
}
if (!blk_queue_io_stat(disk->queue) || blk_rq_is_passthrough(rq) ||
if (!blk_queue_io_stat(disk->queue) ||
(nvme_req(rq)->flags & NVME_MPATH_IO_STATS))
return;
if (blk_rq_is_passthrough(rq) &&
!blk_rq_passthrough_stats(rq, disk->queue))
return;
nvme_req(rq)->flags |= NVME_MPATH_IO_STATS;
nvme_req(rq)->start_time = bdev_start_io_acct(disk->part0, req_op(rq),
@@ -254,10 +268,10 @@ void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl)
srcu_read_unlock(&ctrl->srcu, srcu_idx);
}
void nvme_mpath_revalidate_paths(struct nvme_ns *ns)
void nvme_mpath_revalidate_paths(struct nvme_ns_head *head)
{
struct nvme_ns_head *head = ns->head;
sector_t capacity = get_capacity(head->disk);
struct nvme_ns *ns;
int node;
int srcu_idx;
@@ -511,6 +525,12 @@ static void nvme_ns_head_submit_bio(struct bio *bio)
ns = nvme_find_path(head);
if (likely(ns)) {
bio_set_dev(bio, ns->disk->part0);
/*
* Use BIO_REMAPPED to skip bio_check_eod() when this bio
* enters submit_bio_noacct() for the per-path device. The EOD
* check already passed on the multipath head.
*/
bio_set_flag(bio, BIO_REMAPPED);
bio->bi_opf |= REQ_NVME_MPATH;
trace_block_bio_remap(bio, disk_devt(ns->head->disk),
bio->bi_iter.bi_sector);
@@ -521,10 +541,12 @@ static void nvme_ns_head_submit_bio(struct bio *bio)
spin_lock_irq(&head->requeue_lock);
bio_list_add(&head->requeue_list, bio);
spin_unlock_irq(&head->requeue_lock);
atomic_long_inc(&head->io_requeue_no_usable_path_count);
} else {
dev_warn_ratelimited(dev, "no available path - failing I/O\n");
bio_io_error(bio);
atomic_long_inc(&head->io_fail_no_available_path_count);
}
srcu_read_unlock(&head->srcu, srcu_idx);
@@ -730,7 +752,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
blk_set_stacking_limits(&lim);
lim.dma_alignment = 3;
lim.features |= BLK_FEAT_IO_STAT | BLK_FEAT_NOWAIT |
BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES;
BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES | BLK_FEAT_PCI_P2PDMA;
if (head->ids.csi == NVME_CSI_ZNS)
lim.features |= BLK_FEAT_ZONED;
@@ -1039,16 +1061,14 @@ static ssize_t nvme_subsys_iopolicy_store(struct device *dev,
{
struct nvme_subsystem *subsys =
container_of(dev, struct nvme_subsystem, dev);
int i;
int policy;
for (i = 0; i < ARRAY_SIZE(nvme_iopolicy_names); i++) {
if (sysfs_streq(buf, nvme_iopolicy_names[i])) {
nvme_subsys_iopolicy_update(subsys, i);
return count;
}
}
policy = nvme_iopolicy_parse(buf);
if (policy < 0)
return policy;
return -EINVAL;
nvme_subsys_iopolicy_update(subsys, policy);
return count;
}
SUBSYS_ATTR_RW(iopolicy, S_IRUGO | S_IWUSR,
nvme_subsys_iopolicy_show, nvme_subsys_iopolicy_store);
@@ -1151,6 +1171,90 @@ static ssize_t delayed_removal_secs_store(struct device *dev,
DEVICE_ATTR_RW(delayed_removal_secs);
static ssize_t multipath_failover_count_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
return sysfs_emit(buf, "%lu\n", atomic_long_read(&ns->failover));
}
static ssize_t multipath_failover_count_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
unsigned long failover;
int ret;
struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
ret = kstrtoul(buf, 0, &failover);
if (ret)
return -EINVAL;
atomic_long_set(&ns->failover, failover);
return count;
}
DEVICE_ATTR_RW(multipath_failover_count);
static ssize_t io_requeue_no_usable_path_count_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct gendisk *disk = dev_to_disk(dev);
struct nvme_ns_head *head = disk->private_data;
return sysfs_emit(buf, "%lu\n",
atomic_long_read(&head->io_requeue_no_usable_path_count));
}
static ssize_t io_requeue_no_usable_path_count_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
int err;
unsigned long requeue_cnt;
struct gendisk *disk = dev_to_disk(dev);
struct nvme_ns_head *head = disk->private_data;
err = kstrtoul(buf, 0, &requeue_cnt);
if (err)
return -EINVAL;
atomic_long_set(&head->io_requeue_no_usable_path_count, requeue_cnt);
return count;
}
DEVICE_ATTR_RW(io_requeue_no_usable_path_count);
static ssize_t io_fail_no_available_path_count_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct gendisk *disk = dev_to_disk(dev);
struct nvme_ns_head *head = disk->private_data;
return sysfs_emit(buf, "%lu\n",
atomic_long_read(&head->io_fail_no_available_path_count));
}
static ssize_t io_fail_no_available_path_count_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
int err;
unsigned long fail_cnt;
struct gendisk *disk = dev_to_disk(dev);
struct nvme_ns_head *head = disk->private_data;
err = kstrtoul(buf, 0, &fail_cnt);
if (err)
return -EINVAL;
atomic_long_set(&head->io_fail_no_available_path_count, fail_cnt);
return count;
}
DEVICE_ATTR_RW(io_fail_no_available_path_count);
static int nvme_lookup_ana_group_desc(struct nvme_ctrl *ctrl,
struct nvme_ana_group_desc *desc, void *data)
{

21
drivers/nvme/host/nvme.h
View File

@@ -370,6 +370,8 @@ struct nvme_ctrl {
u16 mtfa;
u32 ctrl_config;
u32 queue_count;
u32 admin_timeout;
u32 io_timeout;
u64 cap;
u32 max_hw_sectors;
@@ -413,6 +415,8 @@ struct nvme_ctrl {
unsigned long ka_last_check_time;
struct work_struct fw_act_work;
unsigned long events;
atomic_long_t errors;
atomic_long_t nr_reset;
#ifdef CONFIG_NVME_MULTIPATH
/* asymmetric namespace access: */
@@ -454,6 +458,8 @@ struct nvme_ctrl {
u16 icdoff;
u16 maxcmd;
int nr_reconnects;
/* accumulate reconenct attempts, as nr_reconnects can reset to zero */
atomic_long_t acc_reconnects;
unsigned long flags;
struct nvmf_ctrl_options *opts;
@@ -563,6 +569,8 @@ struct nvme_ns_head {
unsigned long flags;
struct delayed_work remove_work;
unsigned int delayed_removal_secs;
atomic_long_t io_requeue_no_usable_path_count;
atomic_long_t io_fail_no_available_path_count;
#define NVME_NSHEAD_DISK_LIVE 0
#define NVME_NSHEAD_QUEUE_IF_NO_PATH 1
struct nvme_ns __rcu *current_path[];
@@ -589,7 +597,10 @@ struct nvme_ns {
#ifdef CONFIG_NVME_MULTIPATH
enum nvme_ana_state ana_state;
u32 ana_grpid;
atomic_long_t failover;
#endif
atomic_long_t retries;
atomic_long_t errors;
struct list_head siblings;
struct kref kref;
struct nvme_ns_head *head;
@@ -900,7 +911,7 @@ void nvme_sync_queues(struct nvme_ctrl *ctrl);
void nvme_sync_io_queues(struct nvme_ctrl *ctrl);
void nvme_unfreeze(struct nvme_ctrl *ctrl);
void nvme_wait_freeze(struct nvme_ctrl *ctrl);
int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl, long timeout);
int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl);
void nvme_start_freeze(struct nvme_ctrl *ctrl);
static inline enum req_op nvme_req_op(struct nvme_command *cmd)
@@ -1012,6 +1023,7 @@ extern const struct attribute_group nvme_ns_mpath_attr_group;
extern const struct pr_ops nvme_pr_ops;
extern const struct block_device_operations nvme_ns_head_ops;
extern const struct attribute_group nvme_dev_attrs_group;
extern const struct attribute_group nvme_dev_diag_attrs_group;
extern const struct attribute_group *nvme_subsys_attrs_groups[];
extern const struct attribute_group *nvme_dev_attr_groups[];
extern const struct block_device_operations nvme_bdev_ops;
@@ -1041,7 +1053,7 @@ void nvme_mpath_update(struct nvme_ctrl *ctrl);
void nvme_mpath_uninit(struct nvme_ctrl *ctrl);
void nvme_mpath_stop(struct nvme_ctrl *ctrl);
bool nvme_mpath_clear_current_path(struct nvme_ns *ns);
void nvme_mpath_revalidate_paths(struct nvme_ns *ns);
void nvme_mpath_revalidate_paths(struct nvme_ns_head *head);
void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl);
void nvme_mpath_remove_disk(struct nvme_ns_head *head);
void nvme_mpath_start_request(struct request *rq);
@@ -1061,6 +1073,9 @@ extern struct device_attribute dev_attr_ana_state;
extern struct device_attribute dev_attr_queue_depth;
extern struct device_attribute dev_attr_numa_nodes;
extern struct device_attribute dev_attr_delayed_removal_secs;
extern struct device_attribute dev_attr_multipath_failover_count;
extern struct device_attribute dev_attr_io_requeue_no_usable_path_count;
extern struct device_attribute dev_attr_io_fail_no_available_path_count;
extern struct device_attribute subsys_attr_iopolicy;
static inline bool nvme_disk_is_ns_head(struct gendisk *disk)
@@ -1106,7 +1121,7 @@ static inline bool nvme_mpath_clear_current_path(struct nvme_ns *ns)
{
return false;
}
static inline void nvme_mpath_revalidate_paths(struct nvme_ns *ns)
static inline void nvme_mpath_revalidate_paths(struct nvme_ns_head *head)
{
}
static inline void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl)

16
drivers/nvme/host/pci.c
View File

@@ -587,11 +587,16 @@ static bool nvme_dbbuf_update_and_check_event(u16 value, __le32 *dbbuf_db,
}
static struct nvme_descriptor_pools *
nvme_setup_descriptor_pools(struct nvme_dev *dev, unsigned numa_node)
nvme_setup_descriptor_pools(struct nvme_dev *dev, int numa_node)
{
struct nvme_descriptor_pools *pools = &dev->descriptor_pools[numa_node];
struct nvme_descriptor_pools *pools;
size_t small_align = NVME_SMALL_POOL_SIZE;
if (numa_node == NUMA_NO_NODE)
numa_node = 0;
pools = &dev->descriptor_pools[numa_node];
if (pools->small)
return pools; /* already initialized */
@@ -660,7 +665,7 @@ static int nvme_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
static int nvme_pci_init_request(struct blk_mq_tag_set *set,
struct request *req, unsigned int hctx_idx,
unsigned int numa_node)
int numa_node)
{
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
@@ -2838,6 +2843,7 @@ static const struct attribute_group nvme_pci_dev_attrs_group = {
static const struct attribute_group *nvme_pci_dev_attr_groups[] = {
&nvme_dev_attrs_group,
&nvme_pci_dev_attrs_group,
&nvme_dev_diag_attrs_group,
NULL,
};
@@ -3122,7 +3128,7 @@ static bool __nvme_delete_io_queues(struct nvme_dev *dev, u8 opcode)
unsigned long timeout;
retry:
timeout = NVME_ADMIN_TIMEOUT;
timeout = dev->ctrl.admin_timeout;
while (nr_queues > 0) {
if (nvme_delete_queue(&dev->queues[nr_queues], opcode))
break;
@@ -3304,7 +3310,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)
* if doing a safe shutdown.
*/
if (!dead && shutdown)
nvme_wait_freeze_timeout(&dev->ctrl, NVME_IO_TIMEOUT);
nvme_wait_freeze_timeout(&dev->ctrl);
}
nvme_quiesce_io_queues(&dev->ctrl);

6
drivers/nvme/host/rdma.c
View File

@@ -292,7 +292,7 @@ static void nvme_rdma_exit_request(struct blk_mq_tag_set *set,
static int nvme_rdma_init_request(struct blk_mq_tag_set *set,
struct request *rq, unsigned int hctx_idx,
unsigned int numa_node)
int numa_node)
{
struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(set->driver_data);
struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
@@ -888,7 +888,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
if (!new) {
nvme_start_freeze(&ctrl->ctrl);
nvme_unquiesce_io_queues(&ctrl->ctrl);
if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) {
if (!nvme_wait_freeze_timeout(&ctrl->ctrl)) {
/*
* If we timed out waiting for freeze we are likely to
* be stuck. Fail the controller initialization just
@@ -1110,6 +1110,8 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
dev_info(ctrl->ctrl.device, "Successfully reconnected (%d attempts)\n",
ctrl->ctrl.nr_reconnects);
/* accumulate reconnect attempts before resetting it to zero */
atomic_long_add(ctrl->ctrl.nr_reconnects, &ctrl->ctrl.acc_reconnects);
ctrl->ctrl.nr_reconnects = 0;
return;

311
drivers/nvme/host/sysfs.c
View File

@@ -6,6 +6,7 @@
*/
#include <linux/nvme-auth.h>
#include <linux/blkdev.h>
#include "nvme.h"
#include "fabrics.h"
@@ -335,14 +336,7 @@ static bool multipath_sysfs_group_visible(struct kobject *kobj)
return nvme_disk_is_ns_head(dev_to_disk(dev));
}
static bool multipath_sysfs_attr_visible(struct kobject *kobj,
struct attribute *attr, int n)
{
return false;
}
DEFINE_SYSFS_GROUP_VISIBLE(multipath_sysfs)
DEFINE_SIMPLE_SYSFS_GROUP_VISIBLE(multipath_sysfs)
const struct attribute_group nvme_ns_mpath_attr_group = {
.name = "multipath",
@@ -351,11 +345,114 @@ const struct attribute_group nvme_ns_mpath_attr_group = {
};
#endif
static ssize_t command_retries_count_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
return sysfs_emit(buf, "%lu\n", atomic_long_read(&ns->retries));
}
static ssize_t command_retries_count_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
unsigned long retries;
int err;
struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
err = kstrtoul(buf, 0, &retries);
if (err)
return -EINVAL;
atomic_long_set(&ns->retries, retries);
return count;
}
static DEVICE_ATTR_RW(command_retries_count);
static ssize_t nvme_io_errors_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
return sysfs_emit(buf, "%lu\n", atomic_long_read(&ns->errors));
}
static ssize_t nvme_io_errors_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
unsigned long errors;
int err;
struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
err = kstrtoul(buf, 0, &errors);
if (err)
return -EINVAL;
atomic_long_set(&ns->errors, errors);
return count;
}
struct device_attribute dev_attr_io_errors =
__ATTR(command_error_count, 0644,
nvme_io_errors_show, nvme_io_errors_store);
static struct attribute *nvme_ns_diag_attrs[] = {
&dev_attr_command_retries_count.attr,
&dev_attr_io_errors.attr,
#ifdef CONFIG_NVME_MULTIPATH
&dev_attr_multipath_failover_count.attr,
&dev_attr_io_requeue_no_usable_path_count.attr,
&dev_attr_io_fail_no_available_path_count.attr,
#endif
NULL,
};
static umode_t nvme_ns_diag_attrs_are_visible(struct kobject *kobj,
struct attribute *a, int n)
{
struct device *dev = container_of(kobj, struct device, kobj);
if (a == &dev_attr_command_retries_count.attr) {
if (nvme_disk_is_ns_head(dev_to_disk(dev)))
return 0;
}
if (a == &dev_attr_io_errors.attr) {
struct gendisk *disk = dev_to_disk(dev);
if (nvme_disk_is_ns_head(disk))
return 0;
}
#ifdef CONFIG_NVME_MULTIPATH
if (a == &dev_attr_multipath_failover_count.attr) {
if (nvme_disk_is_ns_head(dev_to_disk(dev)))
return 0;
}
if (a == &dev_attr_io_requeue_no_usable_path_count.attr) {
if (!nvme_disk_is_ns_head(dev_to_disk(dev)))
return 0;
}
if (a == &dev_attr_io_fail_no_available_path_count.attr) {
if (!nvme_disk_is_ns_head(dev_to_disk(dev)))
return 0;
}
#endif
return a->mode;
}
const struct attribute_group nvme_ns_diag_attr_group = {
.name = "diag",
.attrs = nvme_ns_diag_attrs,
.is_visible = nvme_ns_diag_attrs_are_visible,
};
const struct attribute_group *nvme_ns_attr_groups[] = {
&nvme_ns_attr_group,
#ifdef CONFIG_NVME_MULTIPATH
&nvme_ns_mpath_attr_group,
#endif
&nvme_ns_diag_attr_group,
NULL,
};
@@ -623,6 +720,92 @@ static ssize_t quirks_show(struct device *dev, struct device_attribute *attr,
}
static DEVICE_ATTR_RO(quirks);
static ssize_t nvme_admin_timeout_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
return sysfs_emit(buf, "%u\n",
jiffies_to_msecs(ctrl->admin_timeout));
}
static ssize_t nvme_admin_timeout_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
u32 timeout;
int err;
/*
* Wait until the controller reaches the LIVE state to be sure that
* admin_q and fabrics_q are properly initialized.
*/
if (!test_bit(NVME_CTRL_STARTED_ONCE, &ctrl->flags))
return -EBUSY;
err = kstrtou32(buf, 10, &timeout);
if (err || !timeout)
return -EINVAL;
ctrl->admin_timeout = msecs_to_jiffies(timeout);
blk_queue_rq_timeout(ctrl->admin_q, ctrl->admin_timeout);
if (ctrl->fabrics_q)
blk_queue_rq_timeout(ctrl->fabrics_q, ctrl->admin_timeout);
return count;
}
static DEVICE_ATTR(admin_timeout, S_IRUGO | S_IWUSR,
nvme_admin_timeout_show, nvme_admin_timeout_store);
static ssize_t nvme_io_timeout_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
return sysfs_emit(buf, "%u\n", jiffies_to_msecs(ctrl->io_timeout));
}
static ssize_t nvme_io_timeout_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
struct nvme_ns *ns;
u32 timeout;
int err;
/*
* Wait until the controller reaches the LIVE state to be sure that
* connect_q is properly initialized.
*/
if (!test_bit(NVME_CTRL_STARTED_ONCE, &ctrl->flags))
return -EBUSY;
err = kstrtou32(buf, 10, &timeout);
if (err || !timeout)
return -EINVAL;
/* Take the namespaces_lock to avoid racing against nvme_alloc_ns() */
mutex_lock(&ctrl->namespaces_lock);
ctrl->io_timeout = msecs_to_jiffies(timeout);
list_for_each_entry(ns, &ctrl->namespaces, list)
blk_queue_rq_timeout(ns->queue, ctrl->io_timeout);
mutex_unlock(&ctrl->namespaces_lock);
if (ctrl->connect_q)
blk_queue_rq_timeout(ctrl->connect_q, ctrl->io_timeout);
return count;
}
static DEVICE_ATTR(io_timeout, S_IRUGO | S_IWUSR,
nvme_io_timeout_show, nvme_io_timeout_store);
#ifdef CONFIG_NVME_HOST_AUTH
static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
struct device_attribute *attr, char *buf)
@@ -765,6 +948,8 @@ static struct attribute *nvme_dev_attrs[] = {
&dev_attr_cntrltype.attr,
&dev_attr_dctype.attr,
&dev_attr_quirks.attr,
&dev_attr_admin_timeout.attr,
&dev_attr_io_timeout.attr,
#ifdef CONFIG_NVME_HOST_AUTH
&dev_attr_dhchap_secret.attr,
&dev_attr_dhchap_ctrl_secret.attr,
@@ -937,11 +1122,121 @@ static const struct attribute_group nvme_tls_attrs_group = {
};
#endif
static ssize_t nvme_adm_errors_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
return sysfs_emit(buf, "%lu\n",
(unsigned long)atomic_long_read(&ctrl->errors));
}
static ssize_t nvme_adm_errors_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
unsigned long errors;
int err;
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
err = kstrtoul(buf, 0, &errors);
if (err)
return -EINVAL;
atomic_long_set(&ctrl->errors, errors);
return count;
}
struct device_attribute dev_attr_adm_errors =
__ATTR(command_error_count, 0644,
nvme_adm_errors_show, nvme_adm_errors_store);
static ssize_t reset_count_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
return sysfs_emit(buf, "%lu\n", atomic_long_read(&ctrl->nr_reset));
}
static ssize_t reset_count_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
int err;
unsigned long reset_cnt;
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
err = kstrtoul(buf, 0, &reset_cnt);
if (err)
return -EINVAL;
atomic_long_set(&ctrl->nr_reset, reset_cnt);
return count;
}
static ssize_t reconnect_count_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
return sysfs_emit(buf, "%lu\n",
atomic_long_read(&ctrl->acc_reconnects) +
ctrl->nr_reconnects);
}
static ssize_t reconnect_count_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
int err;
unsigned long reconnect_cnt;
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
err = kstrtoul(buf, 0, &reconnect_cnt);
if (err)
return -EINVAL;
atomic_long_set(&ctrl->acc_reconnects, reconnect_cnt);
return count;
}
static DEVICE_ATTR_RW(reconnect_count);
static DEVICE_ATTR_RW(reset_count);
static struct attribute *nvme_dev_diag_attrs[] = {
&dev_attr_adm_errors.attr,
&dev_attr_reset_count.attr,
&dev_attr_reconnect_count.attr,
NULL,
};
static umode_t nvme_dev_diag_attrs_are_visible(struct kobject *kobj,
struct attribute *a, int n)
{
struct device *dev = container_of(kobj, struct device, kobj);
struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
if (a == &dev_attr_reconnect_count.attr && !ctrl->opts)
return 0;
return a->mode;
}
const struct attribute_group nvme_dev_diag_attrs_group = {
.name = "diag",
.attrs = nvme_dev_diag_attrs,
.is_visible = nvme_dev_diag_attrs_are_visible,
};
EXPORT_SYMBOL_GPL(nvme_dev_diag_attrs_group);
const struct attribute_group *nvme_dev_attr_groups[] = {
&nvme_dev_attrs_group,
#ifdef CONFIG_NVME_TCP_TLS
&nvme_tls_attrs_group,
#endif
&nvme_dev_diag_attrs_group,
NULL,
};

127
drivers/nvme/host/tcp.c
View File

@@ -56,44 +56,6 @@ MODULE_PARM_DESC(tls_handshake_timeout,
static atomic_t nvme_tcp_cpu_queues[NR_CPUS];
#ifdef CONFIG_DEBUG_LOCK_ALLOC
/* lockdep can detect a circular dependency of the form
* sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock
* because dependencies are tracked for both nvme-tcp and user contexts. Using
* a separate class prevents lockdep from conflating nvme-tcp socket use with
* user-space socket API use.
*/
static struct lock_class_key nvme_tcp_sk_key[2];
static struct lock_class_key nvme_tcp_slock_key[2];
static void nvme_tcp_reclassify_socket(struct socket *sock)
{
struct sock *sk = sock->sk;
if (WARN_ON_ONCE(!sock_allow_reclassification(sk)))
return;
switch (sk->sk_family) {
case AF_INET:
sock_lock_init_class_and_name(sk, "slock-AF_INET-NVME",
&nvme_tcp_slock_key[0],
"sk_lock-AF_INET-NVME",
&nvme_tcp_sk_key[0]);
break;
case AF_INET6:
sock_lock_init_class_and_name(sk, "slock-AF_INET6-NVME",
&nvme_tcp_slock_key[1],
"sk_lock-AF_INET6-NVME",
&nvme_tcp_sk_key[1]);
break;
default:
WARN_ON_ONCE(1);
}
}
#else
static void nvme_tcp_reclassify_socket(struct socket *sock) { }
#endif
enum nvme_tcp_send_state {
NVME_TCP_SEND_CMD_PDU = 0,
NVME_TCP_SEND_H2C_PDU,
@@ -180,6 +142,11 @@ struct nvme_tcp_queue {
void (*state_change)(struct sock *);
void (*data_ready)(struct sock *);
void (*write_space)(struct sock *);
#ifdef CONFIG_DEBUG_LOCK_ALLOC
struct lock_class_key nvme_tcp_sk_key;
struct lock_class_key nvme_tcp_slock_key;
#endif
};
struct nvme_tcp_ctrl {
@@ -207,6 +174,39 @@ static const struct blk_mq_ops nvme_tcp_mq_ops;
static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
static int nvme_tcp_try_send(struct nvme_tcp_queue *queue);
#ifdef CONFIG_DEBUG_LOCK_ALLOC
/* lockdep can detect a circular dependency of the form
* sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock
* because dependencies are tracked for both nvme-tcp and user contexts. Using
* a separate class prevents lockdep from conflating nvme-tcp socket use with
* user-space socket API use.
*/
static void nvme_tcp_reclassify_socket(struct nvme_tcp_queue *queue)
{
struct sock *sk = queue->sock->sk;
if (WARN_ON_ONCE(!sock_allow_reclassification(sk)))
return;
switch (sk->sk_family) {
case AF_INET:
sock_lock_init_class_and_name(sk, "slock-AF_INET-NVME",
&queue->nvme_tcp_slock_key,
"sk_lock-AF_INET-NVME",
&queue->nvme_tcp_sk_key);
break;
case AF_INET6:
sock_lock_init_class_and_name(sk, "slock-AF_INET6-NVME",
&queue->nvme_tcp_slock_key,
"sk_lock-AF_INET6-NVME",
&queue->nvme_tcp_sk_key);
break;
default:
WARN_ON_ONCE(1);
}
}
#endif
static inline struct nvme_tcp_ctrl *to_tcp_ctrl(struct nvme_ctrl *ctrl)
{
return container_of(ctrl, struct nvme_tcp_ctrl, ctrl);
@@ -340,32 +340,25 @@ static void nvme_tcp_init_iter(struct nvme_tcp_request *req,
unsigned int dir)
{
struct request *rq = blk_mq_rq_from_pdu(req);
struct bio_vec *vec;
unsigned int size;
int nr_bvec;
size_t offset;
if (rq->rq_flags & RQF_SPECIAL_PAYLOAD) {
vec = &rq->special_vec;
nr_bvec = 1;
size = blk_rq_payload_bytes(rq);
offset = 0;
iov_iter_bvec(&req->iter, dir, &rq->special_vec, 1,
blk_rq_payload_bytes(rq));
req->iter.iov_offset = 0;
} else {
struct bio *bio = req->curr_bio;
struct bvec_iter bi;
struct bio_vec bv;
int nr_bvec = 0;
vec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter);
nr_bvec = 0;
bio_for_each_bvec(bv, bio, bi) {
bio_for_each_bvec(bv, bio, bi)
nr_bvec++;
}
size = bio->bi_iter.bi_size;
offset = bio->bi_iter.bi_bvec_done;
}
iov_iter_bvec(&req->iter, dir, vec, nr_bvec, size);
req->iter.iov_offset = offset;
iov_iter_bvec(&req->iter, dir,
__bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter), nr_bvec,
bio->bi_iter.bi_size);
req->iter.iov_offset = bio->bi_iter.bi_bvec_done;
}
}
static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req,
@@ -548,7 +541,7 @@ static void nvme_tcp_exit_request(struct blk_mq_tag_set *set,
static int nvme_tcp_init_request(struct blk_mq_tag_set *set,
struct request *rq, unsigned int hctx_idx,
unsigned int numa_node)
int numa_node)
{
struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(set->driver_data);
struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
@@ -1468,6 +1461,11 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
kfree(queue->pdu);
mutex_destroy(&queue->send_mutex);
mutex_destroy(&queue->queue_lock);
#ifdef CONFIG_DEBUG_LOCK_ALLOC
lockdep_unregister_key(&queue->nvme_tcp_sk_key);
lockdep_unregister_key(&queue->nvme_tcp_slock_key);
#endif
}
static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)
@@ -1813,7 +1811,12 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
}
sk_net_refcnt_upgrade(queue->sock->sk);
nvme_tcp_reclassify_socket(queue->sock);
#ifdef CONFIG_DEBUG_LOCK_ALLOC
lockdep_register_key(&queue->nvme_tcp_sk_key);
lockdep_register_key(&queue->nvme_tcp_slock_key);
nvme_tcp_reclassify_socket(queue);
#endif
/* Single syn retry */
tcp_sock_set_syncnt(queue->sock->sk, 1);
@@ -1918,6 +1921,10 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
/* Use sync variant - see nvme_tcp_free_queue() for explanation */
__fput_sync(queue->sock->file);
queue->sock = NULL;
#ifdef CONFIG_DEBUG_LOCK_ALLOC
lockdep_unregister_key(&queue->nvme_tcp_sk_key);
lockdep_unregister_key(&queue->nvme_tcp_slock_key);
#endif
err_destroy_mutex:
mutex_destroy(&queue->send_mutex);
mutex_destroy(&queue->queue_lock);
@@ -2208,7 +2215,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
if (!new) {
nvme_start_freeze(ctrl);
nvme_unquiesce_io_queues(ctrl);
if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) {
if (!nvme_wait_freeze_timeout(ctrl)) {
/*
* If we timed out waiting for freeze we are likely to
* be stuck. Fail the controller initialization just
@@ -2475,6 +2482,8 @@ static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work)
dev_info(ctrl->device, "Successfully reconnected (attempt %d/%d)\n",
ctrl->nr_reconnects, ctrl->opts->max_reconnects);
/* accumulate reconnect attempts before resetting it to zero */
atomic_long_add(ctrl->nr_reconnects, &ctrl->acc_reconnects);
ctrl->nr_reconnects = 0;
return;
@@ -3053,6 +3062,8 @@ static int __init nvme_tcp_init_module(void)
if (wq_unbound)
wq_flags |= WQ_UNBOUND;
else
wq_flags |= WQ_PERCPU;
nvme_tcp_wq = alloc_workqueue("nvme_tcp_wq", wq_flags, 0);
if (!nvme_tcp_wq)

23
drivers/nvme/target/discovery.c
View File

@@ -166,6 +166,7 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req)
u64 offset = nvmet_get_log_page_offset(req->cmd);
size_t data_len = nvmet_get_log_page_len(req->cmd);
size_t alloc_len;
size_t copy_len;
struct nvmet_subsys_link *p;
struct nvmet_port *r;
u32 numrec = 0;
@@ -242,7 +243,27 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req)
up_read(&nvmet_config_sem);
status = nvmet_copy_to_sgl(req, 0, buffer + offset, data_len);
/*
* Validate the host-supplied log page offset before copying out.
* Without this check, the host controls a 64-bit byte offset into
* a small kzalloc'd buffer: a value past the log page lets the
* subsequent memcpy read adjacent kernel heap, and a value aimed
* at unmapped kernel memory faults the in-kernel copy and crashes
* the target host. The Discovery controller is unauthenticated,
* so the bug is reachable from any reachable fabric peer.
*/
if (offset > alloc_len) {
req->error_loc =
offsetof(struct nvme_get_log_page_command, lpo);
status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
goto out_free_buffer;
}
copy_len = min_t(size_t, data_len, alloc_len - offset);
status = nvmet_copy_to_sgl(req, 0, buffer + offset, copy_len);
if (!status && copy_len < data_len)
status = nvmet_zero_sgl(req, copy_len, data_len - copy_len);
out_free_buffer:
kfree(buffer);
out:
nvmet_req_complete(req, status);

15
drivers/nvme/target/fabrics-cmd-auth.c
View File

@@ -132,13 +132,22 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d)
return 0;
}
static u8 nvmet_auth_reply(struct nvmet_req *req, void *d)
static u8 nvmet_auth_reply(struct nvmet_req *req, void *d, u32 tl)
{
struct nvmet_ctrl *ctrl = req->sq->ctrl;
struct nvmf_auth_dhchap_reply_data *data = d;
u16 dhvlen = le16_to_cpu(data->dhvlen);
u16 dhvlen;
u8 *response;
if (tl < sizeof(*data))
return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
dhvlen = le16_to_cpu(data->dhvlen);
/* Validate that hl and dhvlen fit within the transfer length */
if (sizeof(*data) + 2 * (size_t)data->hl + dhvlen > tl)
return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
pr_debug("%s: ctrl %d qid %d: data hl %d cvalid %d dhvlen %u\n",
__func__, ctrl->cntlid, req->sq->qid,
data->hl, data->cvalid, dhvlen);
@@ -338,7 +347,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
switch (data->auth_id) {
case NVME_AUTH_DHCHAP_MESSAGE_REPLY:
dhchap_status = nvmet_auth_reply(req, d);
dhchap_status = nvmet_auth_reply(req, d, tl);
if (dhchap_status == 0)
req->sq->dhchap_step =
NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1;

33
drivers/nvme/target/loop.c
View File

@@ -202,7 +202,7 @@ static int nvme_loop_init_iod(struct nvme_loop_ctrl *ctrl,
static int nvme_loop_init_request(struct blk_mq_tag_set *set,
struct request *req, unsigned int hctx_idx,
unsigned int numa_node)
int numa_node)
{
struct nvme_loop_ctrl *ctrl = to_loop_ctrl(set->driver_data);
struct nvme_loop_iod *iod = blk_mq_rq_to_pdu(req);
@@ -274,7 +274,6 @@ static void nvme_loop_destroy_admin_queue(struct nvme_loop_ctrl *ctrl)
nvmet_sq_destroy(&ctrl->queues[0].nvme_sq);
nvmet_cq_put(&ctrl->queues[0].nvme_cq);
nvme_remove_admin_tag_set(&ctrl->ctrl);
}
static void nvme_loop_free_ctrl(struct nvme_ctrl *nctrl)
@@ -375,25 +374,18 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
}
ctrl->ctrl.queue_count = 1;
error = nvme_alloc_admin_tag_set(&ctrl->ctrl, &ctrl->admin_tag_set,
&nvme_loop_admin_mq_ops,
sizeof(struct nvme_loop_iod) +
NVME_INLINE_SG_CNT * sizeof(struct scatterlist));
if (error)
goto out_free_sq;
/* reset stopped state for the fresh admin queue */
clear_bit(NVME_CTRL_ADMIN_Q_STOPPED, &ctrl->ctrl.flags);
error = nvmf_connect_admin_queue(&ctrl->ctrl);
if (error)
goto out_cleanup_tagset;
goto out_free_sq;
set_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
error = nvme_enable_ctrl(&ctrl->ctrl);
if (error)
goto out_cleanup_tagset;
goto out_free_sq;
ctrl->ctrl.max_hw_sectors =
(NVME_LOOP_MAX_SEGMENTS - 1) << PAGE_SECTORS_SHIFT;
@@ -402,14 +394,12 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
error = nvme_init_ctrl_finish(&ctrl->ctrl, false);
if (error)
goto out_cleanup_tagset;
goto out_free_sq;
return 0;
out_cleanup_tagset:
clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
nvme_remove_admin_tag_set(&ctrl->ctrl);
out_free_sq:
clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags);
nvmet_sq_destroy(&ctrl->queues[0].nvme_sq);
nvmet_cq_put(&ctrl->queues[0].nvme_cq);
return error;
@@ -432,6 +422,7 @@ static void nvme_loop_shutdown_ctrl(struct nvme_loop_ctrl *ctrl)
static void nvme_loop_delete_ctrl_host(struct nvme_ctrl *ctrl)
{
nvme_loop_shutdown_ctrl(to_loop_ctrl(ctrl));
nvme_remove_admin_tag_set(ctrl);
}
static void nvme_loop_delete_ctrl(struct nvmet_ctrl *nctrl)
@@ -494,6 +485,7 @@ static void nvme_loop_reset_ctrl_work(struct work_struct *work)
nvme_cancel_admin_tagset(&ctrl->ctrl);
nvme_loop_destroy_admin_queue(ctrl);
out_disable:
nvme_remove_admin_tag_set(&ctrl->ctrl);
dev_warn(ctrl->ctrl.device, "Removing after reset failure\n");
nvme_uninit_ctrl(&ctrl->ctrl);
}
@@ -594,10 +586,17 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev,
if (!ctrl->queues)
goto out_uninit_ctrl;
ret = nvme_loop_configure_admin_queue(ctrl);
ret = nvme_alloc_admin_tag_set(&ctrl->ctrl, &ctrl->admin_tag_set,
&nvme_loop_admin_mq_ops,
sizeof(struct nvme_loop_iod) +
NVME_INLINE_SG_CNT * sizeof(struct scatterlist));
if (ret)
goto out_free_queues;
ret = nvme_loop_configure_admin_queue(ctrl);
if (ret)
goto out_remove_admin_tagset;
if (opts->queue_size > ctrl->ctrl.maxcmd) {
/* warn if maxcmd is lower than queue_size */
dev_warn(ctrl->ctrl.device,
@@ -633,6 +632,8 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev,
nvme_quiesce_admin_queue(&ctrl->ctrl);
nvme_cancel_admin_tagset(&ctrl->ctrl);
nvme_loop_destroy_admin_queue(ctrl);
out_remove_admin_tagset:
nvme_remove_admin_tag_set(&ctrl->ctrl);
out_free_queues:
kfree(ctrl->queues);
out_uninit_ctrl:

6
drivers/nvme/target/rdma.c
View File

@@ -1598,8 +1598,10 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id,
pending++;
}
mutex_unlock(&nvmet_rdma_queue_mutex);
if (pending > NVMET_RDMA_BACKLOG)
return NVME_SC_CONNECT_CTRL_BUSY;
if (pending > NVMET_RDMA_BACKLOG) {
ret = NVME_SC_CONNECT_CTRL_BUSY;
goto put_device;
}
}
ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn);

11
drivers/nvme/target/tcp.c
View File

@@ -1844,10 +1844,11 @@ static void nvmet_tcp_tls_handshake_done(void *data, int status,
if (!status)
status = nvmet_tcp_tls_key_lookup(queue, peerid);
if (!status)
status = nvmet_tcp_set_queue_sock(queue);
if (status)
nvmet_tcp_schedule_release_queue(queue);
else
nvmet_tcp_set_queue_sock(queue);
kref_put(&queue->kref, nvmet_tcp_release_queue);
}
@@ -1999,6 +2000,12 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
nvmet_tcp_free_cmd(&queue->connect);
out_ida_remove:
ida_free(&nvmet_tcp_queue_ida, queue->idx);
/*
* Drain the page fragment cache if any allocations were done.
* The first allocation using pf_cache is nvmet_tcp_alloc_cmd()
* for queue->connect after ida_alloc().
*/
page_frag_cache_drain(&queue->pf_cache);
out_sock:
fput(queue->sock->file);
out_free_queue:

2
drivers/scsi/scsi_lib.c
View File

@@ -1973,7 +1973,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
}
static int scsi_mq_init_request(struct blk_mq_tag_set *set, struct request *rq,
unsigned int hctx_idx, unsigned int numa_node)
unsigned int hctx_idx, int numa_node)
{
struct Scsi_Host *shost = set->driver_data;
struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);

15
fs/direct-io.c
View File

@@ -502,12 +502,8 @@ static blk_status_t dio_bio_complete(struct dio *dio, struct bio *bio)
const enum req_op dio_op = dio->opf & REQ_OP_MASK;
bool should_dirty = dio_op == REQ_OP_READ && dio->should_dirty;
if (err) {
if (err == BLK_STS_AGAIN && (bio->bi_opf & REQ_NOWAIT))
dio->io_error = -EAGAIN;
else
dio->io_error = -EIO;
}
if (err)
dio->io_error = -EIO;
if (dio->is_async && should_dirty) {
bio_check_pages_dirty(bio); /* transfers ownership */
@@ -1178,13 +1174,10 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode,
dio->is_async = true;
dio->inode = inode;
if (iov_iter_rw(iter) == WRITE) {
if (iov_iter_rw(iter) == WRITE)
dio->opf = REQ_OP_WRITE | REQ_SYNC | REQ_IDLE;
if (iocb->ki_flags & IOCB_NOWAIT)
dio->opf |= REQ_NOWAIT;
} else {
else
dio->opf = REQ_OP_READ;
}
/*
* For AIO O_(D)SYNC writes we need to defer completions to a workqueue

32
include/linux/bio.h
View File

@@ -283,7 +283,7 @@ static inline void bio_first_folio(struct folio_iter *fi, struct bio *bio,
return;
}
fi->folio = page_folio(bvec->bv_page);
fi->folio = bvec_folio(bvec);
fi->offset = bvec->bv_offset +
PAGE_SIZE * folio_page_idx(fi->folio, bvec->bv_page);
fi->_seg_count = bvec->bv_len;
@@ -347,7 +347,6 @@ enum {
};
extern int bioset_init(struct bio_set *, unsigned int, unsigned int, int flags);
extern void bioset_exit(struct bio_set *);
extern int biovec_init_pool(mempool_t *pool, int pool_entries);
struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs,
blk_opf_t opf, gfp_t gfp, struct bio_set *bs);
@@ -371,17 +370,27 @@ void submit_bio(struct bio *bio);
extern void bio_endio(struct bio *);
/**
* bio_endio_status - end I/O on a bio with a specific status
* @bio: bio
* @status: status to set
*
* Set @bio->bi_status to @status and call bio_endio().
**/
static inline void bio_endio_status(struct bio *bio, blk_status_t status)
{
bio->bi_status = status;
bio_endio(bio);
}
static inline void bio_io_error(struct bio *bio)
{
bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
bio_endio_status(bio, BLK_STS_IOERR);
}
static inline void bio_wouldblock_error(struct bio *bio)
{
bio_set_flag(bio, BIO_QUIET);
bio->bi_status = BLK_STS_AGAIN;
bio_endio(bio);
bio_endio_status(bio, BLK_STS_AGAIN);
}
/*
@@ -479,17 +488,10 @@ int bio_iov_iter_bounce(struct bio *bio, struct iov_iter *iter, size_t maxlen,
size_t minsize);
void bio_iov_iter_unbounce(struct bio *bio, bool is_error, bool mark_dirty);
extern void bio_copy_data_iter(struct bio *dst, struct bvec_iter *dst_iter,
struct bio *src, struct bvec_iter *src_iter);
extern void bio_copy_data(struct bio *dst, struct bio *src);
extern void bio_free_pages(struct bio *bio);
void zero_fill_bio(struct bio *bio);
void guard_bio_eod(struct bio *bio);
void zero_fill_bio_iter(struct bio *bio, struct bvec_iter iter);
static inline void zero_fill_bio(struct bio *bio)
{
zero_fill_bio_iter(bio, bio->bi_iter);
}
static inline void bio_release_pages(struct bio *bio, bool mark_dirty)
{

53
include/linux/blk-mq.h
View File

@@ -428,7 +428,7 @@ struct blk_mq_hw_ctx {
struct blk_mq_tags *sched_tags;
/** @numa_node: NUMA node the storage adapter has been connected to. */
unsigned int numa_node;
int numa_node;
/** @queue_num: Index of this hardware queue. */
unsigned int queue_num;
@@ -653,7 +653,7 @@ struct blk_mq_ops {
* flush request.
*/
int (*init_request)(struct blk_mq_tag_set *set, struct request *,
unsigned int, unsigned int);
unsigned int, int);
/**
* @exit_request: Ditto for exit/teardown.
*/
@@ -1104,6 +1104,7 @@ struct req_iterator {
/*
* blk_rq_pos() : the current sector
* blk_rq_bytes() : bytes left in the entire request
* blk_rq_has_data() : whether the request carries data
* blk_rq_cur_bytes() : bytes left in the current segment
* blk_rq_sectors() : sectors left in the entire request
* blk_rq_cur_sectors() : sectors left in the current segment
@@ -1119,6 +1120,14 @@ static inline unsigned int blk_rq_bytes(const struct request *rq)
return rq->__data_len;
}
static inline bool blk_rq_has_data(const struct request *rq)
{
return blk_rq_bytes(rq) &&
req_op(rq) != REQ_OP_DISCARD &&
req_op(rq) != REQ_OP_SECURE_ERASE &&
req_op(rq) != REQ_OP_WRITE_ZEROES;
}
static inline int blk_rq_cur_bytes(const struct request *rq)
{
if (!rq->bio)
@@ -1243,4 +1252,44 @@ static inline int blk_rq_map_sg(struct request *rq, struct scatterlist *sglist)
}
void blk_dump_rq_flags(struct request *, char *);
/**
* blk_rq_passthrough_stats - check if this request should account stats
* @rq: request to check
* @q: the queue accumulating the stats
*
* Note, @q does not necessarily need to be the request_queue that provides
* @rq.
*
* Return: true if stats should be accounted.
*/
static inline bool blk_rq_passthrough_stats(struct request *rq,
struct request_queue *q)
{
struct bio *bio = rq->bio;
if (!blk_queue_passthrough_stat(q))
return false;
/* Requests without a bio do not transfer data. */
if (!bio)
return false;
/*
* Stats are accumulated in the bdev, so must have one attached to a
* bio to track stats. Most drivers do not set the bdev for passthrough
* requests, but nvme is one that will set it.
*/
if (!bio->bi_bdev)
return false;
/*
* We don't know what a passthrough command does, but we know the
* payload size and data direction. Ensuring the size is aligned to the
* block size filters out most commands with payloads that don't
* represent sector access.
*/
if (blk_rq_bytes(rq) & (bdev_logical_block_size(bio->bi_bdev) - 1))
return false;
return true;
}
#endif /* BLK_MQ_H */

28
include/linux/blkdev.h
View File

@@ -176,6 +176,7 @@ struct gendisk {
#define GD_SUPPRESS_PART_SCAN 5
#define GD_OWNS_QUEUE 6
#define GD_ZONE_APPEND_USED 7
#define GD_ERROR_INJECT 8
struct mutex open_mutex; /* open/close mutex */
unsigned open_partitions; /* number of open partitions */
@@ -227,6 +228,11 @@ struct gendisk {
*/
struct blk_independent_access_ranges *ia_ranges;
#ifdef CONFIG_BLK_ERROR_INJECTION
struct mutex error_injection_lock;
struct list_head error_injection_list;
#endif
struct mutex rqos_state_mutex; /* rqos state change mutex */
};
@@ -1040,7 +1046,6 @@ extern const char *blk_op_str(enum req_op op);
int blk_status_to_errno(blk_status_t status);
blk_status_t errno_to_blk_status(int errno);
const char *blk_status_to_str(blk_status_t status);
/* only poll the hardware once, don't continue until a completion was found */
#define BLK_POLL_ONESHOT (1 << 0)
@@ -1093,15 +1098,17 @@ static inline unsigned int blk_boundary_sectors_left(sector_t offset,
*/
static inline struct queue_limits
queue_limits_start_update(struct request_queue *q)
__acquires(&q->limits_lock)
{
mutex_lock(&q->limits_lock);
return q->limits;
}
int queue_limits_commit_update_frozen(struct request_queue *q,
struct queue_limits *lim);
struct queue_limits *lim) __releases(&q->limits_lock);
int queue_limits_commit_update(struct request_queue *q,
struct queue_limits *lim);
int queue_limits_set(struct request_queue *q, struct queue_limits *lim);
struct queue_limits *lim) __releases(&q->limits_lock);
int queue_limits_set(struct request_queue *q, struct queue_limits *lim)
__must_not_hold(&q->limits_lock);
int blk_validate_limits(struct queue_limits *lim);
/**
@@ -1113,6 +1120,7 @@ int blk_validate_limits(struct queue_limits *lim);
* starting update.
*/
static inline void queue_limits_cancel_update(struct request_queue *q)
__releases(&q->limits_lock)
{
mutex_unlock(&q->limits_lock);
}
@@ -1744,22 +1752,26 @@ void blkdev_show(struct seq_file *seqf, off_t offset);
#endif
struct blk_holder_ops {
void (*mark_dead)(struct block_device *bdev, bool surprise);
void (*mark_dead)(struct block_device *bdev, bool surprise)
__releases(&bdev->bd_holder_lock);
/*
* Sync the file system mounted on the block device.
*/
void (*sync)(struct block_device *bdev);
void (*sync)(struct block_device *bdev)
__releases(&bdev->bd_holder_lock);
/*
* Freeze the file system mounted on the block device.
*/
int (*freeze)(struct block_device *bdev);
int (*freeze)(struct block_device *bdev)
__releases(&bdev->bd_holder_lock);
/*
* Thaw the file system mounted on the block device.
*/
int (*thaw)(struct block_device *bdev);
int (*thaw)(struct block_device *bdev)
__releases(&bdev->bd_holder_lock);
};
/*

112
include/linux/bvec.h
View File

@@ -74,6 +74,21 @@ static inline void bvec_set_virt(struct bio_vec *bv, void *vaddr,
bvec_set_page(bv, virt_to_page(vaddr), len, offset_in_page(vaddr));
}
/**
* bvec_folio - Return the first folio referenced by this bvec
* @bv: bvec to access
*
* A bvec can contain non-folio memory, so this should only be called by
* the creator of the bvec; drivers have no business looking at the owner
* of the memory. It may not even be the right interface for the caller
* to use as a bvec can span multiple folios. You may be better off using
* something like bio_for_each_folio_all() which iterates over all folios.
*/
static inline struct folio *bvec_folio(const struct bio_vec *bv)
{
return page_folio(bv->bv_page);
}
struct bvec_iter {
/*
* Current device address in 512 byte sectors. Only updated by the bio
@@ -104,51 +119,78 @@ struct bvec_iter_all {
unsigned done;
};
/*
* various member access, note that bio_data should of course not be used
* on highmem page vectors
*/
#define __bvec_iter_bvec(bvec, iter) (&(bvec)[(iter).bi_idx])
static __always_inline const struct bio_vec *
__bvec_iter_bvec(const struct bio_vec *bvecs, const struct bvec_iter iter)
{
return bvecs + iter.bi_idx;
}
/* multi-page (mp_bvec) helpers */
#define mp_bvec_iter_page(bvec, iter) \
(__bvec_iter_bvec((bvec), (iter))->bv_page)
static __always_inline struct page *
mp_bvec_iter_page(const struct bio_vec *bvecs, const struct bvec_iter iter)
{
return __bvec_iter_bvec(bvecs, iter)->bv_page;
}
#define mp_bvec_iter_len(bvec, iter) \
min((iter).bi_size, \
__bvec_iter_bvec((bvec), (iter))->bv_len - (iter).bi_bvec_done)
static __always_inline unsigned int
mp_bvec_iter_len(const struct bio_vec *bvecs, const struct bvec_iter iter)
{
return min(__bvec_iter_bvec(bvecs, iter)->bv_len - iter.bi_bvec_done,
iter.bi_size);
}
#define mp_bvec_iter_offset(bvec, iter) \
(__bvec_iter_bvec((bvec), (iter))->bv_offset + (iter).bi_bvec_done)
static __always_inline unsigned int
mp_bvec_iter_offset(const struct bio_vec *bvecs, const struct bvec_iter iter)
{
return __bvec_iter_bvec(bvecs, iter)->bv_offset + iter.bi_bvec_done;
}
#define mp_bvec_iter_page_idx(bvec, iter) \
(mp_bvec_iter_offset((bvec), (iter)) / PAGE_SIZE)
static __always_inline unsigned int
mp_bvec_iter_page_idx(const struct bio_vec *bvecs, const struct bvec_iter iter)
{
return mp_bvec_iter_offset(bvecs, iter) / PAGE_SIZE;
}
#define mp_bvec_iter_bvec(bvec, iter) \
((struct bio_vec) { \
.bv_page = mp_bvec_iter_page((bvec), (iter)), \
.bv_len = mp_bvec_iter_len((bvec), (iter)), \
.bv_offset = mp_bvec_iter_offset((bvec), (iter)), \
})
static __always_inline struct bio_vec
mp_bvec_iter_bvec(const struct bio_vec *bvecs, const struct bvec_iter iter)
{
return (struct bio_vec) {
.bv_page = mp_bvec_iter_page(bvecs, iter),
.bv_len = mp_bvec_iter_len(bvecs, iter),
.bv_offset = mp_bvec_iter_offset(bvecs, iter),
};
}
/* For building single-page bvec in flight */
#define bvec_iter_offset(bvec, iter) \
(mp_bvec_iter_offset((bvec), (iter)) % PAGE_SIZE)
static __always_inline unsigned int
bvec_iter_offset(const struct bio_vec *bvecs, const struct bvec_iter iter)
{
return mp_bvec_iter_offset(bvecs, iter) % PAGE_SIZE;
}
#define bvec_iter_len(bvec, iter) \
min_t(unsigned, mp_bvec_iter_len((bvec), (iter)), \
PAGE_SIZE - bvec_iter_offset((bvec), (iter)))
static __always_inline unsigned int
bvec_iter_len(const struct bio_vec *bvecs, const struct bvec_iter iter)
{
return min(mp_bvec_iter_len(bvecs, iter),
PAGE_SIZE - bvec_iter_offset(bvecs, iter));
}
#define bvec_iter_page(bvec, iter) \
(mp_bvec_iter_page((bvec), (iter)) + \
mp_bvec_iter_page_idx((bvec), (iter)))
static __always_inline struct page *
bvec_iter_page(const struct bio_vec *bvecs, const struct bvec_iter iter)
{
return mp_bvec_iter_page(bvecs, iter) +
mp_bvec_iter_page_idx(bvecs, iter);
}
#define bvec_iter_bvec(bvec, iter) \
((struct bio_vec) { \
.bv_page = bvec_iter_page((bvec), (iter)), \
.bv_len = bvec_iter_len((bvec), (iter)), \
.bv_offset = bvec_iter_offset((bvec), (iter)), \
})
static __always_inline struct bio_vec
bvec_iter_bvec(const struct bio_vec *bvecs, const struct bvec_iter iter)
{
return (struct bio_vec) {
.bv_page = bvec_iter_page(bvecs, iter),
.bv_len = bvec_iter_len(bvecs, iter),
.bv_offset = bvec_iter_offset(bvecs, iter),
};
}
static inline bool bvec_iter_advance(const struct bio_vec *bv,
struct bvec_iter *iter, unsigned bytes)
@@ -247,6 +289,7 @@ static inline void *bvec_kmap_local(struct bio_vec *bvec)
/**
* memcpy_from_bvec - copy data from a bvec
* @to: Kernel virtual address to copy to.
* @bvec: bvec to copy from
*
* Must be called on single-page bvecs only.
@@ -259,6 +302,7 @@ static inline void memcpy_from_bvec(char *to, struct bio_vec *bvec)
/**
* memcpy_to_bvec - copy data to a bvec
* @bvec: bvec to copy to
* @from: Kernel virtual address to copy from.
*
* Must be called on single-page bvecs only.
*/

536
include/linux/drbd_genl.h
View File

@@ -1,536 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* General overview:
* full generic netlink message:
* |nlmsghdr|genlmsghdr|<payload>
*
* payload:
* |optional fixed size family header|<sequence of netlink attributes>
*
* sequence of netlink attributes:
* I chose to have all "top level" attributes NLA_NESTED,
* corresponding to some real struct.
* So we have a sequence of |tla, len|<nested nla sequence>
*
* nested nla sequence:
* may be empty, or contain a sequence of netlink attributes
* representing the struct fields.
*
* The tag number of any field (regardless of containing struct)
* will be available as T_ ## field_name,
* so you cannot have the same field name in two differnt structs.
*
* The tag numbers themselves are per struct, though,
* so should always begin at 1 (not 0, that is the special "NLA_UNSPEC" type,
* which we won't use here).
* The tag numbers are used as index in the respective nla_policy array.
*
* GENL_struct(tag_name, tag_number, struct name, struct fields) - struct and policy
* genl_magic_struct.h
* generates the struct declaration,
* generates an entry in the tla enum,
* genl_magic_func.h
* generates an entry in the static tla policy
* with .type = NLA_NESTED
* generates the static <struct_name>_nl_policy definition,
* and static conversion functions
*
* genl_magic_func.h
*
* GENL_mc_group(group)
* genl_magic_struct.h
* does nothing
* genl_magic_func.h
* defines and registers the mcast group,
* and provides a send helper
*
* GENL_notification(op_name, op_num, mcast_group, tla list)
* These are notifications to userspace.
*
* genl_magic_struct.h
* generates an entry in the genl_ops enum,
* genl_magic_func.h
* does nothing
*
* mcast group: the name of the mcast group this notification should be
* expected on
* tla list: the list of expected top level attributes,
* for documentation and sanity checking.
*
* GENL_op(op_name, op_num, flags and handler, tla list) - "genl operations"
* These are requests from userspace.
*
* _op and _notification share the same "number space",
* op_nr will be assigned to "genlmsghdr->cmd"
*
* genl_magic_struct.h
* generates an entry in the genl_ops enum,
* genl_magic_func.h
* generates an entry in the static genl_ops array,
* and static register/unregister functions to
* genl_register_family().
*
* flags and handler:
* GENL_op_init( .doit = x, .dumpit = y, .flags = something)
* GENL_doit(x) => .dumpit = NULL, .flags = GENL_ADMIN_PERM
* tla list: the list of expected top level attributes,
* for documentation and sanity checking.
*/
/*
* STRUCTS
*/
/* this is sent kernel -> userland on various error conditions, and contains
* informational textual info, which is supposedly human readable.
* The computer relevant return code is in the drbd_genlmsghdr.
*/
GENL_struct(DRBD_NLA_CFG_REPLY, 1, drbd_cfg_reply,
/* "arbitrary" size strings, nla_policy.len = 0 */
__str_field(1, 0, info_text, 0)
)
/* Configuration requests typically need a context to operate on.
* Possible keys are device minor (fits in the drbd_genlmsghdr),
* the replication link (aka connection) name,
* and/or the replication group (aka resource) name,
* and the volume id within the resource. */
GENL_struct(DRBD_NLA_CFG_CONTEXT, 2, drbd_cfg_context,
__u32_field(1, 0, ctx_volume)
__str_field(2, 0, ctx_resource_name, 128)
__bin_field(3, 0, ctx_my_addr, 128)
__bin_field(4, 0, ctx_peer_addr, 128)
)
GENL_struct(DRBD_NLA_DISK_CONF, 3, disk_conf,
__str_field(1, DRBD_F_REQUIRED | DRBD_F_INVARIANT, backing_dev, 128)
__str_field(2, DRBD_F_REQUIRED | DRBD_F_INVARIANT, meta_dev, 128)
__s32_field(3, DRBD_F_REQUIRED | DRBD_F_INVARIANT, meta_dev_idx)
/* use the resize command to try and change the disk_size */
__u64_field(4, DRBD_F_INVARIANT, disk_size)
/* we could change the max_bio_bvecs,
* but it won't propagate through the stack */
__u32_field(5, DRBD_F_INVARIANT, max_bio_bvecs)
__u32_field_def(6, 0, on_io_error, DRBD_ON_IO_ERROR_DEF)
__u32_field_def(7, 0, fencing, DRBD_FENCING_DEF)
__u32_field_def(8, 0, resync_rate, DRBD_RESYNC_RATE_DEF)
__s32_field_def(9, 0, resync_after, DRBD_MINOR_NUMBER_DEF)
__u32_field_def(10, 0, al_extents, DRBD_AL_EXTENTS_DEF)
__u32_field_def(11, 0, c_plan_ahead, DRBD_C_PLAN_AHEAD_DEF)
__u32_field_def(12, 0, c_delay_target, DRBD_C_DELAY_TARGET_DEF)
__u32_field_def(13, 0, c_fill_target, DRBD_C_FILL_TARGET_DEF)
__u32_field_def(14, 0, c_max_rate, DRBD_C_MAX_RATE_DEF)
__u32_field_def(15, 0, c_min_rate, DRBD_C_MIN_RATE_DEF)
__u32_field_def(20, 0, disk_timeout, DRBD_DISK_TIMEOUT_DEF)
__u32_field_def(21, 0 /* OPTIONAL */, read_balancing, DRBD_READ_BALANCING_DEF)
__u32_field_def(25, 0 /* OPTIONAL */, rs_discard_granularity, DRBD_RS_DISCARD_GRANULARITY_DEF)
__flg_field_def(16, 0, disk_barrier, DRBD_DISK_BARRIER_DEF)
__flg_field_def(17, 0, disk_flushes, DRBD_DISK_FLUSHES_DEF)
__flg_field_def(18, 0, disk_drain, DRBD_DISK_DRAIN_DEF)
__flg_field_def(19, 0, md_flushes, DRBD_MD_FLUSHES_DEF)
__flg_field_def(23, 0 /* OPTIONAL */, al_updates, DRBD_AL_UPDATES_DEF)
__flg_field_def(24, 0 /* OPTIONAL */, discard_zeroes_if_aligned, DRBD_DISCARD_ZEROES_IF_ALIGNED_DEF)
__flg_field_def(26, 0 /* OPTIONAL */, disable_write_same, DRBD_DISABLE_WRITE_SAME_DEF)
)
GENL_struct(DRBD_NLA_RESOURCE_OPTS, 4, res_opts,
__str_field_def(1, 0, cpu_mask, DRBD_CPU_MASK_SIZE)
__u32_field_def(2, 0, on_no_data, DRBD_ON_NO_DATA_DEF)
)
GENL_struct(DRBD_NLA_NET_CONF, 5, net_conf,
__str_field_def(1, DRBD_F_SENSITIVE,
shared_secret, SHARED_SECRET_MAX)
__str_field_def(2, 0, cram_hmac_alg, SHARED_SECRET_MAX)
__str_field_def(3, 0, integrity_alg, SHARED_SECRET_MAX)
__str_field_def(4, 0, verify_alg, SHARED_SECRET_MAX)
__str_field_def(5, 0, csums_alg, SHARED_SECRET_MAX)
__u32_field_def(6, 0, wire_protocol, DRBD_PROTOCOL_DEF)
__u32_field_def(7, 0, connect_int, DRBD_CONNECT_INT_DEF)
__u32_field_def(8, 0, timeout, DRBD_TIMEOUT_DEF)
__u32_field_def(9, 0, ping_int, DRBD_PING_INT_DEF)
__u32_field_def(10, 0, ping_timeo, DRBD_PING_TIMEO_DEF)
__u32_field_def(11, 0, sndbuf_size, DRBD_SNDBUF_SIZE_DEF)
__u32_field_def(12, 0, rcvbuf_size, DRBD_RCVBUF_SIZE_DEF)
__u32_field_def(13, 0, ko_count, DRBD_KO_COUNT_DEF)
__u32_field_def(14, 0, max_buffers, DRBD_MAX_BUFFERS_DEF)
__u32_field_def(15, 0, max_epoch_size, DRBD_MAX_EPOCH_SIZE_DEF)
__u32_field_def(16, 0, unplug_watermark, DRBD_UNPLUG_WATERMARK_DEF)
__u32_field_def(17, 0, after_sb_0p, DRBD_AFTER_SB_0P_DEF)
__u32_field_def(18, 0, after_sb_1p, DRBD_AFTER_SB_1P_DEF)
__u32_field_def(19, 0, after_sb_2p, DRBD_AFTER_SB_2P_DEF)
__u32_field_def(20, 0, rr_conflict, DRBD_RR_CONFLICT_DEF)
__u32_field_def(21, 0, on_congestion, DRBD_ON_CONGESTION_DEF)
__u32_field_def(22, 0, cong_fill, DRBD_CONG_FILL_DEF)
__u32_field_def(23, 0, cong_extents, DRBD_CONG_EXTENTS_DEF)
__flg_field_def(24, 0, two_primaries, DRBD_ALLOW_TWO_PRIMARIES_DEF)
__flg_field(25, DRBD_F_INVARIANT, discard_my_data)
__flg_field_def(26, 0, tcp_cork, DRBD_TCP_CORK_DEF)
__flg_field_def(27, 0, always_asbp, DRBD_ALWAYS_ASBP_DEF)
__flg_field(28, DRBD_F_INVARIANT, tentative)
__flg_field_def(29, 0, use_rle, DRBD_USE_RLE_DEF)
/* 9: __u32_field_def(30, 0, fencing_policy, DRBD_FENCING_DEF) */
/* 9: __str_field_def(31, 0, name, SHARED_SECRET_MAX) */
/* 9: __u32_field(32, DRBD_F_REQUIRED | DRBD_F_INVARIANT, peer_node_id) */
__flg_field_def(33, 0 /* OPTIONAL */, csums_after_crash_only, DRBD_CSUMS_AFTER_CRASH_ONLY_DEF)
__u32_field_def(34, 0 /* OPTIONAL */, sock_check_timeo, DRBD_SOCKET_CHECK_TIMEO_DEF)
)
GENL_struct(DRBD_NLA_SET_ROLE_PARMS, 6, set_role_parms,
__flg_field(1, 0, assume_uptodate)
)
GENL_struct(DRBD_NLA_RESIZE_PARMS, 7, resize_parms,
__u64_field(1, 0, resize_size)
__flg_field(2, 0, resize_force)
__flg_field(3, 0, no_resync)
__u32_field_def(4, 0 /* OPTIONAL */, al_stripes, DRBD_AL_STRIPES_DEF)
__u32_field_def(5, 0 /* OPTIONAL */, al_stripe_size, DRBD_AL_STRIPE_SIZE_DEF)
)
GENL_struct(DRBD_NLA_STATE_INFO, 8, state_info,
/* the reason of the broadcast,
* if this is an event triggered broadcast. */
__u32_field(1, 0, sib_reason)
__u32_field(2, DRBD_F_REQUIRED, current_state)
__u64_field(3, 0, capacity)
__u64_field(4, 0, ed_uuid)
/* These are for broadcast from after state change work.
* prev_state and new_state are from the moment the state change took
* place, new_state is not neccessarily the same as current_state,
* there may have been more state changes since. Which will be
* broadcasted soon, in their respective after state change work. */
__u32_field(5, 0, prev_state)
__u32_field(6, 0, new_state)
/* if we have a local disk: */
__bin_field(7, 0, uuids, (UI_SIZE*sizeof(__u64)))
__u32_field(8, 0, disk_flags)
__u64_field(9, 0, bits_total)
__u64_field(10, 0, bits_oos)
/* and in case resync or online verify is active */
__u64_field(11, 0, bits_rs_total)
__u64_field(12, 0, bits_rs_failed)
/* for pre and post notifications of helper execution */
__str_field(13, 0, helper, 32)
__u32_field(14, 0, helper_exit_code)
__u64_field(15, 0, send_cnt)
__u64_field(16, 0, recv_cnt)
__u64_field(17, 0, read_cnt)
__u64_field(18, 0, writ_cnt)
__u64_field(19, 0, al_writ_cnt)
__u64_field(20, 0, bm_writ_cnt)
__u32_field(21, 0, ap_bio_cnt)
__u32_field(22, 0, ap_pending_cnt)
__u32_field(23, 0, rs_pending_cnt)
)
GENL_struct(DRBD_NLA_START_OV_PARMS, 9, start_ov_parms,
__u64_field(1, 0, ov_start_sector)
__u64_field(2, 0, ov_stop_sector)
)
GENL_struct(DRBD_NLA_NEW_C_UUID_PARMS, 10, new_c_uuid_parms,
__flg_field(1, 0, clear_bm)
)
GENL_struct(DRBD_NLA_TIMEOUT_PARMS, 11, timeout_parms,
__u32_field(1, DRBD_F_REQUIRED, timeout_type)
)
GENL_struct(DRBD_NLA_DISCONNECT_PARMS, 12, disconnect_parms,
__flg_field(1, 0, force_disconnect)
)
GENL_struct(DRBD_NLA_DETACH_PARMS, 13, detach_parms,
__flg_field(1, 0, force_detach)
)
GENL_struct(DRBD_NLA_RESOURCE_INFO, 15, resource_info,
__u32_field(1, 0, res_role)
__flg_field(2, 0, res_susp)
__flg_field(3, 0, res_susp_nod)
__flg_field(4, 0, res_susp_fen)
/* __flg_field(5, 0, res_weak) */
)
GENL_struct(DRBD_NLA_DEVICE_INFO, 16, device_info,
__u32_field(1, 0, dev_disk_state)
)
GENL_struct(DRBD_NLA_CONNECTION_INFO, 17, connection_info,
__u32_field(1, 0, conn_connection_state)
__u32_field(2, 0, conn_role)
)
GENL_struct(DRBD_NLA_PEER_DEVICE_INFO, 18, peer_device_info,
__u32_field(1, 0, peer_repl_state)
__u32_field(2, 0, peer_disk_state)
__u32_field(3, 0, peer_resync_susp_user)
__u32_field(4, 0, peer_resync_susp_peer)
__u32_field(5, 0, peer_resync_susp_dependency)
)
GENL_struct(DRBD_NLA_RESOURCE_STATISTICS, 19, resource_statistics,
__u32_field(1, 0, res_stat_write_ordering)
)
GENL_struct(DRBD_NLA_DEVICE_STATISTICS, 20, device_statistics,
__u64_field(1, 0, dev_size) /* (sectors) */
__u64_field(2, 0, dev_read) /* (sectors) */
__u64_field(3, 0, dev_write) /* (sectors) */
__u64_field(4, 0, dev_al_writes) /* activity log writes (count) */
__u64_field(5, 0, dev_bm_writes) /* bitmap writes (count) */
__u32_field(6, 0, dev_upper_pending) /* application requests in progress */
__u32_field(7, 0, dev_lower_pending) /* backing device requests in progress */
__flg_field(8, 0, dev_upper_blocked)
__flg_field(9, 0, dev_lower_blocked)
__flg_field(10, 0, dev_al_suspended) /* activity log suspended */
__u64_field(11, 0, dev_exposed_data_uuid)
__u64_field(12, 0, dev_current_uuid)
__u32_field(13, 0, dev_disk_flags)
__bin_field(14, 0, history_uuids, HISTORY_UUIDS * sizeof(__u64))
)
GENL_struct(DRBD_NLA_CONNECTION_STATISTICS, 21, connection_statistics,
__flg_field(1, 0, conn_congested)
)
GENL_struct(DRBD_NLA_PEER_DEVICE_STATISTICS, 22, peer_device_statistics,
__u64_field(1, 0, peer_dev_received) /* sectors */
__u64_field(2, 0, peer_dev_sent) /* sectors */
__u32_field(3, 0, peer_dev_pending) /* number of requests */
__u32_field(4, 0, peer_dev_unacked) /* number of requests */
__u64_field(5, 0, peer_dev_out_of_sync) /* sectors */
__u64_field(6, 0, peer_dev_resync_failed) /* sectors */
__u64_field(7, 0, peer_dev_bitmap_uuid)
__u32_field(9, 0, peer_dev_flags)
)
GENL_struct(DRBD_NLA_NOTIFICATION_HEADER, 23, drbd_notification_header,
__u32_field(1, 0, nh_type)
)
GENL_struct(DRBD_NLA_HELPER, 24, drbd_helper_info,
__str_field(1, 0, helper_name, 32)
__u32_field(2, 0, helper_status)
)
/*
* Notifications and commands (genlmsghdr->cmd)
*/
GENL_mc_group(events)
/* kernel -> userspace announcement of changes */
GENL_notification(
DRBD_EVENT, 1, events,
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_STATE_INFO, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_NET_CONF, 0)
GENL_tla_expected(DRBD_NLA_DISK_CONF, 0)
GENL_tla_expected(DRBD_NLA_SYNCER_CONF, 0)
)
/* query kernel for specific or all info */
GENL_op(
DRBD_ADM_GET_STATUS, 2,
GENL_op_init(
.doit = drbd_adm_get_status,
.dumpit = drbd_adm_get_status_all,
/* anyone may ask for the status,
* it is broadcasted anyways */
),
/* To select the object .doit.
* Or a subset of objects in .dumpit. */
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0)
)
/* add DRBD minor devices as volumes to resources */
GENL_op(DRBD_ADM_NEW_MINOR, 5, GENL_doit(drbd_adm_new_minor),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_DEL_MINOR, 6, GENL_doit(drbd_adm_del_minor),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
/* add or delete resources */
GENL_op(DRBD_ADM_NEW_RESOURCE, 7, GENL_doit(drbd_adm_new_resource),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_DEL_RESOURCE, 8, GENL_doit(drbd_adm_del_resource),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_RESOURCE_OPTS, 9,
GENL_doit(drbd_adm_resource_opts),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_RESOURCE_OPTS, 0)
)
GENL_op(
DRBD_ADM_CONNECT, 10,
GENL_doit(drbd_adm_connect),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_NET_CONF, DRBD_F_REQUIRED)
)
GENL_op(
DRBD_ADM_CHG_NET_OPTS, 29,
GENL_doit(drbd_adm_net_opts),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_NET_CONF, DRBD_F_REQUIRED)
)
GENL_op(DRBD_ADM_DISCONNECT, 11, GENL_doit(drbd_adm_disconnect),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_ATTACH, 12,
GENL_doit(drbd_adm_attach),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_DISK_CONF, DRBD_F_REQUIRED)
)
GENL_op(DRBD_ADM_CHG_DISK_OPTS, 28,
GENL_doit(drbd_adm_disk_opts),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_DISK_OPTS, DRBD_F_REQUIRED)
)
GENL_op(
DRBD_ADM_RESIZE, 13,
GENL_doit(drbd_adm_resize),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_RESIZE_PARMS, 0)
)
GENL_op(
DRBD_ADM_PRIMARY, 14,
GENL_doit(drbd_adm_set_role),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_SET_ROLE_PARMS, DRBD_F_REQUIRED)
)
GENL_op(
DRBD_ADM_SECONDARY, 15,
GENL_doit(drbd_adm_set_role),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_SET_ROLE_PARMS, DRBD_F_REQUIRED)
)
GENL_op(
DRBD_ADM_NEW_C_UUID, 16,
GENL_doit(drbd_adm_new_c_uuid),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_NEW_C_UUID_PARMS, 0)
)
GENL_op(
DRBD_ADM_START_OV, 17,
GENL_doit(drbd_adm_start_ov),
GENL_tla_expected(DRBD_NLA_START_OV_PARMS, 0)
)
GENL_op(DRBD_ADM_DETACH, 18, GENL_doit(drbd_adm_detach),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_DETACH_PARMS, 0))
GENL_op(DRBD_ADM_INVALIDATE, 19, GENL_doit(drbd_adm_invalidate),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_INVAL_PEER, 20, GENL_doit(drbd_adm_invalidate_peer),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_PAUSE_SYNC, 21, GENL_doit(drbd_adm_pause_sync),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_RESUME_SYNC, 22, GENL_doit(drbd_adm_resume_sync),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_SUSPEND_IO, 23, GENL_doit(drbd_adm_suspend_io),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_RESUME_IO, 24, GENL_doit(drbd_adm_resume_io),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_OUTDATE, 25, GENL_doit(drbd_adm_outdate),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_GET_TIMEOUT_TYPE, 26, GENL_doit(drbd_adm_get_timeout_type),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_DOWN, 27, GENL_doit(drbd_adm_down),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED))
GENL_op(DRBD_ADM_GET_RESOURCES, 30,
GENL_op_init(
.dumpit = drbd_adm_dump_resources,
),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0)
GENL_tla_expected(DRBD_NLA_RESOURCE_INFO, 0)
GENL_tla_expected(DRBD_NLA_RESOURCE_STATISTICS, 0))
GENL_op(DRBD_ADM_GET_DEVICES, 31,
GENL_op_init(
.dumpit = drbd_adm_dump_devices,
.done = drbd_adm_dump_devices_done,
),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0)
GENL_tla_expected(DRBD_NLA_DEVICE_INFO, 0)
GENL_tla_expected(DRBD_NLA_DEVICE_STATISTICS, 0))
GENL_op(DRBD_ADM_GET_CONNECTIONS, 32,
GENL_op_init(
.dumpit = drbd_adm_dump_connections,
.done = drbd_adm_dump_connections_done,
),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0)
GENL_tla_expected(DRBD_NLA_CONNECTION_INFO, 0)
GENL_tla_expected(DRBD_NLA_CONNECTION_STATISTICS, 0))
GENL_op(DRBD_ADM_GET_PEER_DEVICES, 33,
GENL_op_init(
.dumpit = drbd_adm_dump_peer_devices,
.done = drbd_adm_dump_peer_devices_done,
),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0)
GENL_tla_expected(DRBD_NLA_PEER_DEVICE_INFO, 0)
GENL_tla_expected(DRBD_NLA_PEER_DEVICE_STATISTICS, 0))
GENL_notification(
DRBD_RESOURCE_STATE, 34, events,
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_NOTIFICATION_HEADER, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_RESOURCE_INFO, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_RESOURCE_STATISTICS, DRBD_F_REQUIRED))
GENL_notification(
DRBD_DEVICE_STATE, 35, events,
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_NOTIFICATION_HEADER, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_DEVICE_INFO, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_DEVICE_STATISTICS, DRBD_F_REQUIRED))
GENL_notification(
DRBD_CONNECTION_STATE, 36, events,
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_NOTIFICATION_HEADER, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_CONNECTION_INFO, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_CONNECTION_STATISTICS, DRBD_F_REQUIRED))
GENL_notification(
DRBD_PEER_DEVICE_STATE, 37, events,
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_NOTIFICATION_HEADER, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_PEER_DEVICE_INFO, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_PEER_DEVICE_STATISTICS, DRBD_F_REQUIRED))
GENL_op(
DRBD_ADM_GET_INITIAL_STATE, 38,
GENL_op_init(
.dumpit = drbd_adm_get_initial_state,
),
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0))
GENL_notification(
DRBD_HELPER, 40, events,
GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)
GENL_tla_expected(DRBD_NLA_HELPER, DRBD_F_REQUIRED))
GENL_notification(
DRBD_INITIAL_STATE_DONE, 41, events,
GENL_tla_expected(DRBD_NLA_NOTIFICATION_HEADER, DRBD_F_REQUIRED))

56
include/linux/drbd_genl_api.h
View File

@@ -1,56 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef DRBD_GENL_STRUCT_H
#define DRBD_GENL_STRUCT_H
/**
* struct drbd_genlmsghdr - DRBD specific header used in NETLINK_GENERIC requests
* @minor:
* For admin requests (user -> kernel): which minor device to operate on.
* For (unicast) replies or informational (broadcast) messages
* (kernel -> user): which minor device the information is about.
* If we do not operate on minors, but on connections or resources,
* the minor value shall be (~0), and the attribute DRBD_NLA_CFG_CONTEXT
* is used instead.
* @flags: possible operation modifiers (relevant only for user->kernel):
* DRBD_GENL_F_SET_DEFAULTS
* @volume:
* When creating a new minor (adding it to a resource), the resource needs
* to know which volume number within the resource this is supposed to be.
* The volume number corresponds to the same volume number on the remote side,
* whereas the minor number on the remote side may be different
* (union with flags).
* @ret_code: kernel->userland unicast cfg reply return code (union with flags);
*/
struct drbd_genlmsghdr {
__u32 minor;
union {
__u32 flags;
__s32 ret_code;
};
};
/* To be used in drbd_genlmsghdr.flags */
enum {
DRBD_GENL_F_SET_DEFAULTS = 1,
};
enum drbd_state_info_bcast_reason {
SIB_GET_STATUS_REPLY = 1,
SIB_STATE_CHANGE = 2,
SIB_HELPER_PRE = 3,
SIB_HELPER_POST = 4,
SIB_SYNC_PROGRESS = 5,
};
/* hack around predefined gcc/cpp "linux=1",
* we cannot possibly include <1/drbd_genl.h> */
#undef linux
#include <linux/drbd.h>
#define GENL_MAGIC_VERSION 1
#define GENL_MAGIC_FAMILY drbd
#define GENL_MAGIC_FAMILY_HDRSZ sizeof(struct drbd_genlmsghdr)
#define GENL_MAGIC_INCLUDE_FILE <linux/drbd_genl.h>
#include <linux/genl_magic_struct.h>
#endif

413
include/linux/genl_magic_func.h
View File

@@ -1,413 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef GENL_MAGIC_FUNC_H
#define GENL_MAGIC_FUNC_H
#include <linux/args.h>
#include <linux/build_bug.h>
#include <linux/genl_magic_struct.h>
/*
* Magic: declare tla policy {{{1
* Magic: declare nested policies
* {{{2
*/
#undef GENL_mc_group
#define GENL_mc_group(group)
#undef GENL_notification
#define GENL_notification(op_name, op_num, mcast_group, tla_list)
#undef GENL_op
#define GENL_op(op_name, op_num, handler, tla_list)
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
[tag_name] = { .type = NLA_NESTED },
static struct nla_policy CONCATENATE(GENL_MAGIC_FAMILY, _tla_nl_policy)[] = {
#include GENL_MAGIC_INCLUDE_FILE
};
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
static struct nla_policy s_name ## _nl_policy[] __read_mostly = \
{ s_fields };
#undef __field
#define __field(attr_nr, attr_flag, name, nla_type, _type, __get, \
__put, __is_signed) \
[attr_nr] = { .type = nla_type },
#undef __array
#define __array(attr_nr, attr_flag, name, nla_type, _type, maxlen, \
__get, __put, __is_signed) \
[attr_nr] = { .type = nla_type, \
.len = maxlen - (nla_type == NLA_NUL_STRING) },
#include GENL_MAGIC_INCLUDE_FILE
#ifndef __KERNEL__
#ifndef pr_info
#define pr_info(args...) fprintf(stderr, args);
#endif
#endif
#ifdef GENL_MAGIC_DEBUG
static void dprint_field(const char *dir, int nla_type,
const char *name, void *valp)
{
__u64 val = valp ? *(__u32 *)valp : 1;
switch (nla_type) {
case NLA_U8: val = (__u8)val;
case NLA_U16: val = (__u16)val;
case NLA_U32: val = (__u32)val;
pr_info("%s attr %s: %d 0x%08x\n", dir,
name, (int)val, (unsigned)val);
break;
case NLA_U64:
val = *(__u64*)valp;
pr_info("%s attr %s: %lld 0x%08llx\n", dir,
name, (long long)val, (unsigned long long)val);
break;
case NLA_FLAG:
if (val)
pr_info("%s attr %s: set\n", dir, name);
break;
}
}
static void dprint_array(const char *dir, int nla_type,
const char *name, const char *val, unsigned len)
{
switch (nla_type) {
case NLA_NUL_STRING:
if (len && val[len-1] == '\0')
len--;
pr_info("%s attr %s: [len:%u] '%s'\n", dir, name, len, val);
break;
default:
/* we can always show 4 byte,
* thats what nlattr are aligned to. */
pr_info("%s attr %s: [len:%u] %02x%02x%02x%02x ...\n",
dir, name, len, val[0], val[1], val[2], val[3]);
}
}
#define DPRINT_TLA(a, op, b) pr_info("%s %s %s\n", a, op, b);
/* Name is a member field name of the struct s.
* If s is NULL (only parsing, no copy requested in *_from_attrs()),
* nla is supposed to point to the attribute containing the information
* corresponding to that struct member. */
#define DPRINT_FIELD(dir, nla_type, name, s, nla) \
do { \
if (s) \
dprint_field(dir, nla_type, #name, &s->name); \
else if (nla) \
dprint_field(dir, nla_type, #name, \
(nla_type == NLA_FLAG) ? NULL \
: nla_data(nla)); \
} while (0)
#define DPRINT_ARRAY(dir, nla_type, name, s, nla) \
do { \
if (s) \
dprint_array(dir, nla_type, #name, \
s->name, s->name ## _len); \
else if (nla) \
dprint_array(dir, nla_type, #name, \
nla_data(nla), nla_len(nla)); \
} while (0)
#else
#define DPRINT_TLA(a, op, b) do {} while (0)
#define DPRINT_FIELD(dir, nla_type, name, s, nla) do {} while (0)
#define DPRINT_ARRAY(dir, nla_type, name, s, nla) do {} while (0)
#endif
/*
* Magic: provide conversion functions {{{1
* populate struct from attribute table:
* {{{2
*/
/* processing of generic netlink messages is serialized.
* use one static buffer for parsing of nested attributes */
static struct nlattr *nested_attr_tb[128];
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
/* *_from_attrs functions are static, but potentially unused */ \
static int __ ## s_name ## _from_attrs(struct s_name *s, \
struct genl_info *info, bool exclude_invariants) \
{ \
const int maxtype = ARRAY_SIZE(s_name ## _nl_policy)-1; \
struct nlattr *tla = info->attrs[tag_number]; \
struct nlattr **ntb = nested_attr_tb; \
struct nlattr *nla; \
int err; \
BUILD_BUG_ON(ARRAY_SIZE(s_name ## _nl_policy) > ARRAY_SIZE(nested_attr_tb)); \
if (!tla) \
return -ENOMSG; \
DPRINT_TLA(#s_name, "<=-", #tag_name); \
err = nla_parse_nested_deprecated(ntb, maxtype, tla, \
s_name ## _nl_policy, NULL); \
if (err) \
return err; \
\
s_fields \
return 0; \
} __attribute__((unused)) \
static int s_name ## _from_attrs(struct s_name *s, \
struct genl_info *info) \
{ \
return __ ## s_name ## _from_attrs(s, info, false); \
} __attribute__((unused)) \
static int s_name ## _from_attrs_for_change(struct s_name *s, \
struct genl_info *info) \
{ \
return __ ## s_name ## _from_attrs(s, info, true); \
} __attribute__((unused)) \
#define __assign(attr_nr, attr_flag, name, nla_type, type, assignment...) \
nla = ntb[attr_nr]; \
if (nla) { \
if (exclude_invariants && !!((attr_flag) & DRBD_F_INVARIANT)) { \
pr_info("<< must not change invariant attr: %s\n", #name); \
return -EEXIST; \
} \
assignment; \
} else if (exclude_invariants && !!((attr_flag) & DRBD_F_INVARIANT)) { \
/* attribute missing from payload, */ \
/* which was expected */ \
} else if ((attr_flag) & DRBD_F_REQUIRED) { \
pr_info("<< missing attr: %s\n", #name); \
return -ENOMSG; \
}
#undef __field
#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \
__is_signed) \
__assign(attr_nr, attr_flag, name, nla_type, type, \
if (s) \
s->name = __get(nla); \
DPRINT_FIELD("<<", nla_type, name, s, nla))
/* validate_nla() already checked nla_len <= maxlen appropriately. */
#undef __array
#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \
__get, __put, __is_signed) \
__assign(attr_nr, attr_flag, name, nla_type, type, \
if (s) \
s->name ## _len = \
__get(s->name, nla, maxlen); \
DPRINT_ARRAY("<<", nla_type, name, s, nla))
#include GENL_MAGIC_INCLUDE_FILE
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields)
/*
* Magic: define op number to op name mapping {{{1
* {{{2
*/
static const char *CONCATENATE(GENL_MAGIC_FAMILY, _genl_cmd_to_str)(__u8 cmd)
{
switch (cmd) {
#undef GENL_op
#define GENL_op(op_name, op_num, handler, tla_list) \
case op_num: return #op_name;
#include GENL_MAGIC_INCLUDE_FILE
default:
return "unknown";
}
}
#ifdef __KERNEL__
#include <linux/stringify.h>
/*
* Magic: define genl_ops {{{1
* {{{2
*/
#undef GENL_op
#define GENL_op(op_name, op_num, handler, tla_list) \
{ \
handler \
.cmd = op_name, \
},
#define ZZZ_genl_ops CONCATENATE(GENL_MAGIC_FAMILY, _genl_ops)
static struct genl_ops ZZZ_genl_ops[] __read_mostly = {
#include GENL_MAGIC_INCLUDE_FILE
};
#undef GENL_op
#define GENL_op(op_name, op_num, handler, tla_list)
/*
* Define the genl_family, multicast groups, {{{1
* and provide register/unregister functions.
* {{{2
*/
#define ZZZ_genl_family CONCATENATE(GENL_MAGIC_FAMILY, _genl_family)
static struct genl_family ZZZ_genl_family;
/*
* Magic: define multicast groups
* Magic: define multicast group registration helper
*/
#define ZZZ_genl_mcgrps CONCATENATE(GENL_MAGIC_FAMILY, _genl_mcgrps)
static const struct genl_multicast_group ZZZ_genl_mcgrps[] = {
#undef GENL_mc_group
#define GENL_mc_group(group) { .name = #group, },
#include GENL_MAGIC_INCLUDE_FILE
};
enum CONCATENATE(GENL_MAGIC_FAMILY, group_ids) {
#undef GENL_mc_group
#define GENL_mc_group(group) CONCATENATE(GENL_MAGIC_FAMILY, _group_ ## group),
#include GENL_MAGIC_INCLUDE_FILE
};
#undef GENL_mc_group
#define GENL_mc_group(group) \
static int CONCATENATE(GENL_MAGIC_FAMILY, _genl_multicast_ ## group)( \
struct sk_buff *skb, gfp_t flags) \
{ \
unsigned int group_id = \
CONCATENATE(GENL_MAGIC_FAMILY, _group_ ## group); \
return genlmsg_multicast(&ZZZ_genl_family, skb, 0, \
group_id, flags); \
}
#include GENL_MAGIC_INCLUDE_FILE
#undef GENL_mc_group
#define GENL_mc_group(group)
static struct genl_family ZZZ_genl_family __ro_after_init = {
.name = __stringify(GENL_MAGIC_FAMILY),
.version = GENL_MAGIC_VERSION,
#ifdef GENL_MAGIC_FAMILY_HDRSZ
.hdrsize = NLA_ALIGN(GENL_MAGIC_FAMILY_HDRSZ),
#endif
.maxattr = ARRAY_SIZE(CONCATENATE(GENL_MAGIC_FAMILY, _tla_nl_policy))-1,
.policy = CONCATENATE(GENL_MAGIC_FAMILY, _tla_nl_policy),
#ifdef GENL_MAGIC_FAMILY_PRE_DOIT
.pre_doit = GENL_MAGIC_FAMILY_PRE_DOIT,
.post_doit = GENL_MAGIC_FAMILY_POST_DOIT,
#endif
.ops = ZZZ_genl_ops,
.n_ops = ARRAY_SIZE(ZZZ_genl_ops),
.mcgrps = ZZZ_genl_mcgrps,
.resv_start_op = 42, /* drbd is currently the only user */
.n_mcgrps = ARRAY_SIZE(ZZZ_genl_mcgrps),
.module = THIS_MODULE,
};
int CONCATENATE(GENL_MAGIC_FAMILY, _genl_register)(void)
{
return genl_register_family(&ZZZ_genl_family);
}
void CONCATENATE(GENL_MAGIC_FAMILY, _genl_unregister)(void)
{
genl_unregister_family(&ZZZ_genl_family);
}
/*
* Magic: provide conversion functions {{{1
* populate skb from struct.
* {{{2
*/
#undef GENL_op
#define GENL_op(op_name, op_num, handler, tla_list)
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
static int s_name ## _to_skb(struct sk_buff *skb, struct s_name *s, \
const bool exclude_sensitive) \
{ \
struct nlattr *tla = nla_nest_start(skb, tag_number); \
if (!tla) \
goto nla_put_failure; \
DPRINT_TLA(#s_name, "-=>", #tag_name); \
s_fields \
nla_nest_end(skb, tla); \
return 0; \
\
nla_put_failure: \
if (tla) \
nla_nest_cancel(skb, tla); \
return -EMSGSIZE; \
} \
static inline int s_name ## _to_priv_skb(struct sk_buff *skb, \
struct s_name *s) \
{ \
return s_name ## _to_skb(skb, s, 0); \
} \
static inline int s_name ## _to_unpriv_skb(struct sk_buff *skb, \
struct s_name *s) \
{ \
return s_name ## _to_skb(skb, s, 1); \
}
#undef __field
#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \
__is_signed) \
if (!exclude_sensitive || !((attr_flag) & DRBD_F_SENSITIVE)) { \
DPRINT_FIELD(">>", nla_type, name, s, NULL); \
if (__put(skb, attr_nr, s->name)) \
goto nla_put_failure; \
}
#undef __array
#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \
__get, __put, __is_signed) \
if (!exclude_sensitive || !((attr_flag) & DRBD_F_SENSITIVE)) { \
DPRINT_ARRAY(">>",nla_type, name, s, NULL); \
if (__put(skb, attr_nr, min_t(int, maxlen, \
s->name ## _len + (nla_type == NLA_NUL_STRING)),\
s->name)) \
goto nla_put_failure; \
}
#include GENL_MAGIC_INCLUDE_FILE
/* Functions for initializing structs to default values. */
#undef __field
#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \
__is_signed)
#undef __array
#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \
__get, __put, __is_signed)
#undef __u32_field_def
#define __u32_field_def(attr_nr, attr_flag, name, default) \
x->name = default;
#undef __s32_field_def
#define __s32_field_def(attr_nr, attr_flag, name, default) \
x->name = default;
#undef __flg_field_def
#define __flg_field_def(attr_nr, attr_flag, name, default) \
x->name = default;
#undef __str_field_def
#define __str_field_def(attr_nr, attr_flag, name, maxlen) \
memset(x->name, 0, sizeof(x->name)); \
x->name ## _len = 0;
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
static void set_ ## s_name ## _defaults(struct s_name *x) __attribute__((unused)); \
static void set_ ## s_name ## _defaults(struct s_name *x) { \
s_fields \
}
#include GENL_MAGIC_INCLUDE_FILE
#endif /* __KERNEL__ */
/* }}}1 */
#endif /* GENL_MAGIC_FUNC_H */

272
include/linux/genl_magic_struct.h
View File

@@ -1,272 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef GENL_MAGIC_STRUCT_H
#define GENL_MAGIC_STRUCT_H
#ifndef GENL_MAGIC_FAMILY
# error "you need to define GENL_MAGIC_FAMILY before inclusion"
#endif
#ifndef GENL_MAGIC_VERSION
# error "you need to define GENL_MAGIC_VERSION before inclusion"
#endif
#ifndef GENL_MAGIC_INCLUDE_FILE
# error "you need to define GENL_MAGIC_INCLUDE_FILE before inclusion"
#endif
#include <linux/args.h>
#include <linux/types.h>
#include <net/genetlink.h>
extern int CONCATENATE(GENL_MAGIC_FAMILY, _genl_register)(void);
extern void CONCATENATE(GENL_MAGIC_FAMILY, _genl_unregister)(void);
/*
* Extension of genl attribute validation policies {{{2
*/
/*
* Flags specific to drbd and not visible at the netlink layer, used in
* <struct>_from_attrs and <struct>_to_skb:
*
* @DRBD_F_REQUIRED: Attribute is required; a request without this attribute is
* invalid.
*
* @DRBD_F_SENSITIVE: Attribute includes sensitive information and must not be
* included in unpriviledged get requests or broadcasts.
*
* @DRBD_F_INVARIANT: Attribute is set when an object is initially created, but
* cannot subsequently be changed.
*/
#define DRBD_F_REQUIRED (1 << 0)
#define DRBD_F_SENSITIVE (1 << 1)
#define DRBD_F_INVARIANT (1 << 2)
/* }}}1
* MAGIC
* multi-include macro expansion magic starts here
*/
/* MAGIC helpers {{{2 */
static inline int nla_put_u64_0pad(struct sk_buff *skb, int attrtype, u64 value)
{
return nla_put_64bit(skb, attrtype, sizeof(u64), &value, 0);
}
/* possible field types */
#define __flg_field(attr_nr, attr_flag, name) \
__field(attr_nr, attr_flag, name, NLA_U8, char, \
nla_get_u8, nla_put_u8, false)
#define __u8_field(attr_nr, attr_flag, name) \
__field(attr_nr, attr_flag, name, NLA_U8, unsigned char, \
nla_get_u8, nla_put_u8, false)
#define __u16_field(attr_nr, attr_flag, name) \
__field(attr_nr, attr_flag, name, NLA_U16, __u16, \
nla_get_u16, nla_put_u16, false)
#define __u32_field(attr_nr, attr_flag, name) \
__field(attr_nr, attr_flag, name, NLA_U32, __u32, \
nla_get_u32, nla_put_u32, false)
#define __s32_field(attr_nr, attr_flag, name) \
__field(attr_nr, attr_flag, name, NLA_U32, __s32, \
nla_get_u32, nla_put_u32, true)
#define __u64_field(attr_nr, attr_flag, name) \
__field(attr_nr, attr_flag, name, NLA_U64, __u64, \
nla_get_u64, nla_put_u64_0pad, false)
#define __str_field(attr_nr, attr_flag, name, maxlen) \
__array(attr_nr, attr_flag, name, NLA_NUL_STRING, char, maxlen, \
nla_strscpy, nla_put, false)
#define __bin_field(attr_nr, attr_flag, name, maxlen) \
__array(attr_nr, attr_flag, name, NLA_BINARY, char, maxlen, \
nla_memcpy, nla_put, false)
/* fields with default values */
#define __flg_field_def(attr_nr, attr_flag, name, default) \
__flg_field(attr_nr, attr_flag, name)
#define __u32_field_def(attr_nr, attr_flag, name, default) \
__u32_field(attr_nr, attr_flag, name)
#define __s32_field_def(attr_nr, attr_flag, name, default) \
__s32_field(attr_nr, attr_flag, name)
#define __str_field_def(attr_nr, attr_flag, name, maxlen) \
__str_field(attr_nr, attr_flag, name, maxlen)
#define GENL_op_init(args...) args
#define GENL_doit(handler) \
.doit = handler, \
.flags = GENL_ADMIN_PERM,
#define GENL_dumpit(handler) \
.dumpit = handler, \
.flags = GENL_ADMIN_PERM,
/* }}}1
* Magic: define the enum symbols for genl_ops
* Magic: define the enum symbols for top level attributes
* Magic: define the enum symbols for nested attributes
* {{{2
*/
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields)
#undef GENL_mc_group
#define GENL_mc_group(group)
#undef GENL_notification
#define GENL_notification(op_name, op_num, mcast_group, tla_list) \
op_name = op_num,
#undef GENL_op
#define GENL_op(op_name, op_num, handler, tla_list) \
op_name = op_num,
enum {
#include GENL_MAGIC_INCLUDE_FILE
};
#undef GENL_notification
#define GENL_notification(op_name, op_num, mcast_group, tla_list)
#undef GENL_op
#define GENL_op(op_name, op_num, handler, attr_list)
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
tag_name = tag_number,
enum {
#include GENL_MAGIC_INCLUDE_FILE
};
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
enum { \
s_fields \
};
#undef __field
#define __field(attr_nr, attr_flag, name, nla_type, type, \
__get, __put, __is_signed) \
T_ ## name = (__u16)(attr_nr),
#undef __array
#define __array(attr_nr, attr_flag, name, nla_type, type, \
maxlen, __get, __put, __is_signed) \
T_ ## name = (__u16)(attr_nr),
#include GENL_MAGIC_INCLUDE_FILE
/* }}}1
* Magic: compile time assert unique numbers for operations
* Magic: -"- unique numbers for top level attributes
* Magic: -"- unique numbers for nested attributes
* {{{2
*/
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields)
#undef GENL_op
#define GENL_op(op_name, op_num, handler, attr_list) \
case op_name:
#undef GENL_notification
#define GENL_notification(op_name, op_num, mcast_group, tla_list) \
case op_name:
static inline void ct_assert_unique_operations(void)
{
switch (0) {
#include GENL_MAGIC_INCLUDE_FILE
case 0:
;
}
}
#undef GENL_op
#define GENL_op(op_name, op_num, handler, attr_list)
#undef GENL_notification
#define GENL_notification(op_name, op_num, mcast_group, tla_list)
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
case tag_number:
static inline void ct_assert_unique_top_level_attributes(void)
{
switch (0) {
#include GENL_MAGIC_INCLUDE_FILE
case 0:
;
}
}
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
static inline void ct_assert_unique_ ## s_name ## _attributes(void) \
{ \
switch (0) { \
s_fields \
case 0: \
; \
} \
}
#undef __field
#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \
__is_signed) \
case attr_nr:
#undef __array
#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \
__get, __put, __is_signed) \
case attr_nr:
#include GENL_MAGIC_INCLUDE_FILE
/* }}}1
* Magic: declare structs
* struct <name> {
* fields
* };
* {{{2
*/
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
struct s_name { s_fields };
#undef __field
#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \
__is_signed) \
type name;
#undef __array
#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \
__get, __put, __is_signed) \
type name[maxlen]; \
__u32 name ## _len;
#include GENL_MAGIC_INCLUDE_FILE
#undef GENL_struct
#define GENL_struct(tag_name, tag_number, s_name, s_fields) \
enum { \
s_fields \
};
#undef __field
#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \
is_signed) \
F_ ## name ## _IS_SIGNED = is_signed,
#undef __array
#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \
__get, __put, is_signed) \
F_ ## name ## _IS_SIGNED = is_signed,
#include GENL_MAGIC_INCLUDE_FILE
/* }}}1 */
#endif /* GENL_MAGIC_STRUCT_H */

59
include/trace/events/block.h
View File

@@ -226,6 +226,65 @@ DECLARE_EVENT_CLASS(block_rq,
IOPRIO_PRIO_LEVEL(__entry->ioprio), __entry->comm)
);
/**
* block_rq_tag_wait - triggered when a request is starved of a tag
* @q: request queue of the target device
* @hctx: hardware context of the request experiencing starvation
* @is_sched_tag: indicates whether the starved pool is the software scheduler
* @alloc_flags: allocation flags dictating the specific tag pool
*
* Called immediately before the submitting context is forced to block due
* to the exhaustion of available tags (i.e., physical hardware driver
* tags, software scheduler tags, or reserved tags). This trace point
* indicates that the context will be placed into an uninterruptible state
* via sbitmap_prepare_to_wait(). If a tag is not acquired in the final
* lockless retry, the context will yield the CPU via io_schedule() until
* an active request completes and relinquishes its assigned tag.
*/
TRACE_EVENT(block_rq_tag_wait,
TP_PROTO(struct request_queue *q, struct blk_mq_hw_ctx *hctx,
bool is_sched_tag, unsigned int alloc_flags),
TP_ARGS(q, hctx, is_sched_tag, alloc_flags),
TP_STRUCT__entry(
__field( dev_t, dev )
__field( u32, hctx_id )
__field( u32, nr_tags )
__field( bool, is_sched_tag )
__field( bool, is_reserved )
),
TP_fast_assign(
__entry->dev = q->disk ? disk_devt(q->disk) : 0;
__entry->hctx_id = hctx->queue_num;
__entry->is_sched_tag = is_sched_tag;
__entry->is_reserved = alloc_flags & BLK_MQ_REQ_RESERVED;
if (__entry->is_reserved) {
__entry->nr_tags = is_sched_tag ?
hctx->sched_tags->nr_reserved_tags :
hctx->tags->nr_reserved_tags;
} else {
if (is_sched_tag)
__entry->nr_tags = hctx->sched_tags->nr_tags -
hctx->sched_tags->nr_reserved_tags;
else
__entry->nr_tags = hctx->tags->nr_tags -
hctx->tags->nr_reserved_tags;
}
),
TP_printk("%d,%d hctx=%u starved on %s%s tags (depth=%u)",
MAJOR(__entry->dev), MINOR(__entry->dev),
__entry->hctx_id,
__entry->is_sched_tag ? "scheduler" : "hardware",
__entry->is_reserved ? " reserved" : "",
__entry->nr_tags)
);
/**
* block_rq_insert - insert block operation request into queue
* @rq: block IO operation request

73
include/linux/drbd.h → include/uapi/linux/drbd.h
View File

@@ -1,4 +1,4 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/* SPDX-License-Identifier: GPL-2.0-or-later WITH Linux-syscall-note */
/*
drbd.h
Kernel module for 2.6.x Kernels
@@ -11,32 +11,10 @@
*/
#ifndef DRBD_H
#define DRBD_H
#include <asm/types.h>
#ifdef __KERNEL__
#ifndef _UAPI_LINUX_DRBD_H
#define _UAPI_LINUX_DRBD_H
#include <linux/types.h>
#include <asm/byteorder.h>
#else
#include <sys/types.h>
#include <sys/wait.h>
#include <limits.h>
/* Although the Linux source code makes a difference between
generic endianness and the bitfields' endianness, there is no
architecture as of Linux-2.6.24-rc4 where the bitfields' endianness
does not match the generic endianness. */
#if __BYTE_ORDER == __LITTLE_ENDIAN
#define __LITTLE_ENDIAN_BITFIELD
#elif __BYTE_ORDER == __BIG_ENDIAN
#define __BIG_ENDIAN_BITFIELD
#else
# error "sorry, weird endianness on this box"
#endif
#endif
enum drbd_io_error_p {
EP_PASS_ON, /* FIXME should the better be named "Ignore"? */
@@ -333,6 +311,9 @@ enum drbd_uuid_index {
#define HISTORY_UUIDS MAX_PEERS
#define DRBD_NL_UUIDS_SIZE (UI_SIZE * sizeof(__u64))
#define DRBD_NL_HISTORY_UUIDS_SIZE (HISTORY_UUIDS * sizeof(__u64))
enum drbd_timeout_flag {
UT_DEFAULT = 0,
UT_DEGRADED = 1,
@@ -389,4 +370,44 @@ enum write_ordering_e {
#define DRBD_CPU_MASK_SIZE 32
#endif
/**
* struct drbd_genlmsghdr - DRBD specific header used in NETLINK_GENERIC requests
* @minor:
* For admin requests (user -> kernel): which minor device to operate on.
* For (unicast) replies or informational (broadcast) messages
* (kernel -> user): which minor device the information is about.
* If we do not operate on minors, but on connections or resources,
* the minor value shall be (~0), and the attribute DRBD_NLA_CFG_CONTEXT
* is used instead.
* @flags: possible operation modifiers (relevant only for user->kernel):
* DRBD_GENL_F_SET_DEFAULTS
* @volume:
* When creating a new minor (adding it to a resource), the resource needs
* to know which volume number within the resource this is supposed to be.
* The volume number corresponds to the same volume number on the remote side,
* whereas the minor number on the remote side may be different
* (union with flags).
* @ret_code: kernel->userland unicast cfg reply return code (union with flags);
*/
struct drbd_genlmsghdr {
__u32 minor;
union {
__u32 flags;
__s32 ret_code;
};
};
/* To be used in drbd_genlmsghdr.flags */
enum {
DRBD_GENL_F_SET_DEFAULTS = 1,
};
enum drbd_state_info_bcast_reason {
SIB_GET_STATUS_REPLY = 1,
SIB_STATE_CHANGE = 2,
SIB_HELPER_PRE = 3,
SIB_HELPER_POST = 4,
SIB_SYNC_PROGRESS = 5,
};
#endif /* _UAPI_LINUX_DRBD_H */

359
include/uapi/linux/drbd_genl.h Normal file
View File

@@ -0,0 +1,359 @@
/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
#ifndef _UAPI_LINUX_DRBD_GENL_H
#define _UAPI_LINUX_DRBD_GENL_H
#define DRBD_FAMILY_NAME "drbd"
#define DRBD_FAMILY_VERSION 1
enum {
DRBD_NLA_CFG_REPLY = 1,
DRBD_NLA_CFG_CONTEXT,
DRBD_NLA_DISK_CONF,
DRBD_NLA_RESOURCE_OPTS,
DRBD_NLA_NET_CONF,
DRBD_NLA_SET_ROLE_PARMS,
DRBD_NLA_RESIZE_PARMS,
DRBD_NLA_STATE_INFO,
DRBD_NLA_START_OV_PARMS,
DRBD_NLA_NEW_C_UUID_PARMS,
DRBD_NLA_TIMEOUT_PARMS,
DRBD_NLA_DISCONNECT_PARMS,
DRBD_NLA_DETACH_PARMS,
DRBD_NLA_RESOURCE_INFO = 15,
DRBD_NLA_DEVICE_INFO,
DRBD_NLA_CONNECTION_INFO,
DRBD_NLA_PEER_DEVICE_INFO,
DRBD_NLA_RESOURCE_STATISTICS,
DRBD_NLA_DEVICE_STATISTICS,
DRBD_NLA_CONNECTION_STATISTICS,
DRBD_NLA_PEER_DEVICE_STATISTICS,
DRBD_NLA_NOTIFICATION_HEADER,
DRBD_NLA_HELPER,
__DRBD_NLA_MAX,
DRBD_NLA_MAX = (__DRBD_NLA_MAX - 1)
};
enum {
DRBD_A_DRBD_CFG_REPLY_INFO_TEXT = 1,
__DRBD_A_DRBD_CFG_REPLY_MAX,
DRBD_A_DRBD_CFG_REPLY_MAX = (__DRBD_A_DRBD_CFG_REPLY_MAX - 1)
};
enum {
DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME = 1,
DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME,
DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR,
DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR,
__DRBD_A_DRBD_CFG_CONTEXT_MAX,
DRBD_A_DRBD_CFG_CONTEXT_MAX = (__DRBD_A_DRBD_CFG_CONTEXT_MAX - 1)
};
enum {
DRBD_A_DISK_CONF_BACKING_DEV = 1,
DRBD_A_DISK_CONF_META_DEV,
DRBD_A_DISK_CONF_META_DEV_IDX,
DRBD_A_DISK_CONF_DISK_SIZE,
DRBD_A_DISK_CONF_MAX_BIO_BVECS,
DRBD_A_DISK_CONF_ON_IO_ERROR,
DRBD_A_DISK_CONF_FENCING,
DRBD_A_DISK_CONF_RESYNC_RATE,
DRBD_A_DISK_CONF_RESYNC_AFTER,
DRBD_A_DISK_CONF_AL_EXTENTS,
DRBD_A_DISK_CONF_C_PLAN_AHEAD,
DRBD_A_DISK_CONF_C_DELAY_TARGET,
DRBD_A_DISK_CONF_C_FILL_TARGET,
DRBD_A_DISK_CONF_C_MAX_RATE,
DRBD_A_DISK_CONF_C_MIN_RATE,
DRBD_A_DISK_CONF_DISK_BARRIER,
DRBD_A_DISK_CONF_DISK_FLUSHES,
DRBD_A_DISK_CONF_DISK_DRAIN,
DRBD_A_DISK_CONF_MD_FLUSHES,
DRBD_A_DISK_CONF_DISK_TIMEOUT,
DRBD_A_DISK_CONF_READ_BALANCING,
DRBD_A_DISK_CONF_AL_UPDATES = 23,
DRBD_A_DISK_CONF_DISCARD_ZEROES_IF_ALIGNED,
DRBD_A_DISK_CONF_RS_DISCARD_GRANULARITY,
DRBD_A_DISK_CONF_DISABLE_WRITE_SAME,
__DRBD_A_DISK_CONF_MAX,
DRBD_A_DISK_CONF_MAX = (__DRBD_A_DISK_CONF_MAX - 1)
};
enum {
DRBD_A_RES_OPTS_CPU_MASK = 1,
DRBD_A_RES_OPTS_ON_NO_DATA,
__DRBD_A_RES_OPTS_MAX,
DRBD_A_RES_OPTS_MAX = (__DRBD_A_RES_OPTS_MAX - 1)
};
enum {
DRBD_A_NET_CONF_SHARED_SECRET = 1,
DRBD_A_NET_CONF_CRAM_HMAC_ALG,
DRBD_A_NET_CONF_INTEGRITY_ALG,
DRBD_A_NET_CONF_VERIFY_ALG,
DRBD_A_NET_CONF_CSUMS_ALG,
DRBD_A_NET_CONF_WIRE_PROTOCOL,
DRBD_A_NET_CONF_CONNECT_INT,
DRBD_A_NET_CONF_TIMEOUT,
DRBD_A_NET_CONF_PING_INT,
DRBD_A_NET_CONF_PING_TIMEO,
DRBD_A_NET_CONF_SNDBUF_SIZE,
DRBD_A_NET_CONF_RCVBUF_SIZE,
DRBD_A_NET_CONF_KO_COUNT,
DRBD_A_NET_CONF_MAX_BUFFERS,
DRBD_A_NET_CONF_MAX_EPOCH_SIZE,
DRBD_A_NET_CONF_UNPLUG_WATERMARK,
DRBD_A_NET_CONF_AFTER_SB_0P,
DRBD_A_NET_CONF_AFTER_SB_1P,
DRBD_A_NET_CONF_AFTER_SB_2P,
DRBD_A_NET_CONF_RR_CONFLICT,
DRBD_A_NET_CONF_ON_CONGESTION,
DRBD_A_NET_CONF_CONG_FILL,
DRBD_A_NET_CONF_CONG_EXTENTS,
DRBD_A_NET_CONF_TWO_PRIMARIES,
DRBD_A_NET_CONF_DISCARD_MY_DATA,
DRBD_A_NET_CONF_TCP_CORK,
DRBD_A_NET_CONF_ALWAYS_ASBP,
DRBD_A_NET_CONF_TENTATIVE,
DRBD_A_NET_CONF_USE_RLE,
DRBD_A_NET_CONF_CSUMS_AFTER_CRASH_ONLY = 33,
DRBD_A_NET_CONF_SOCK_CHECK_TIMEO,
__DRBD_A_NET_CONF_MAX,
DRBD_A_NET_CONF_MAX = (__DRBD_A_NET_CONF_MAX - 1)
};
enum {
DRBD_A_SET_ROLE_PARMS_ASSUME_UPTODATE = 1,
__DRBD_A_SET_ROLE_PARMS_MAX,
DRBD_A_SET_ROLE_PARMS_MAX = (__DRBD_A_SET_ROLE_PARMS_MAX - 1)
};
enum {
DRBD_A_RESIZE_PARMS_RESIZE_SIZE = 1,
DRBD_A_RESIZE_PARMS_RESIZE_FORCE,
DRBD_A_RESIZE_PARMS_NO_RESYNC,
DRBD_A_RESIZE_PARMS_AL_STRIPES,
DRBD_A_RESIZE_PARMS_AL_STRIPE_SIZE,
__DRBD_A_RESIZE_PARMS_MAX,
DRBD_A_RESIZE_PARMS_MAX = (__DRBD_A_RESIZE_PARMS_MAX - 1)
};
enum {
DRBD_A_STATE_INFO_SIB_REASON = 1,
DRBD_A_STATE_INFO_CURRENT_STATE,
DRBD_A_STATE_INFO_CAPACITY,
DRBD_A_STATE_INFO_ED_UUID,
DRBD_A_STATE_INFO_PREV_STATE,
DRBD_A_STATE_INFO_NEW_STATE,
DRBD_A_STATE_INFO_UUIDS,
DRBD_A_STATE_INFO_DISK_FLAGS,
DRBD_A_STATE_INFO_BITS_TOTAL,
DRBD_A_STATE_INFO_BITS_OOS,
DRBD_A_STATE_INFO_BITS_RS_TOTAL,
DRBD_A_STATE_INFO_BITS_RS_FAILED,
DRBD_A_STATE_INFO_HELPER,
DRBD_A_STATE_INFO_HELPER_EXIT_CODE,
DRBD_A_STATE_INFO_SEND_CNT,
DRBD_A_STATE_INFO_RECV_CNT,
DRBD_A_STATE_INFO_READ_CNT,
DRBD_A_STATE_INFO_WRIT_CNT,
DRBD_A_STATE_INFO_AL_WRIT_CNT,
DRBD_A_STATE_INFO_BM_WRIT_CNT,
DRBD_A_STATE_INFO_AP_BIO_CNT,
DRBD_A_STATE_INFO_AP_PENDING_CNT,
DRBD_A_STATE_INFO_RS_PENDING_CNT,
__DRBD_A_STATE_INFO_MAX,
DRBD_A_STATE_INFO_MAX = (__DRBD_A_STATE_INFO_MAX - 1)
};
enum {
DRBD_A_START_OV_PARMS_OV_START_SECTOR = 1,
DRBD_A_START_OV_PARMS_OV_STOP_SECTOR,
__DRBD_A_START_OV_PARMS_MAX,
DRBD_A_START_OV_PARMS_MAX = (__DRBD_A_START_OV_PARMS_MAX - 1)
};
enum {
DRBD_A_NEW_C_UUID_PARMS_CLEAR_BM = 1,
__DRBD_A_NEW_C_UUID_PARMS_MAX,
DRBD_A_NEW_C_UUID_PARMS_MAX = (__DRBD_A_NEW_C_UUID_PARMS_MAX - 1)
};
enum {
DRBD_A_TIMEOUT_PARMS_TIMEOUT_TYPE = 1,
__DRBD_A_TIMEOUT_PARMS_MAX,
DRBD_A_TIMEOUT_PARMS_MAX = (__DRBD_A_TIMEOUT_PARMS_MAX - 1)
};
enum {
DRBD_A_DISCONNECT_PARMS_FORCE_DISCONNECT = 1,
__DRBD_A_DISCONNECT_PARMS_MAX,
DRBD_A_DISCONNECT_PARMS_MAX = (__DRBD_A_DISCONNECT_PARMS_MAX - 1)
};
enum {
DRBD_A_DETACH_PARMS_FORCE_DETACH = 1,
__DRBD_A_DETACH_PARMS_MAX,
DRBD_A_DETACH_PARMS_MAX = (__DRBD_A_DETACH_PARMS_MAX - 1)
};
enum {
DRBD_A_RESOURCE_INFO_RES_ROLE = 1,
DRBD_A_RESOURCE_INFO_RES_SUSP,
DRBD_A_RESOURCE_INFO_RES_SUSP_NOD,
DRBD_A_RESOURCE_INFO_RES_SUSP_FEN,
__DRBD_A_RESOURCE_INFO_MAX,
DRBD_A_RESOURCE_INFO_MAX = (__DRBD_A_RESOURCE_INFO_MAX - 1)
};
enum {
DRBD_A_DEVICE_INFO_DEV_DISK_STATE = 1,
__DRBD_A_DEVICE_INFO_MAX,
DRBD_A_DEVICE_INFO_MAX = (__DRBD_A_DEVICE_INFO_MAX - 1)
};
enum {
DRBD_A_CONNECTION_INFO_CONN_CONNECTION_STATE = 1,
DRBD_A_CONNECTION_INFO_CONN_ROLE,
__DRBD_A_CONNECTION_INFO_MAX,
DRBD_A_CONNECTION_INFO_MAX = (__DRBD_A_CONNECTION_INFO_MAX - 1)
};
enum {
DRBD_A_PEER_DEVICE_INFO_PEER_REPL_STATE = 1,
DRBD_A_PEER_DEVICE_INFO_PEER_DISK_STATE,
DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_USER,
DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_PEER,
DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_DEPENDENCY,
__DRBD_A_PEER_DEVICE_INFO_MAX,
DRBD_A_PEER_DEVICE_INFO_MAX = (__DRBD_A_PEER_DEVICE_INFO_MAX - 1)
};
enum {
DRBD_A_RESOURCE_STATISTICS_RES_STAT_WRITE_ORDERING = 1,
__DRBD_A_RESOURCE_STATISTICS_MAX,
DRBD_A_RESOURCE_STATISTICS_MAX = (__DRBD_A_RESOURCE_STATISTICS_MAX - 1)
};
enum {
DRBD_A_DEVICE_STATISTICS_DEV_SIZE = 1,
DRBD_A_DEVICE_STATISTICS_DEV_READ,
DRBD_A_DEVICE_STATISTICS_DEV_WRITE,
DRBD_A_DEVICE_STATISTICS_DEV_AL_WRITES,
DRBD_A_DEVICE_STATISTICS_DEV_BM_WRITES,
DRBD_A_DEVICE_STATISTICS_DEV_UPPER_PENDING,
DRBD_A_DEVICE_STATISTICS_DEV_LOWER_PENDING,
DRBD_A_DEVICE_STATISTICS_DEV_UPPER_BLOCKED,
DRBD_A_DEVICE_STATISTICS_DEV_LOWER_BLOCKED,
DRBD_A_DEVICE_STATISTICS_DEV_AL_SUSPENDED,
DRBD_A_DEVICE_STATISTICS_DEV_EXPOSED_DATA_UUID,
DRBD_A_DEVICE_STATISTICS_DEV_CURRENT_UUID,
DRBD_A_DEVICE_STATISTICS_DEV_DISK_FLAGS,
DRBD_A_DEVICE_STATISTICS_HISTORY_UUIDS,
__DRBD_A_DEVICE_STATISTICS_MAX,
DRBD_A_DEVICE_STATISTICS_MAX = (__DRBD_A_DEVICE_STATISTICS_MAX - 1)
};
enum {
DRBD_A_CONNECTION_STATISTICS_CONN_CONGESTED = 1,
__DRBD_A_CONNECTION_STATISTICS_MAX,
DRBD_A_CONNECTION_STATISTICS_MAX = (__DRBD_A_CONNECTION_STATISTICS_MAX - 1)
};
enum {
DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_RECEIVED = 1,
DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_SENT,
DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_PENDING,
DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_UNACKED,
DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_OUT_OF_SYNC,
DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_RESYNC_FAILED,
DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_BITMAP_UUID,
DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_FLAGS = 9,
__DRBD_A_PEER_DEVICE_STATISTICS_MAX,
DRBD_A_PEER_DEVICE_STATISTICS_MAX = (__DRBD_A_PEER_DEVICE_STATISTICS_MAX - 1)
};
enum {
DRBD_A_DRBD_NOTIFICATION_HEADER_NH_TYPE = 1,
__DRBD_A_DRBD_NOTIFICATION_HEADER_MAX,
DRBD_A_DRBD_NOTIFICATION_HEADER_MAX = (__DRBD_A_DRBD_NOTIFICATION_HEADER_MAX - 1)
};
enum {
DRBD_A_DRBD_HELPER_INFO_HELPER_NAME = 1,
DRBD_A_DRBD_HELPER_INFO_HELPER_STATUS,
__DRBD_A_DRBD_HELPER_INFO_MAX,
DRBD_A_DRBD_HELPER_INFO_MAX = (__DRBD_A_DRBD_HELPER_INFO_MAX - 1)
};
enum {
DRBD_ADM_EVENT = 1,
DRBD_ADM_GET_STATUS,
DRBD_ADM_NEW_MINOR = 5,
DRBD_ADM_DEL_MINOR,
DRBD_ADM_NEW_RESOURCE,
DRBD_ADM_DEL_RESOURCE,
DRBD_ADM_RESOURCE_OPTS,
DRBD_ADM_CONNECT,
DRBD_ADM_DISCONNECT,
DRBD_ADM_ATTACH,
DRBD_ADM_RESIZE,
DRBD_ADM_PRIMARY,
DRBD_ADM_SECONDARY,
DRBD_ADM_NEW_C_UUID,
DRBD_ADM_START_OV,
DRBD_ADM_DETACH,
DRBD_ADM_INVALIDATE,
DRBD_ADM_INVAL_PEER,
DRBD_ADM_PAUSE_SYNC,
DRBD_ADM_RESUME_SYNC,
DRBD_ADM_SUSPEND_IO,
DRBD_ADM_RESUME_IO,
DRBD_ADM_OUTDATE,
DRBD_ADM_GET_TIMEOUT_TYPE,
DRBD_ADM_DOWN,
DRBD_ADM_CHG_DISK_OPTS,
DRBD_ADM_CHG_NET_OPTS,
DRBD_ADM_GET_RESOURCES,
DRBD_ADM_GET_DEVICES,
DRBD_ADM_GET_CONNECTIONS,
DRBD_ADM_GET_PEER_DEVICES,
DRBD_ADM_RESOURCE_STATE,
DRBD_ADM_DEVICE_STATE,
DRBD_ADM_CONNECTION_STATE,
DRBD_ADM_PEER_DEVICE_STATE,
DRBD_ADM_GET_INITIAL_STATE,
DRBD_ADM_HELPER = 40,
DRBD_ADM_INITIAL_STATE_DONE,
__DRBD_ADM_MAX,
DRBD_ADM_MAX = (__DRBD_ADM_MAX - 1)
};
#define DRBD_MCGRP_EVENTS "events"
#endif /* _UAPI_LINUX_DRBD_GENL_H */

10
include/linux/drbd_limits.h → include/uapi/linux/drbd_limits.h
View File

@@ -1,4 +1,4 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */
/*
drbd_limits.h
This file is part of DRBD by Philipp Reisner and Lars Ellenberg.
@@ -11,10 +11,10 @@
* feedback about nonsense settings for certain configurable values.
*/
#ifndef DRBD_LIMITS_H
#define DRBD_LIMITS_H 1
#ifndef _UAPI_LINUX_DRBD_LIMITS_H
#define _UAPI_LINUX_DRBD_LIMITS_H
#define DEBUG_RANGE_CHECK 0
#include <linux/drbd.h>
#define DRBD_MINOR_COUNT_MIN 1U
#define DRBD_MINOR_COUNT_MAX 255U
@@ -248,4 +248,4 @@
#define DRBD_RS_DISCARD_GRANULARITY_DEF 0U /* disabled by default */
#define DRBD_RS_DISCARD_GRANULARITY_SCALE '1' /* bytes */
#endif
#endif /* _UAPI_LINUX_DRBD_LIMITS_H */

2
io_uring/rsrc.c
View File

@@ -152,7 +152,7 @@ static void io_release_ubuf(void *priv)
unsigned int i;
for (i = 0; i < imu->nr_bvecs; i++) {
struct folio *folio = page_folio(imu->bvec[i].bv_page);
struct folio *folio = bvec_folio(&imu->bvec[i]);
unpin_user_folio(folio, 1);
}

4
mm/page_io.c
View File

@@ -490,7 +490,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)
if (ret == sio->len) {
for (p = 0; p < sio->pages; p++) {
struct folio *folio = page_folio(sio->bvec[p].bv_page);
struct folio *folio = bvec_folio(&sio->bvec[p]);
count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
@@ -500,7 +500,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret)
count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT);
} else {
for (p = 0; p < sio->pages; p++) {
struct folio *folio = page_folio(sio->bvec[p].bv_page);
struct folio *folio = bvec_folio(&sio->bvec[p]);
folio_unlock(folio);
}

20
rust/kernel/block/mq/gen_disk.rs
View File

@@ -150,6 +150,19 @@ pub fn build<T: Operations>(
// SAFETY: `gendisk` is a valid pointer as we initialized it above
unsafe { (*gendisk).fops = &TABLE };
let cleanup_failure = ScopeGuard::new_with_data((gendisk, data), |(gendisk, data)| {
// SAFETY: `gendisk` came from `__blk_mq_alloc_disk()` above and
// has not been added to the VFS on this cleanup path.
unsafe { bindings::put_disk(gendisk) };
// SAFETY: `data` came from `into_foreign()` above and has not been
// converted back on this cleanup path.
drop(unsafe { T::QueueData::from_foreign(data) });
});
// The failure guard now owns both pieces of cleanup; the early guard
// must not run on this path anymore.
recover_data.dismiss();
let mut writer = NullTerminatedFormatter::new(
// SAFETY: `gendisk` points to a valid and initialized instance. We
// have exclusive access, since the disk is not added to the VFS
@@ -172,7 +185,7 @@ pub fn build<T: Operations>(
},
)?;
recover_data.dismiss();
cleanup_failure.dismiss();
// INVARIANT: `gendisk` was initialized above.
// INVARIANT: `gendisk` was added to the VFS via `device_add_disk` above.
@@ -215,6 +228,11 @@ fn drop(&mut self) {
// to the VFS.
unsafe { bindings::del_gendisk(self.gendisk) };
// SAFETY: By type invariant, `self.gendisk` was added to the VFS, so
// `put_disk()` must follow `del_gendisk()` to drop the final gendisk
// reference and trigger the remaining release path.
unsafe { bindings::put_disk(self.gendisk) };
// SAFETY: `queue.queuedata` was created by `GenDiskBuilder::build` with
// a call to `ForeignOwnable::into_foreign` to create `queuedata`.
// `ForeignOwnable::from_foreign` is only called here.

2
rust/kernel/block/mq/operations.rs
View File

@@ -218,7 +218,7 @@ impl<T: Operations> OperationsVTable<T> {
_set: *mut bindings::blk_mq_tag_set,
rq: *mut bindings::request,
_hctx_idx: crate::ffi::c_uint,
_numa_node: crate::ffi::c_uint,
_numa_node: crate::ffi::c_int,
) -> crate::ffi::c_int {
from_result(|| {
// SAFETY: By the safety requirements of this function, `rq` points