mirrors/linux
2
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-08-27 07:04:21 -04:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'fuse-update-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse

Browse Source 
Pull fuse updates from Miklos Szeredi:

 - Improve performance of the io-uring transport by introducing buffer
   pools and zero-copy (Joanne)

 - Fix lots of bugs (Baokun Li)

 - Fix io-uring initialization issues (Joanne, Bernd)

 - More prep work for large folios (Joanne)

 - Don't limit buffered read to 128k (Jim Harris)

 - Fix zeroing of page end (dirtied with mmap) on file size extension
   (Jimmy Zuber)

 - Improve performance in certain cases with wake_up_sync() when queuing
   request (Xuewen Yan)

 - Misc fixes and cleanups (Xuewen Yan)

* tag 'fuse-update-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse: (35 commits)
  fuse: zero the partial EOF page when extending a file
  io_uring: Add missing include for ITER_SOURCE and ITER_DEST
  fuse: Fix the condition to enable over-io-uring
  fuse: invalidate the correct range after O_APPEND direct write
  selftests/fuse: test post-EOF page zeroing when a file is extended
  fuse: wake one waiter per freed slot when raising max_background
  fuse: use min_not_zero() in fuse_init_server_timeout()
  fuse: copy request headers via a stack buffer for io-uring
  fuse: give wakeup hints to the scheduler for synchronous requests
  fuse: check for NULL root inode in fuse_fill_super_submount
  fuse: reject a duplicate fd= mount option
  cuse: wait for pending RCU callbacks on module exit
  fuse: fix invalidate lock leak on open O_TRUNC DAX failure
  fuse: fix invalidate lock leak on setattr writeback failure
  fuse: wait for FR_FINISHED on abort_on_kill to prevent use-after-free
  fuse: make dentry_tree_work static
  docs: fuse: document io-uring buffer pool and zero-copy uapi
  fuse: add zero-copy over io-uring
  fuse: support registered buffer pools in io-uring
  fuse: add io-uring buffer pools
  ...
This commit is contained in:
Linus Torvalds
2026-08-25 07:59:44 -07:00
parent ce14fe4cd7 34b5c4a6e4
commit 9cebfe6504
27 changed files with 1474 additions and 236 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
Documentation/block/ublk.rst
View File

@@ -382,17 +382,17 @@ Zero copy
---------
ublk zero copy relies on io_uring's fixed kernel buffer, which provides
two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`.
two APIs: `io_buffer_register_request()` and `io_buffer_unregister`.
ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call
`io_buffer_register_bvec()` for ublk server to register client request
`io_buffer_register_request()` for ublk server to register client request
buffer into io_uring buffer table, then ublk server can submit io_uring
IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF`
calls `io_buffer_unregister_bvec()` to unregister the buffer, which is
guaranteed to be live between calling `io_buffer_register_bvec()` and
`io_buffer_unregister_bvec()`. Any io_uring operation which supports this
kind of kernel buffer will grab one reference of the buffer until the
operation is completed.
calls `io_buffer_unregister()` to unregister the buffer, which is guaranteed
to be live between calling `io_buffer_register_request()` and
`io_buffer_unregister()`. Any io_uring operation which supports this kind of
kernel buffer will grab one reference of the buffer until the operation is
completed.
ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and
be trusted, because it is ublk server's responsibility to make sure IO buffer

32
Documentation/filesystems/fuse/fuse-io-uring.rst
View File

@@ -11,6 +11,9 @@ and works. For generic details about FUSE see fuse.rst.
This document also covers the current interface, which is
still in development and might change.
For the userspace protocol, see
Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst.
Limitations
===========
As of now not all requests types are supported through io-uring, userspace
@@ -95,5 +98,34 @@ Sending requests with CQEs
| <fuse_unlink() |
| <sys_unlink() |
Buffer pools
============
Without a buffer pool, every entry needs to pass a dedicated payload buffer
large enough for the maximum payload size. A buffer pool decouples entries
from payload buffers. The server hands the kernel one contiguous buffer pool
of memory and when the kernel sends the server a request, it indicates the
offset into the pool for that request's payload. Internally, the kernel is
able to manage/optimize the buffer pool memory however it likes.
A server may also register the pool region with io_uring as a fixed buffer.
The backing pages are then pinned once, avoiding per-request pinning and
address translation. This also allows servers to use the same registered
buffers for subsequent backing store I/O through io-uring, keeping data
in the same pinned pages without additional pinning / mapping overhead.
Zero-copy
=========
Zero-copy lets the server read from / write to the client's pages (pinned
user pages for direct I/O, or page-cache folios for buffered I/O) without an
intermediary payload copy. This requires CAP_SYS_ADMIN privileges.
When a fuse request arrives for a file that opted into zero-copy, the kernel
registers the relevant pages (pinned user pages for direct i/o or underlying
page cache folios for buffered i/o) into a sparse slot in the server's
io_uring registered buffer table. The server can then operate on these pages
directly using io-uring fixed buffer operations (eg read_fixed / write_fixed)
and the kernel unregisters these pages when the request completes.
Non-page-backed args (eg op out headers) will go through the payload buffer as
normal.

1
Documentation/filesystems/fuse/index.rst
View File

@@ -12,3 +12,4 @@ FUSE (Filesystem in Userspace) Technical Documentation
fuse-io
fuse-io-uring
fuse-passthrough
uapi/fuse-uapi-io-uring

126
Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst Normal file
View File

@@ -0,0 +1,126 @@
.. SPDX-License-Identifier: GPL-2.0
=====================================
FUSE-over-io-uring uapi documentation
=====================================
Commands
========
``enum fuse_uring_cmd``:
``FUSE_IO_URING_CMD_ADD_QUEUE``
Create a queue identified by ``fuse_uring_cmd_req.qid``. Queue-wide
options are passed in ``fuse_uring_cmd_req.flags``:
``FUSE_URING_ZERO_COPY``
Enable zero-copy on this queue. Requires ``CAP_SYS_ADMIN`` and a buffer
pool, which is added separately via ``ADD_BUFPOOL`` before registering
entries (see `Zero-copy`_).
``FUSE_IO_URING_CMD_ADD_BUFPOOL``
Register the payload buffer pool for an existing queue. The server provides
a single contiguous region in ``fuse_uring_cmd_req.bufpool.uaddr`` /
``.len``. This command must be issued after ``ADD_QUEUE`` and before
registering any payload-carrying entries on that queue.
``fuse_uring_cmd_req.flags`` must be 0. Submitting this command with
``IORING_URING_CMD_FIXED`` marks the pool as registered, which avoids per
i/o pinning/unpinning and mapping overhead (see `Buffer pools`_).
``FUSE_IO_URING_CMD_REGISTER``
Register a ring entry (a long-lived SQE that carries the request header
iovec). For a zero-copy queue, ``fuse_uring_cmd_req.ent_zero_copy_buf_index``
indicates the reserved registered buffer table slot this entry uses for
zero-copy (see `Zero-copy`_).
``FUSE_IO_URING_CMD_COMMIT_AND_FETCH``
Commit the reply for a completed request and fetch the next one. The
request is identified by ``fuse_uring_cmd_req.commit_id`` (the value the
kernel reported in ``fuse_uring_ent_in_out.commit_id``).
Structures
==========
``struct fuse_uring_cmd_req`` (80-byte SQE command area):
============================ ==================================================
Field Meaning
============================ ==================================================
``flags`` Command-specific flags (see each command).
``commit_id`` Request id, for ``COMMIT_AND_FETCH``.
``qid`` Queue index.
``bufpool.uaddr`` Pool base address, for ``ADD_BUFPOOL``.
``bufpool.len`` Pool length in bytes, for ``ADD_BUFPOOL``.
``bufpool.reserved`` Must be 0, for ``ADD_BUFPOOL``.
``ent_zero_copy_buf_index`` Per-entry zero-copy slot, for ``REGISTER``.
============================ ==================================================
``struct fuse_uring_ent_in_out`` (reported by the kernel per request):
============================ ==================================================
Field Meaning
============================ ==================================================
``flags`` ``FUSE_URING_ENT_ZERO_COPY`` if zero-copied.
``commit_id`` Id to echo back in ``COMMIT_AND_FETCH``.
``payload_sz`` Total payload size in bytes (see `Zero-copy`_).
``offset`` Payload buffer offset within the pool.
============================ ==================================================
Buffer pools
============
Setup:
* Issue ``ADD_QUEUE`` for the qid.
* Issue ``ADD_BUFPOOL`` with ``bufpool.uaddr`` and ``bufpool.len`` pointing
at the region.
* Register entries with ``REGISTER``.
For every request that has a payload, the kernel reports where the payload
lives in ``struct fuse_uring_ent_in_out`` (part of
``struct fuse_uring_req_header``):
``offset``
Byte offset, within the pool region, for this request's payload buffer.
The server adds this to the pool base address to locate the payload.
``payload_sz``
Number of payload bytes for this request.
To use registered buffers, the server registers the pool region with io_uring
and submits ``ADD_BUFPOOL`` with ``IORING_URING_CMD_FIXED`` set in
``sqe->uring_cmd_flags`` and the index of the registered bufpool in
``sqe->buf_index``. Every SQE the server submits afterwards must follow the
same fixed-buffer protocol, carrying ``IORING_URING_CMD_FIXED`` and that same
``sqe->buf_index``. The same registered buffer can be reused for the server's
backing-store I/O as well (e.g. ``IORING_OP_READ_FIXED`` /
``IORING_OP_WRITE_FIXED``).
Zero-copy
=========
Requirements:
* The server must be privileged (``CAP_SYS_ADMIN``).
* A zero-copy queue: ``ADD_QUEUE`` with the ``FUSE_URING_ZERO_COPY`` flag set.
* A buffer pool: ``ADD_BUFPOOL``.
* For each entry, ``REGISTER`` with ``ent_zero_copy_buf_index`` set to the
index this entry uses in the server's io_uring registered-buffer table.
This is where the kernel registers the request's pages for the server to
access (it is separate from the payload pool). On a non-zero-copy queue this
field must be 0.
Zero-copy is selected per open file. The server sets the open-file flag in
the ``FUSE_OPEN`` / ``FUSE_CREATE`` reply:
``FOPEN_IO_URING_ZERO_COPY``
Reads/writes on this open file should use zero-copy.
For a request that is zero-copied, the kernel sets ``FUSE_URING_ENT_ZERO_COPY``
in ``fuse_uring_ent_in_out.flags`` and places the request's pages at the
entry's ``ent_zero_copy_buf_index``. The server then issues
``IORING_OP_READ_FIXED`` / ``IORING_OP_WRITE_FIXED`` against that index to
transfer the data directly to/from the client's pages.
For such a request, ``payload_sz`` includes the zero-copied page bytes
(transferred via the registered buffer at ``ent_zero_copy_buf_index``). Any
non-page-backed args (e.g. op headers) are still copied through the pool
payload buffer at ``offset``.

22
drivers/block/ublk_drv.c
View File

@@ -1699,8 +1699,8 @@ ublk_auto_buf_register(const struct ublk_queue *ubq, struct request *req,
{
int ret;
ret = io_buffer_register_bvec(cmd, req, ublk_io_release,
io->buf.auto_reg.index, issue_flags);
ret = io_buffer_register_request(cmd, req, ublk_io_release,
io->buf.auto_reg.index, issue_flags);
if (ret) {
if (io->buf.auto_reg.flags & UBLK_AUTO_BUF_REG_FALLBACK) {
ublk_auto_buf_reg_fallback(ubq, req->tag);
@@ -1909,7 +1909,7 @@ static noinline void ublk_batch_dispatch_fail(struct ublk_queue *ubq,
ublk_io_unlock(io);
if (index != -1)
io_buffer_unregister_bvec(data->cmd, index,
io_buffer_unregister(data->cmd, index,
data->issue_flags);
}
@@ -3192,8 +3192,8 @@ static int ublk_register_io_buf(struct io_uring_cmd *cmd,
if (!req)
return -EINVAL;
ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index,
issue_flags);
ret = io_buffer_register_request(cmd, req, ublk_io_release, index,
issue_flags);
if (ret) {
ublk_put_req_ref(io, req);
return ret;
@@ -3224,8 +3224,8 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd,
if (!ublk_dev_support_zero_copy(ub) || !blk_rq_has_data(req))
return -EINVAL;
ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index,
issue_flags);
ret = io_buffer_register_request(cmd, req, ublk_io_release, index,
issue_flags);
if (ret)
return ret;
@@ -3240,7 +3240,7 @@ static int ublk_unregister_io_buf(struct io_uring_cmd *cmd,
if (!(ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY))
return -EINVAL;
return io_buffer_unregister_bvec(cmd, index, issue_flags);
return io_buffer_unregister(cmd, index, issue_flags);
}
static int ublk_check_fetch_buf(const struct ublk_device *ub, __u64 buf_addr)
@@ -3384,7 +3384,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
goto out;
/*
* io_buffer_unregister_bvec() doesn't access the ubq or io,
* io_buffer_unregister() doesn't access the ubq or io,
* so no need to validate the q_id, tag, or task
*/
if (_IOC_NR(cmd_op) == UBLK_IO_UNREGISTER_IO_BUF)
@@ -3456,7 +3456,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
req = ublk_fill_io_cmd(io, cmd);
ublk_apply_io_buf(ub, io, cmd, addr, &auto_buf, &buf_idx);
if (buf_idx != UBLK_INVALID_BUF_IDX)
io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
io_buffer_unregister(cmd, buf_idx, issue_flags);
compl = ublk_need_complete_req(ub, io);
if (req_op(req) == REQ_OP_ZONE_APPEND)
@@ -3801,7 +3801,7 @@ static int ublk_batch_commit_io(struct ublk_queue *ubq,
}
if (buf_idx != UBLK_INVALID_BUF_IDX)
io_buffer_unregister_bvec(data->cmd, buf_idx, data->issue_flags);
io_buffer_unregister(data->cmd, buf_idx, data->issue_flags);
if (req_op(req) == REQ_OP_ZONE_APPEND)
req->__sector = ublk_batch_zone_lba(uc, elem);
if (compl)

2
fs/fuse/args.h
View File

@@ -42,6 +42,8 @@ struct fuse_args {
bool is_pinned:1;
bool invalidate_vmap:1;
bool abort_on_kill:1;
/* server requested io-uring zero-copy for this op */
bool zero_copy:1;
struct fuse_in_arg in_args[4];
struct fuse_arg out_args[2];
void (*end)(struct fuse_args *args, int error);

8
fs/fuse/cuse.c
View File

@@ -530,7 +530,8 @@ static int cuse_channel_open(struct inode *inode, struct file *file)
INIT_LIST_HEAD(&cc->list);
cc->fc.chan->initialized = 1;
/* Pairs with smp_load_acquire() readers of fch->initialized */
smp_store_release(&cc->fc.chan->initialized, 1);
rc = cuse_send_init(cc);
if (rc) {
fuse_dev_put(fud);
@@ -653,6 +654,11 @@ static void __exit cuse_exit(void)
{
misc_deregister(&cuse_miscdev);
class_destroy(cuse_class);
/*
* Wait for pending call_rcu() callbacks that call back into
* this module via fc->release (cuse_fc_release).
*/
rcu_barrier();
}
module_init(cuse_init);

104
fs/fuse/dev.c
View File

@@ -75,17 +75,23 @@ void fuse_chan_set_initialized(struct fuse_chan *fch, struct fuse_chan_param *pa
fch->minor = param->minor;
fch->max_write = param->max_write;
fch->max_pages = param->max_pages;
if (param->io_uring_enabled)
fuse_uring_conn_init(fch);
}
/* Make sure stores before this are seen on another CPU */
smp_wmb();
fch->initialized = 1;
/* Pairs with smp_load_acquire() readers of fch->initialized */
smp_store_release(&fch->initialized, 1);
wake_up_all(&fch->blocked_waitq);
}
static bool fuse_block_alloc(struct fuse_chan *fch, bool for_background)
{
return !fch->initialized || (for_background && fch->blocked) ||
/* Pairs with smp_store_release() in fuse_chan_set_initialized() */
if (!smp_load_acquire(&fch->initialized))
return true;
return (for_background && fch->blocked) ||
(fch->io_uring && fch->connected && !fuse_uring_ready(fch));
}
@@ -120,9 +126,6 @@ static struct fuse_req *fuse_get_req(struct fuse_chan *fch, bool for_background)
goto out;
}
/* Matches smp_wmb() in fuse_chan_set_initialized() */
smp_rmb();
err = -ENOTCONN;
if (!fch->connected)
goto out;
@@ -210,10 +213,13 @@ EXPORT_SYMBOL_GPL(fuse_req_hash);
/*
* A new request is available, wake fiq->waitq
*/
static void fuse_dev_wake_and_unlock(struct fuse_iqueue *fiq)
static void fuse_dev_wake_and_unlock(struct fuse_iqueue *fiq, bool sync)
__releases(fiq->lock)
{
wake_up(&fiq->waitq);
if (sync)
wake_up_sync(&fiq->waitq);
else
wake_up(&fiq->waitq);
kill_fasync(&fiq->fasync, SIGIO, POLL_IN);
spin_unlock(&fiq->lock);
}
@@ -230,7 +236,7 @@ void fuse_dev_queue_forget(struct fuse_iqueue *fiq,
if (fiq->connected) {
fiq->forget_list_tail->next = forget;
fiq->forget_list_tail = forget;
fuse_dev_wake_and_unlock(fiq);
fuse_dev_wake_and_unlock(fiq, false);
} else {
kfree(forget);
spin_unlock(&fiq->lock);
@@ -240,7 +246,8 @@ void fuse_dev_queue_forget(struct fuse_iqueue *fiq,
void fuse_dev_queue_interrupt(struct fuse_iqueue *fiq, struct fuse_req *req)
{
spin_lock(&fiq->lock);
if (list_empty(&req->intr_entry)) {
/* Repeat FR_SENT test after obtaining the lock to prevent race with fuse_resend() */
if (list_empty(&req->intr_entry) && test_bit(FR_SENT, &req->flags)) {
list_add_tail(&req->intr_entry, &fiq->interrupts);
/*
* Pairs with smp_mb() implied by test_and_set_bit()
@@ -251,7 +258,7 @@ void fuse_dev_queue_interrupt(struct fuse_iqueue *fiq, struct fuse_req *req)
list_del_init(&req->intr_entry);
spin_unlock(&fiq->lock);
} else {
fuse_dev_wake_and_unlock(fiq);
fuse_dev_wake_and_unlock(fiq, false);
}
} else {
spin_unlock(&fiq->lock);
@@ -281,11 +288,13 @@ EXPORT_SYMBOL_GPL(fuse_request_assign_unique);
static void fuse_dev_queue_req(struct fuse_iqueue *fiq, struct fuse_req *req)
{
bool sync = test_and_clear_bit(FR_SYNC_WAKEUP, &req->flags);
spin_lock(&fiq->lock);
if (fiq->connected) {
fuse_request_assign_unique_locked(fiq, req);
list_add_tail(&req->list, &fiq->pending);
fuse_dev_wake_and_unlock(fiq);
fuse_dev_wake_and_unlock(fiq, sync);
} else {
spin_unlock(&fiq->lock);
req->out.h.error = -ENOTCONN;
@@ -397,7 +406,8 @@ void fuse_chan_max_background_set(struct fuse_chan *fch, unsigned int val)
fch->max_background = val;
fch->blocked = fch->num_background >= fch->max_background;
if (!fch->blocked)
wake_up(&fch->blocked_waitq);
wake_up_nr(&fch->blocked_waitq,
fch->max_background - fch->num_background);
spin_unlock(&fch->bg_lock);
}
@@ -411,11 +421,6 @@ void fuse_chan_set_fc(struct fuse_chan *fch, struct fuse_conn *fc)
fch->conn = fc;
}
void fuse_chan_io_uring_enable(struct fuse_chan *fch)
{
fch->io_uring = 1;
}
void fuse_pqueue_init(struct fuse_pqueue *fpq)
{
spin_lock_init(&fpq->lock);
@@ -725,7 +730,7 @@ static void request_wait_answer(struct fuse_req *req)
if (req->args->abort_on_kill) {
fuse_chan_abort(fch, false);
return;
goto wait_for_finish;
}
if (test_bit(FR_URING, &req->flags))
@@ -736,6 +741,7 @@ static void request_wait_answer(struct fuse_req *req)
return;
}
wait_for_finish:
/*
* Either request is already in userspace, or it was forced.
* Wait it out.
@@ -752,6 +758,11 @@ static void __fuse_request_send(struct fuse_req *req)
/* acquire extra reference, since request is still needed after
fuse_request_end() */
__fuse_get_request(req);
/*
* This is a synchronous request: the caller will block waiting for
* the answer. Hint the scheduler via wake_up_sync().
*/
set_bit(FR_SYNC_WAKEUP, &req->flags);
fuse_send_one(fiq, req);
request_wait_answer(req);
@@ -1249,11 +1260,25 @@ int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
if (folio) {
size = folio_size(folio);
if (zeroing && count < size)
folio_zero_range(folio, 0, size);
if (zeroing && count < size) {
/*
* When the copy is skipped the folio already holds the
* payload, so only the bytes outside [offset, offset +
* count) may be zeroed.
*
* Otherwise, the whole folio is cleared first so that a
* failed copy leaves zeros rather than stale folio
* contents.
*/
if (cs->skip_folio_copy)
folio_zero_segments(folio, 0, offset,
offset + count, size);
else
folio_zero_range(folio, 0, size);
}
}
while (count) {
while (!cs->skip_folio_copy && count) {
if (cs->write && cs->pipebufs && folio) {
/*
* Can't control lifetime of pipe buffers, so always
@@ -1346,6 +1371,10 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs,
for (i = 0; !err && i < numargs; i++) {
struct fuse_arg *arg = &args[i];
if (i == numargs - 1 && argpages)
/*
* if cs->skip_folio_copy is set, this just does any
* needed zeroing. No copying is involved.
*/
err = fuse_copy_folios(cs, arg->size, zeroing);
else
err = fuse_copy_one(cs, arg->value, arg->size);
@@ -1760,7 +1789,7 @@ static ssize_t fuse_dev_splice_read(struct file *in, loff_t *ppos,
void fuse_chan_resend(struct fuse_chan *fch)
{
struct fuse_dev *fud;
struct fuse_req *req, *next;
struct fuse_req *req;
struct fuse_iqueue *fiq = &fch->iq;
LIST_HEAD(to_queue);
unsigned int i;
@@ -1775,24 +1804,20 @@ void fuse_chan_resend(struct fuse_chan *fch)
struct fuse_pqueue *fpq = &fud->pq;
spin_lock(&fpq->lock);
for (i = 0; i < FUSE_PQ_HASH_SIZE; i++)
list_splice_tail_init(&fpq->processing[i], &to_queue);
for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) {
struct list_head *this_queue = &fpq->processing[i];
list_for_each_entry(req, this_queue, list)
clear_bit(FR_SENT, &req->flags);
list_splice_tail_init(this_queue, &to_queue);
}
spin_unlock(&fpq->lock);
}
spin_unlock(&fch->lock);
list_for_each_entry_safe(req, next, &to_queue, list) {
set_bit(FR_PENDING, &req->flags);
clear_bit(FR_SENT, &req->flags);
/* mark the request as resend request */
req->in.h.unique |= FUSE_UNIQUE_RESEND;
}
spin_lock(&fiq->lock);
if (!fiq->connected) {
spin_unlock(&fiq->lock);
list_for_each_entry(req, &to_queue, list)
clear_bit(FR_PENDING, &req->flags);
fuse_dev_end_requests(&to_queue);
return;
}
@@ -1801,12 +1826,16 @@ void fuse_chan_resend(struct fuse_chan *fch)
* intr_entry on fiq->interrupts after the request is re-queued.
*/
list_for_each_entry(req, &to_queue, list) {
set_bit(FR_PENDING, &req->flags);
/* mark the request as resend request */
req->in.h.unique |= FUSE_UNIQUE_RESEND;
if (test_bit(FR_INTERRUPTED, &req->flags))
list_del_init(&req->intr_entry);
}
/* iq and pq requests are both oldest to newest */
list_splice(&to_queue, &fiq->pending);
fuse_dev_wake_and_unlock(fiq);
fuse_dev_wake_and_unlock(fiq, false);
}
/* Look up request on processing list by unique ID */
@@ -1888,7 +1917,8 @@ static ssize_t fuse_dev_do_write(struct fuse_dev *fud,
* initialized and connected state
*/
err = -EINVAL;
if (!fch->initialized || !fch->connected)
/* Pairs with smp_store_release() in fuse_chan_set_initialized() */
if (!smp_load_acquire(&fch->initialized) || !fch->connected)
goto copy_finish;
/* Don't try to move folios (yet) */

2
fs/fuse/dev.h
View File

@@ -22,6 +22,7 @@ struct fuse_chan_param {
unsigned int minor;
unsigned int max_write;
unsigned int max_pages;
bool io_uring_enabled;
};
struct fuse_chan *fuse_chan_new(void);
@@ -34,7 +35,6 @@ void fuse_chan_max_background_set(struct fuse_chan *fch, unsigned int val);
unsigned int fuse_chan_num_waiting(struct fuse_chan *fch);
void fuse_chan_set_fc(struct fuse_chan *fch, struct fuse_conn *fc);
void fuse_chan_set_initialized(struct fuse_chan *fch, struct fuse_chan_param *param);
void fuse_chan_io_uring_enable(struct fuse_chan *fch);
ssize_t fuse_chan_send(struct fuse_chan *fch, struct fuse_args *args);
int fuse_chan_send_bg(struct fuse_chan *fch, struct fuse_args *args, gfp_t gfp_flags);
int fuse_chan_send_notify_reply(struct fuse_chan *fch, struct fuse_args *args, u64 unique);

633
fs/fuse/dev_uring.c
View File

@@ -9,6 +9,7 @@
#include "dev_uring_i.h"
#include "fuse_trace.h"
#include <linux/bitmap.h>
#include <linux/fs.h>
#include <linux/io_uring/cmd.h>
@@ -21,6 +22,8 @@ MODULE_PARM_DESC(enable_uring,
#define FUSE_URING_IOV_HEADERS 0
#define FUSE_URING_IOV_PAYLOAD 1
#define FUSE_URING_ADD_QUEUE_FLAGS (FUSE_URING_ZERO_COPY)
bool fuse_uring_enabled(void)
{
return enable_uring;
@@ -30,6 +33,11 @@ struct fuse_uring_pdu {
struct fuse_ring_ent *ent;
};
struct fuse_zero_copy_bvs {
unsigned int nr_bvs;
struct bio_vec bvs[];
};
static const struct fuse_iqueue_ops fuse_io_uring_ops;
enum fuse_uring_header_type {
@@ -41,6 +49,32 @@ enum fuse_uring_header_type {
FUSE_URING_HEADER_RING_ENT,
};
static inline bool bufpool_enabled(struct fuse_ring_queue *queue)
{
return queue->payload_mode == FUSE_PAYLOAD_BUFPOOL;
}
static inline bool bufpool_registered(struct fuse_ring_queue *queue)
{
return queue->bufpool && queue->bufpool->registered;
}
/*
* For a registered bufpool, every sqe that drives a payload import (REGISTER,
* COMMIT_AND_FETCH) must carry the registered buffer index of the pool.
* This also must be called from the command's issue handler, where cmd->sqe is
* still valid
*/
static inline bool fuse_uring_cmd_index_ok(struct io_uring_cmd *cmd,
struct fuse_ring_queue *queue)
{
if (!bufpool_registered(queue))
return true;
return (cmd->flags & IORING_URING_CMD_FIXED) &&
READ_ONCE(cmd->sqe->buf_index) == queue->bufpool->registered_index;
}
static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd,
struct fuse_ring_ent *ring_ent)
{
@@ -86,8 +120,36 @@ static void fuse_uring_flush_bg(struct fuse_ring_queue *queue)
}
}
static bool can_zero_copy_req(struct fuse_ring_ent *ent, struct fuse_req *req)
{
struct fuse_args *args = req->args;
if (!ent->queue->zero_copy || !args->zero_copy)
return false;
if (args->opcode != FUSE_READ && args->opcode != FUSE_WRITE)
return false;
return args->in_pages || args->out_pages;
}
static void zero_copy_unregister(struct io_uring_cmd *cmd,
struct fuse_ring_ent *ent,
unsigned int issue_flags)
{
if (ent->zero_copied) {
int err = io_buffer_unregister(cmd, ent->zero_copy_index,
issue_flags);
if (err)
pr_warn_ratelimited("qid=%d zero-copy unregister failed: %d\n",
ent->queue->qid, err);
ent->zero_copied = false;
}
}
static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req,
int error)
int error, unsigned int issue_flags)
{
struct fuse_ring_queue *queue = ent->queue;
struct fuse_ring *ring = queue->ring;
@@ -107,6 +169,8 @@ static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req,
spin_unlock(&queue->lock);
zero_copy_unregister(ent->cmd, ent, issue_flags);
if (error)
req->out.h.error = error;
@@ -204,7 +268,7 @@ void fuse_uring_destruct(struct fuse_chan *fch)
return;
for (qid = 0; qid < ring->nr_queues; qid++) {
struct fuse_ring_queue *queue = ring->queues[qid];
struct fuse_ring_queue *queue = READ_ONCE(ring->queues[qid]);
struct fuse_ring_ent *ent, *next;
if (!queue)
@@ -222,8 +286,9 @@ void fuse_uring_destruct(struct fuse_chan *fch)
}
kfree(queue->fpq.processing);
kfree(queue->bufpool);
kfree(queue);
ring->queues[qid] = NULL;
WRITE_ONCE(ring->queues[qid], NULL);
}
kfree(ring->queues);
@@ -238,7 +303,6 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch)
{
struct fuse_ring *ring;
size_t nr_queues = num_possible_cpus();
struct fuse_ring *res = NULL;
size_t max_payload_size;
ring = kzalloc_obj(*ring, GFP_KERNEL_ACCOUNT);
@@ -258,12 +322,6 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch)
spin_unlock(&fch->lock);
goto out_err;
}
if (fch->ring) {
/* race, another thread created the ring in the meantime */
spin_unlock(&fch->lock);
res = fch->ring;
goto out_err;
}
init_waitqueue_head(&ring->stop_waitq);
@@ -278,11 +336,18 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch)
out_err:
kfree(ring->queues);
kfree(ring);
return res;
return NULL;
}
void fuse_uring_conn_init(struct fuse_chan *fch)
{
if (fuse_uring_create(fch))
fch->io_uring = 1;
}
static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
int qid)
int qid, bool zero_copy,
bool fail_if_exists)
{
struct fuse_chan *fch = ring->chan;
struct fuse_ring_queue *queue;
@@ -290,16 +355,17 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
queue = kzalloc_obj(*queue, GFP_KERNEL_ACCOUNT);
if (!queue)
return NULL;
return ERR_PTR(-ENOMEM);
pq = fuse_pqueue_alloc();
if (!pq) {
kfree(queue);
return NULL;
return ERR_PTR(-ENOMEM);
}
queue->qid = qid;
queue->ring = ring;
spin_lock_init(&queue->lock);
queue->zero_copy = zero_copy;
INIT_LIST_HEAD(&queue->ent_avail_queue);
INIT_LIST_HEAD(&queue->ent_commit_queue);
@@ -316,14 +382,17 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
if (ring->queues[qid]) {
spin_unlock(&fch->lock);
kfree(queue->fpq.processing);
kfree(queue->bufpool);
kfree(queue);
return ring->queues[qid];
return fail_if_exists ? ERR_PTR(-EEXIST) : ring->queues[qid];
}
/*
* write_once and lock as the caller mostly doesn't take the lock at all
* fch->lock serializes concurrent creators for this qid.
* smp_store_release() are for the lockless readers who must see a
* fully initialized queue after &ring->queues[qid] is set
*/
WRITE_ONCE(ring->queues[qid], queue);
smp_store_release(&ring->queues[qid], queue);
spin_unlock(&fch->lock);
return queue;
@@ -434,7 +503,7 @@ static void fuse_uring_log_ent_state(struct fuse_ring *ring)
struct fuse_ring_ent *ent;
for (qid = 0; qid < ring->nr_queues; qid++) {
struct fuse_ring_queue *queue = ring->queues[qid];
struct fuse_ring_queue *queue = READ_ONCE(ring->queues[qid]);
if (!queue)
continue;
@@ -643,30 +712,57 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
return 0;
}
static int fuse_uring_import_payload(struct fuse_ring_ent *ent, int dir,
struct iov_iter *iter,
unsigned int issue_flags)
{
void __user *base = ent->payload.iov_base;
size_t len = ent->payload.iov_len;
int err = 0;
if (!base) {
memset(iter, 0, sizeof(*iter));
return 0;
}
if (bufpool_registered(ent->queue))
err = io_uring_cmd_import_fixed((u64)(uintptr_t)base, len, dir,
iter, ent->cmd, issue_flags);
else
err = import_ubuf(dir, base, len, iter);
if (err)
pr_info_ratelimited("fuse: Import of user buffer failed\n");
return err;
}
static int setup_fuse_copy_state(struct fuse_copy_state *cs,
struct fuse_ring *ring, struct fuse_req *req,
struct fuse_req *req,
struct fuse_ring_ent *ent, int dir,
struct iov_iter *iter)
struct iov_iter *iter,
unsigned int issue_flags)
{
int err;
err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);
if (err) {
pr_info_ratelimited("fuse: Import of user buffer failed\n");
err = fuse_uring_import_payload(ent, dir, iter, issue_flags);
if (err)
return err;
}
fuse_copy_init(cs, dir == ITER_DEST, iter);
if (ent->zero_copied)
cs->skip_folio_copy = true;
cs->is_uring = true;
cs->req = req;
return 0;
}
static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
struct fuse_req *req,
struct fuse_ring_ent *ent)
static int fuse_uring_copy_from_ring(struct fuse_req *req,
struct fuse_ring_ent *ent,
unsigned int issue_flags)
{
struct fuse_copy_state cs;
struct fuse_args *args = req->args;
@@ -679,7 +775,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
if (err)
return err;
err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_SOURCE, &iter);
err = setup_fuse_copy_state(&cs, req, ent, ITER_SOURCE, &iter,
issue_flags);
if (err)
return err;
@@ -688,11 +785,68 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
return err;
}
static void fuse_zero_copy_release(void *priv)
{
struct fuse_zero_copy_bvs *zc_bvs = priv;
unsigned int i;
for (i = 0; i < zc_bvs->nr_bvs; i++)
folio_put(page_folio(zc_bvs->bvs[i].bv_page));
kvfree(zc_bvs);
}
static int fuse_uring_set_up_zero_copy(struct fuse_ring_ent *ent,
struct fuse_req *req,
unsigned int issue_flags)
{
struct fuse_args_pages *ap;
int err, i, ddir = 0;
struct fuse_zero_copy_bvs *zc_bvs;
struct bio_vec *bvs;
/* out_pages indicates a read, in_pages indicates a write */
if (req->args->out_pages)
ddir |= IO_BUF_DEST;
if (req->args->in_pages)
ddir |= IO_BUF_SOURCE;
ap = container_of(req->args, typeof(*ap), args);
zc_bvs = kvmalloc_flex(*zc_bvs, bvs, ap->num_folios,
GFP_KERNEL_ACCOUNT);
if (!zc_bvs)
return -ENOMEM;
zc_bvs->nr_bvs = ap->num_folios;
bvs = zc_bvs->bvs;
for (i = 0; i < ap->num_folios; i++) {
bvs[i].bv_page = folio_page(ap->folios[i], 0);
bvs[i].bv_offset = ap->descs[i].offset;
bvs[i].bv_len = ap->descs[i].length;
folio_get(ap->folios[i]);
}
err = io_buffer_register_bvec(ent->cmd, bvs, ap->num_folios,
fuse_zero_copy_release, zc_bvs,
ddir, ent->zero_copy_index,
issue_flags);
if (err) {
fuse_zero_copy_release(zc_bvs);
return err;
}
ent->zero_copied = true;
return 0;
}
/*
* Copy data from the req to the ring buffer
*/
static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
struct fuse_ring_ent *ent)
static int fuse_uring_args_to_ring(struct fuse_req *req,
struct fuse_ring_ent *ent,
unsigned int issue_flags)
{
struct fuse_copy_state cs;
struct fuse_args *args = req->args;
@@ -705,7 +859,15 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
.commit_id = req->in.h.unique,
};
err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_DEST, &iter);
if (can_zero_copy_req(ent, req)) {
ent_in_out.flags |= FUSE_URING_ENT_ZERO_COPY;
err = fuse_uring_set_up_zero_copy(ent, req, issue_flags);
if (err)
return err;
}
err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter,
issue_flags);
if (err)
return err;
@@ -735,15 +897,32 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
}
ent_in_out.payload_sz = cs.ring.copied_sz;
/*
* on a zero-copied write the pages are registered for the server to
* read via a fixed-buffer op rather than copied into the payload
* buffer, so copied_sz does not account for it. The server still needs
* the total inbound size to know how many bytes to read from the
* registered buffer, so add the page arg (always the last in-arg) back
* in
*/
if (cs.skip_folio_copy && args->in_pages)
ent_in_out.payload_sz +=
args->in_args[args->in_numargs - 1].size;
if (bufpool_enabled(ent->queue) && ent->payload.iov_base)
ent_in_out.offset =
(uintptr_t)ent->payload.iov_base - ent->queue->bufpool->base_uaddr;
return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
&ent_in_out, sizeof(ent_in_out));
}
static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
struct fuse_req *req)
struct fuse_req *req,
unsigned int issue_flags)
{
struct fuse_ring_queue *queue = ent->queue;
struct fuse_ring *ring = queue->ring;
struct fuse_in_header in_header;
int err;
err = -EIO;
@@ -758,23 +937,124 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
return err;
/* copy the request */
err = fuse_uring_args_to_ring(ring, req, ent);
err = fuse_uring_args_to_ring(req, ent, issue_flags);
if (unlikely(err)) {
pr_info_ratelimited("Copy to ring failed: %d\n", err);
return err;
}
/* copy fuse_in_header */
return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->in.h,
sizeof(req->in.h));
in_header = req->in.h;
return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &in_header,
sizeof(in_header));
}
static bool fuse_uring_req_has_copyable_payload(struct fuse_ring_ent *ent,
struct fuse_req *req)
{
struct fuse_args *args = req->args;
if (!can_zero_copy_req(ent, req))
return args->in_numargs > 1 || args->out_numargs;
/*
* the asymmetry between in_numargs > 2 and out_numargs > 1 is because
* the per-op header is extracted before fuse_copy_args() for inargs but
* not for outargs
*/
if ((args->in_numargs > 1) && (!args->in_pages || args->in_numargs > 2))
return true;
if (args->out_numargs && (!args->out_pages || args->out_numargs > 1))
return true;
return false;
}
static int fuse_uring_select_buffer(struct fuse_ring_ent *ent)
{
struct fuse_ring_queue *queue = ent->queue;
struct fuse_bufpool *pool = queue->bufpool;
unsigned int id;
lockdep_assert_held(&queue->lock);
id = find_first_bit(pool->free_map, pool->nr_bufs);
if (id >= pool->nr_bufs)
return -ENOBUFS;
WARN_ON_ONCE(ent->payload.iov_base);
__clear_bit(id, pool->free_map);
ent->buf_id = id;
ent->payload.iov_base =
(void __user *)(pool->base_uaddr + id * pool->buf_size);
ent->payload.iov_len = pool->buf_size;
return 0;
}
static void fuse_uring_recycle_buffer(struct fuse_ring_ent *ent)
{
struct iovec *ent_payload = &ent->payload;
struct fuse_ring_queue *queue = ent->queue;
struct fuse_bufpool *pool;
lockdep_assert_held(&queue->lock);
if (!bufpool_enabled(queue) || !ent_payload->iov_base)
return;
pool = queue->bufpool;
/* a buffer should never be recycled twice */
WARN_ON_ONCE(test_bit(ent->buf_id, pool->free_map));
__set_bit(ent->buf_id, pool->free_map);
memset(ent_payload, 0, sizeof(*ent_payload));
ent->buf_id = 0;
}
static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent,
struct fuse_req *req)
{
bool buffer_selected;
bool has_payload;
if (!bufpool_enabled(ent->queue))
return 0;
buffer_selected = !!ent->payload.iov_base;
has_payload = fuse_uring_req_has_copyable_payload(ent, req);
if (has_payload && !buffer_selected)
return fuse_uring_select_buffer(ent);
if (!has_payload && buffer_selected)
fuse_uring_recycle_buffer(ent);
return 0;
}
static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
struct fuse_req *req)
{
if (!bufpool_enabled(ent->queue))
return 0;
/* no payload to copy, can skip selecting a buffer */
if (!fuse_uring_req_has_copyable_payload(ent, req))
return 0;
return fuse_uring_select_buffer(ent);
}
static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
struct fuse_req *req)
struct fuse_req *req,
unsigned int issue_flags)
{
int err;
err = fuse_uring_copy_to_ring(ent, req);
err = fuse_uring_copy_to_ring(ent, req, issue_flags);
if (!err) {
set_bit(FR_SENT, &req->flags);
trace_fuse_request_sent(req);
@@ -788,7 +1068,7 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
ent->state = FRRS_INVALID;
spin_unlock(&ent->queue->lock);
fuse_uring_req_end(ent, req, err);
fuse_uring_req_end(ent, req, err, issue_flags);
}
return err;
@@ -856,9 +1136,12 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)
/* get and assign the next entry while it is still holding the lock */
req = list_first_entry_or_null(req_queue, struct fuse_req, list);
if (req)
fuse_uring_add_req_to_ring_ent(ent, req);
if (!req || fuse_uring_next_req_update_buffer(ent, req)) {
fuse_uring_recycle_buffer(ent);
return NULL;
}
fuse_uring_add_req_to_ring_ent(ent, req);
return req;
}
@@ -870,12 +1153,13 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)
static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
unsigned int issue_flags)
{
struct fuse_ring *ring = ent->queue->ring;
struct fuse_out_header out_header;
ssize_t err = -EFAULT;
if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h,
sizeof(req->out.h)))
if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &out_header,
sizeof(out_header)))
goto out;
req->out.h = out_header;
err = fuse_uring_out_header_has_err(&req->out.h, req);
if (err) {
@@ -883,9 +1167,9 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
goto out;
}
err = fuse_uring_copy_from_ring(ring, req, ent);
err = fuse_uring_copy_from_ring(req, ent, issue_flags);
out:
fuse_uring_req_end(ent, req, err);
fuse_uring_req_end(ent, req, err, issue_flags);
}
/*
@@ -895,7 +1179,8 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
* Else, there is no next fuse request and this returns false.
*/
static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
struct fuse_ring_queue *queue)
struct fuse_ring_queue *queue,
unsigned int issue_flags)
{
int err;
struct fuse_req *req;
@@ -907,7 +1192,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
spin_unlock(&queue->lock);
if (req) {
err = fuse_uring_prepare_send(ent, req);
err = fuse_uring_prepare_send(ent, req, issue_flags);
if (err)
goto retry;
}
@@ -967,7 +1252,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
if (qid >= ring->nr_queues)
return -EINVAL;
queue = ring->queues[qid];
queue = READ_ONCE(ring->queues[qid]);
if (!queue)
return err;
fpq = &queue->fpq;
@@ -981,6 +1266,11 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
return err;
}
if (!fuse_uring_cmd_index_ok(cmd, queue)) {
spin_unlock(&queue->lock);
return -EINVAL;
}
/* Find a request based on the unique ID of the fuse request
* This should get revised, as it needs a hash calculation and list
* search. And full struct fuse_pqueue is needed (memory overhead).
@@ -1002,8 +1292,14 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
if (err != 0) {
pr_info_ratelimited("qid=%d commit_id %llu state %d",
queue->qid, commit_id, ent->state);
fuse_uring_recycle_buffer(ent);
spin_unlock(&queue->lock);
fuse_uring_req_end(ent, req, err);
/*
* Unregister any zero copyable pages since ent->cmd is null
* when it hits fuse_uring_req_end() in this path
*/
zero_copy_unregister(cmd, ent, issue_flags);
fuse_uring_req_end(ent, req, err, issue_flags);
return err;
}
@@ -1019,8 +1315,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
* fuse requests would otherwise not get processed - committing
* and fetching is done in one step vs legacy fuse, which has separated
* read (fetch request) and write (commit result).
*
* If there is no next request or if all buffers are busy (if using a
* bufpool), the cmd is not returned to userspace. The entry is left
* available and the cmd only returns to userspace when there's a
* next request and an available buffer.
*/
if (fuse_uring_get_next_fuse_req(ent, queue))
if (fuse_uring_get_next_fuse_req(ent, queue, issue_flags))
fuse_uring_send(ent, cmd, 0, issue_flags);
return 0;
}
@@ -1035,7 +1336,7 @@ static bool is_ring_ready(struct fuse_ring *ring, int current_qid)
if (current_qid == qid)
continue;
queue = ring->queues[qid];
queue = READ_ONCE(ring->queues[qid]);
if (!queue) {
ready = false;
break;
@@ -1122,10 +1423,14 @@ static struct fuse_ring_ent *
fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
struct fuse_ring_queue *queue)
{
const struct fuse_uring_cmd_req *cmd_req =
io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
struct fuse_ring *ring = queue->ring;
struct fuse_ring_ent *ent;
struct iovec iov[FUSE_URING_IOV_SEGS];
struct iovec *headers, *payload;
unsigned int zero_copy_index;
int err;
err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
@@ -1135,6 +1440,10 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
return ERR_PTR(err);
}
zero_copy_index = READ_ONCE(cmd_req->ent_zero_copy_buf_index);
if (zero_copy_index && !queue->zero_copy)
return ERR_PTR(-EINVAL);
err = -EINVAL;
headers = &iov[FUSE_URING_IOV_HEADERS];
if (headers->iov_len < sizeof(struct fuse_uring_req_header)) {
@@ -1143,11 +1452,29 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
}
payload = &iov[FUSE_URING_IOV_PAYLOAD];
if (payload->iov_len < ring->max_payload_sz) {
pr_info_ratelimited("Invalid req payload len %zu\n",
payload->iov_len);
return ERR_PTR(err);
spin_lock(&queue->lock);
if (bufpool_enabled(queue)) {
if (payload->iov_base || payload->iov_len ||
!fuse_uring_cmd_index_ok(cmd, queue)) {
spin_unlock(&queue->lock);
return ERR_PTR(err);
}
} else {
if (payload->iov_len < ring->max_payload_sz) {
spin_unlock(&queue->lock);
pr_info_ratelimited("Invalid req payload len %zu\n",
payload->iov_len);
return ERR_PTR(err);
}
if (queue->zero_copy) {
spin_unlock(&queue->lock);
pr_info_ratelimited("Can only use zero copy with bufpools\n");
return ERR_PTR(err);
}
queue->payload_mode = FUSE_PAYLOAD_PER_ENT;
}
spin_unlock(&queue->lock);
err = -ENOMEM;
ent = kzalloc_obj(*ent, GFP_KERNEL_ACCOUNT);
@@ -1158,7 +1485,9 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
ent->queue = queue;
ent->headers = headers->iov_base;
ent->payload = payload->iov_base;
if (queue->payload_mode == FUSE_PAYLOAD_PER_ENT)
ent->payload = *payload;
ent->zero_copy_index = zero_copy_index;
atomic_inc(&ring->queue_refs);
return ent;
@@ -1176,26 +1505,21 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
struct fuse_ring *ring = smp_load_acquire(&fch->ring);
struct fuse_ring_queue *queue;
struct fuse_ring_ent *ent;
int err;
unsigned int qid = READ_ONCE(cmd_req->qid);
err = -ENOMEM;
if (!ring) {
ring = fuse_uring_create(fch);
if (!ring)
return err;
}
if (!ring)
return -EINVAL;
if (qid >= ring->nr_queues) {
pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid);
return -EINVAL;
}
queue = ring->queues[qid];
queue = READ_ONCE(ring->queues[qid]);
if (!queue) {
queue = fuse_uring_create_queue(ring, qid);
if (!queue)
return err;
queue = fuse_uring_create_queue(ring, qid, false, false);
if (IS_ERR(queue))
return PTR_ERR(queue);
}
/*
@@ -1210,6 +1534,110 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
return fuse_uring_do_register(ent, cmd, issue_flags);
}
static int fuse_uring_add_queue(struct io_uring_cmd *cmd, struct fuse_chan *fch)
{
const struct fuse_uring_cmd_req *cmd_req =
io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
struct fuse_ring *ring = smp_load_acquire(&fch->ring);
unsigned int qid = READ_ONCE(cmd_req->qid);
uint64_t flags = READ_ONCE(cmd_req->flags);
struct fuse_ring_queue *queue;
bool zero_copy = flags & FUSE_URING_ZERO_COPY;
if (!ring)
return -EINVAL;
if (qid >= ring->nr_queues) {
pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid);
return -EINVAL;
}
if (flags & ~FUSE_URING_ADD_QUEUE_FLAGS)
return -EINVAL;
if (zero_copy && !capable(CAP_SYS_ADMIN))
return -EPERM;
queue = fuse_uring_create_queue(ring, qid, zero_copy, true);
if (IS_ERR(queue))
return PTR_ERR(queue);
return 0;
}
static int fuse_uring_add_bufpool(struct io_uring_cmd *cmd,
struct fuse_chan *fch)
{
const struct fuse_uring_cmd_req *cmd_req =
io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
unsigned int qid = READ_ONCE(cmd_req->qid);
uint64_t flags = READ_ONCE(cmd_req->flags);
/* paired with the smp_store_release() in fuse_uring_create */
struct fuse_ring *ring = smp_load_acquire(&fch->ring);
struct fuse_ring_queue *queue;
struct fuse_bufpool *pool;
uintptr_t pool_uaddr;
unsigned int pool_len, nr_bufs;
size_t pool_size, buf_size;
bool registered = cmd->flags & IORING_URING_CMD_FIXED;
if (!ring || qid >= ring->nr_queues || flags)
return -EINVAL;
/* reserved for future use, must be zero */
if (READ_ONCE(cmd_req->bufpool.reserved))
return -EINVAL;
/* Pairs with smp_store_release() in fuse_uring_create_queue() */
queue = smp_load_acquire(&ring->queues[qid]);
if (!queue)
return -EINVAL;
pool_uaddr = READ_ONCE(cmd_req->bufpool.uaddr);
pool_len = READ_ONCE(cmd_req->bufpool.len);
/* each buffer holds the max payload size */
buf_size = queue->ring->max_payload_sz;
nr_bufs = pool_len / buf_size;
if (!nr_bufs)
return -EINVAL;
pool_size = struct_size(pool, free_map, BITS_TO_LONGS(nr_bufs));
pool = kzalloc(pool_size, GFP_KERNEL_ACCOUNT);
if (!pool)
return -ENOMEM;
pool->base_uaddr = pool_uaddr;
pool->buf_size = buf_size;
pool->nr_bufs = nr_bufs;
/* all buffers are free */
bitmap_set(pool->free_map, 0, nr_bufs);
/*
* A registered bufpool is reached through an io_uring fixed buffer, so
* the pool is registered iff this command was submitted with
* IORING_URING_CMD_FIXED. The registered buffer index is taken from
* sqe->buf_index.
*/
if (registered) {
pool->registered = true;
pool->registered_index = READ_ONCE(cmd->sqe->buf_index);
}
spin_lock(&queue->lock);
if (queue->payload_mode != FUSE_PAYLOAD_UNSET) {
spin_unlock(&queue->lock);
kfree(pool);
return -EINVAL;
}
queue->bufpool = pool;
queue->payload_mode = FUSE_PAYLOAD_BUFPOOL;
spin_unlock(&queue->lock);
return 0;
}
/*
* Entry function from io_uring to handle the given passthrough command
* (op code IORING_OP_URING_CMD)
@@ -1237,23 +1665,30 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)
}
fch = fud->chan;
/* Once a connection has io-uring enabled on it, it can't be disabled */
if (!enable_uring && !fch->io_uring) {
pr_info_ratelimited("fuse-io-uring is disabled\n");
return -EOPNOTSUPP;
}
/*
* The ring is sized from values negotiated by FUSE_INIT
*
* Pairs with smp_store_release() in fuse_chan_set_initialized()
*/
if (!smp_load_acquire(&fch->initialized))
return -EAGAIN;
if (fch->abort_with_err)
return -ECONNABORTED;
if (!fch->connected)
return -ENOTCONN;
/*
* fuse_uring_register() needs the ring to be initialized,
* we need to know the max payload size
*/
if (!fch->initialized)
return -EAGAIN;
/* Once a connection has io-uring enabled on it, it can't be disabled */
if (!enable_uring && !fch->io_uring) {
pr_info_ratelimited("fuse-io-uring is disabled by module parameter\n");
return -EOPNOTSUPP;
}
if (!fch->io_uring) {
pr_info_ratelimited(
"fuse-io-uring not enabled on this connection\n");
return -EOPNOTSUPP;
}
switch (cmd_op) {
case FUSE_IO_URING_CMD_REGISTER:
@@ -1274,6 +1709,18 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)
return err;
}
break;
case FUSE_IO_URING_CMD_ADD_QUEUE:
err = fuse_uring_add_queue(cmd, fch);
if (err)
pr_info_once("FUSE_IO_URING_CMD_ADD_QUEUE failed err=%d\n",
err);
return err;
case FUSE_IO_URING_CMD_ADD_BUFPOOL:
err = fuse_uring_add_bufpool(cmd, fch);
if (err)
pr_info_once("FUSE_IO_URING_ADD_BUFPOOL failed err=%d\n",
err);
return err;
default:
return -EINVAL;
}
@@ -1295,9 +1742,10 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
int err;
if (!tw.cancel) {
err = fuse_uring_prepare_send(ent, ent->fuse_req);
err = fuse_uring_prepare_send(ent, ent->fuse_req, issue_flags);
if (err) {
if (!fuse_uring_get_next_fuse_req(ent, queue))
if (!fuse_uring_get_next_fuse_req(ent, queue,
issue_flags))
return;
err = 0;
}
@@ -1307,11 +1755,12 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
spin_lock(&queue->lock);
list_del_init(&ent->list);
fuse_uring_recycle_buffer(ent);
spin_unlock(&queue->lock);
io_uring_cmd_done(cmd, err, issue_flags);
fuse_uring_req_end(ent, ent->fuse_req, err);
fuse_uring_req_end(ent, ent->fuse_req, err, issue_flags);
kfree(ent);
if (atomic_dec_and_test(&queue->ring->queue_refs))
wake_up_all(&queue->ring->stop_waitq);
@@ -1330,7 +1779,7 @@ static struct fuse_ring_queue *fuse_uring_task_to_queue(struct fuse_ring *ring)
ring->nr_queues))
qid = 0;
queue = ring->queues[qid];
queue = READ_ONCE(ring->queues[qid]);
WARN_ONCE(!queue, "Missing queue for qid %d\n", qid);
return queue;
@@ -1368,15 +1817,16 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req)
req->ring_queue = queue;
ent = list_first_entry_or_null(&queue->ent_avail_queue,
struct fuse_ring_ent, list);
if (ent)
fuse_uring_add_req_to_ring_ent(ent, req);
else
if (!ent || fuse_uring_prep_buffer(ent, req)) {
list_add_tail(&req->list, &queue->fuse_req_queue);
spin_unlock(&queue->lock);
return;
}
fuse_uring_add_req_to_ring_ent(ent, req);
spin_unlock(&queue->lock);
if (ent)
fuse_uring_dispatch_ent(ent);
fuse_uring_dispatch_ent(ent);
return;
err_unlock:
@@ -1424,10 +1874,9 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)
*/
req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req,
list);
if (ent && req) {
if (ent && req && !fuse_uring_prep_buffer(ent, req)) {
fuse_uring_add_req_to_ring_ent(ent, req);
spin_unlock(&queue->lock);
fuse_uring_dispatch_ent(ent);
} else {
spin_unlock(&queue->lock);

56
fs/fuse/dev_uring_i.h
View File

@@ -7,6 +7,8 @@
#ifndef _FS_FUSE_DEV_URING_I_H
#define _FS_FUSE_DEV_URING_I_H
#include <linux/uio.h>
#include "fuse_dev_i.h"
#ifdef CONFIG_FUSE_IO_URING
@@ -36,11 +38,50 @@ enum fuse_ring_req_state {
FRRS_RELEASED,
};
/* how a queue's payload buffers are provided */
enum fuse_queue_payload_mode {
/* not yet committed (a bufpool may still be added) */
FUSE_PAYLOAD_UNSET = 0,
/* each entry registers its own payload buffer */
FUSE_PAYLOAD_PER_ENT,
/* each entry's payload buffer is assigned from a bufpool */
FUSE_PAYLOAD_BUFPOOL,
};
struct fuse_bufpool {
bool registered;
/*
* io_uring registered buffer table index for this pool, bound at
* ADD_BUFPOOL time. Only valid if the bufpool is registered
*/
u16 registered_index;
/* starting uaddr of the bufpool */
uintptr_t base_uaddr;
/* size of each buffer in the pool */
size_t buf_size;
/* total number of buffers in the pool */
unsigned int nr_bufs;
/* bitmap tracking which buffers are free */
unsigned long free_map[];
};
/** A fuse ring entry, part of the ring queue */
struct fuse_ring_ent {
/* userspace buffer */
struct fuse_uring_req_header __user *headers;
void __user *payload;
struct iovec payload;
/* buffer id in the pool, if bufpools are used. ignored otherwise */
unsigned int buf_id;
/* true if the request's pages are being zero-copied */
bool zero_copied;
unsigned int zero_copy_index;
/* the ring queue that owns the request */
struct fuse_ring_queue *queue;
@@ -99,6 +140,14 @@ struct fuse_ring_queue {
unsigned int active_background;
bool stopped;
/* how this queue's payload buffers are provided */
enum fuse_queue_payload_mode payload_mode;
/* only allocated when payload_mode == FUSE_PAYLOAD_BUFPOOL */
struct fuse_bufpool *bufpool;
bool zero_copy;
};
/*
@@ -135,6 +184,7 @@ struct fuse_ring {
bool ready;
};
void fuse_uring_conn_init(struct fuse_chan *fch);
void fuse_uring_stop_queues(struct fuse_ring *ring);
void fuse_uring_abort_end_requests(struct fuse_ring *ring);
int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags);
@@ -174,6 +224,10 @@ static inline bool fuse_uring_ready(struct fuse_chan *fch)
#else /* CONFIG_FUSE_IO_URING */
static inline void fuse_uring_conn_init(struct fuse_chan *fch)
{
}
static inline void fuse_uring_abort(struct fuse_chan *fch)
{
}

37
fs/fuse/dir.c
View File

@@ -34,7 +34,7 @@ struct dentry_bucket {
#define FUSE_HASH_BITS 5
#define FUSE_HASH_SIZE (1 << FUSE_HASH_BITS)
static struct dentry_bucket dentry_hash[FUSE_HASH_SIZE];
struct delayed_work dentry_tree_work;
static struct delayed_work dentry_tree_work;
/* Minimum invalidation work queue frequency */
#define FUSE_DENTRY_INVAL_FREQ_MIN 5
@@ -96,6 +96,7 @@ static void fuse_advise_use_readdirplus(struct inode *dir)
struct fuse_dentry {
u64 time;
u64 epoch;
union {
struct rcu_head rcu;
struct rb_node node;
@@ -236,6 +237,13 @@ void fuse_dentry_tree_cleanup(void)
WARN_ON_ONCE(!RB_EMPTY_ROOT(&dentry_hash[i].tree));
}
void fuse_dentry_set_epoch(struct dentry *dentry, u64 epoch)
{
struct fuse_dentry *fd = dentry->d_fsdata;
fd->epoch = epoch;
}
static inline void __fuse_dentry_settime(struct dentry *dentry, u64 time)
{
((struct fuse_dentry *) dentry->d_fsdata)->time = time;
@@ -387,10 +395,11 @@ static int fuse_dentry_revalidate(struct inode *dir, const struct qstr *name,
struct fuse_mount *fm;
struct fuse_conn *fc;
struct fuse_inode *fi;
struct fuse_dentry *fd = entry->d_fsdata;
int ret;
fc = get_fuse_conn_super(dir->i_sb);
if (entry->d_time < atomic_read(&fc->epoch))
if (fd->epoch < atomic_read(&fc->epoch))
goto invalid;
inode = d_inode_rcu(entry);
@@ -480,10 +489,10 @@ static int fuse_dentry_init(struct dentry *dentry)
RB_CLEAR_NODE(&fd->node);
dentry->d_fsdata = fd;
/*
* Initialising d_time (epoch) to '0' ensures the dentry is invalid
* Initialising epoch to '0' ensures the dentry is invalid
* if compared to fc->epoch, which is initialized to '1'.
*/
dentry->d_time = 0;
fuse_dentry_set_epoch(dentry, 0);
return 0;
}
@@ -641,7 +650,7 @@ static struct dentry *fuse_lookup(struct inode *dir, struct dentry *entry,
goto out_err;
entry = newent ? newent : entry;
entry->d_time = epoch;
fuse_dentry_set_epoch(entry, epoch);
if (outarg_valid)
fuse_change_entry_timeout(entry, &outarg);
else
@@ -898,7 +907,7 @@ static int fuse_create_open(struct mnt_idmap *idmap, struct inode *dir,
}
kfree(forget);
d_instantiate(entry, inode);
entry->d_time = epoch;
fuse_dentry_set_epoch(entry, epoch);
fuse_change_entry_timeout(entry, &outentry);
fuse_dir_changed(dir);
err = generic_file_open(inode, file);
@@ -1028,10 +1037,10 @@ static struct dentry *create_new_entry(struct mnt_idmap *idmap, struct fuse_moun
return d;
if (d) {
d->d_time = epoch;
fuse_dentry_set_epoch(d, epoch);
fuse_change_entry_timeout(d, &outarg);
} else {
entry->d_time = epoch;
fuse_dentry_set_epoch(entry, epoch);
fuse_change_entry_timeout(entry, &outarg);
}
fuse_dir_changed(dir);
@@ -2169,10 +2178,8 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
filemap_invalidate_lock(mapping);
fault_blocked = true;
err = fuse_dax_break_layouts(inode, 0, -1);
if (err) {
filemap_invalidate_unlock(mapping);
return err;
}
if (err)
goto unlock;
}
if (attr->ia_valid & ATTR_OPEN) {
@@ -2199,7 +2206,7 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
ATTR_TIMES_SET)) {
err = write_inode_now(inode, true);
if (err)
return err;
goto unlock;
fuse_set_nowrite(inode);
fuse_release_nowrite(inode);
@@ -2290,6 +2297,9 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
*/
if ((is_truncate || !is_wb) &&
S_ISREG(inode->i_mode) && oldsize != outarg.attr.size) {
if (outarg.attr.size > oldsize)
truncate_pagecache_range(inode, oldsize,
outarg.attr.size - 1);
truncate_pagecache(inode, outarg.attr.size);
invalidate_inode_pages2(mapping);
}
@@ -2307,6 +2317,7 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
clear_bit(FUSE_I_SIZE_UNSTABLE, &fi->state);
unlock:
if (fault_blocked)
filemap_invalidate_unlock(mapping);
return err;

20
fs/fuse/file.c
View File

@@ -272,7 +272,7 @@ static int fuse_open(struct inode *inode, struct file *file)
filemap_invalidate_lock(inode->i_mapping);
err = fuse_dax_break_layouts(inode, 0, -1);
if (err)
goto out_inode_unlock;
goto out_unlock;
}
if (is_wb_truncate || dax_truncate)
@@ -296,9 +296,9 @@ static int fuse_open(struct inode *inode, struct file *file)
else if (!(ff->open_flags & FOPEN_KEEP_CACHE))
invalidate_inode_pages2(inode->i_mapping);
}
out_unlock:
if (dax_truncate)
filemap_invalidate_unlock(inode->i_mapping);
out_inode_unlock:
if (is_wb_truncate || dax_truncate)
inode_unlock(inode);
@@ -605,6 +605,7 @@ void fuse_read_args_fill(struct fuse_io_args *ia, struct file *file, loff_t pos,
args->out_argvar = true;
args->out_numargs = 1;
args->out_args[0].size = count;
args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY;
}
static void fuse_release_user_pages(struct fuse_args_pages *ap, ssize_t nres,
@@ -1153,6 +1154,7 @@ static void fuse_write_args_fill(struct fuse_io_args *ia, struct fuse_file *ff,
args->out_numargs = 1;
args->out_args[0].size = sizeof(ia->write.out);
args->out_args[0].value = &ia->write.out;
args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY;
}
static unsigned int fuse_write_flags(struct kiocb *iocb)
@@ -1346,9 +1348,13 @@ static ssize_t fuse_perform_write(struct kiocb *iocb, struct iov_iter *ii)
struct fuse_conn *fc = get_fuse_conn(inode);
struct fuse_inode *fi = get_fuse_inode(inode);
loff_t pos = iocb->ki_pos;
loff_t old_size = i_size_read(inode);
int err = 0;
ssize_t res = 0;
if (pos > old_size)
truncate_pagecache_range(inode, old_size, pos - 1);
if (inode->i_size < pos + iov_iter_count(ii))
set_bit(FUSE_I_SIZE_UNSTABLE, &fi->state);
@@ -1787,13 +1793,14 @@ static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from)
{
struct inode *inode = file_inode(iocb->ki_filp);
struct address_space *mapping = inode->i_mapping;
loff_t pos = iocb->ki_pos;
ssize_t res;
bool exclusive;
fuse_dio_lock(iocb, from, &exclusive);
res = generic_write_checks(iocb, from);
if (res > 0) {
loff_t pos = iocb->ki_pos;
task_io_account_write(res);
if (!is_sync_kiocb(iocb)) {
res = fuse_direct_IO(iocb, from);
@@ -1808,7 +1815,7 @@ static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from)
/*
* As in generic_file_direct_write(), invalidate after
* write, to invalidate read-ahead cache that may have
* with the write.
* competed with the write.
*/
invalidate_inode_pages2_range(mapping,
pos >> PAGE_SHIFT,
@@ -2894,6 +2901,11 @@ static long fuse_file_fallocate(struct file *file, int mode, loff_t offset,
/* we could have extended the file */
if (!(mode & FALLOC_FL_KEEP_SIZE)) {
loff_t oldsize = i_size_read(inode);
if (offset + length > oldsize)
truncate_pagecache_range(inode, oldsize,
offset + length - 1);
if (fuse_write_update_attr(inode, offset + length, length))
file_update_time(file);
}

5
fs/fuse/fuse_dev_i.h
View File

@@ -38,6 +38,8 @@ struct fuse_iqueue;
* @FR_PRIVATE: request is on private list
* @FR_ASYNC: request is asynchronous
* @FR_URING: request is handled through fuse-io-uring
* @FR_SYNC_WAKEUP: use synchronous wakeup when queueing this request to
* give the scheduler a hint about the waker task
*/
enum fuse_req_flag {
FR_ISREPLY,
@@ -53,6 +55,7 @@ enum fuse_req_flag {
FR_PRIVATE,
FR_ASYNC,
FR_URING,
FR_SYNC_WAKEUP,
};
/**
@@ -325,6 +328,8 @@ struct fuse_copy_state {
bool write:1;
bool move_folios:1;
bool is_uring:1;
/* set when the payload is zero-copied. folios are filled in place */
bool skip_folio_copy:1;
struct {
unsigned int copied_sz; /* copied size into the user buffer */
} ring;

2
fs/fuse/fuse_i.h
View File

@@ -1054,6 +1054,8 @@ u64 fuse_time_to_jiffies(u64 sec, u32 nsec);
void fuse_change_entry_timeout(struct dentry *entry, struct fuse_entry_out *o);
void fuse_dentry_set_epoch(struct dentry *dentry, u64 epoch);
/*
* Initialize fuse_conn
*/

16
fs/fuse/inode.c
View File

@@ -791,6 +791,9 @@ static int fuse_opt_fd(struct fs_context *fsc, struct file *file)
{
struct fuse_fs_context *ctx = fsc->fs_private;
if (ctx->fud)
return invalfc(fsc, "Multiple fd specified");
if (file->f_op != &fuse_dev_operations)
return invalfc(fsc, "fd is not a fuse device");
/*
@@ -1272,6 +1275,7 @@ static void process_init_reply(struct fuse_args *args, int error)
struct fuse_mount *fm = ia->fm;
struct fuse_conn *fc = fm->fc;
struct fuse_init_out *arg = &ia->out;
bool io_uring_enabled = false;
bool ok = true;
if (error || arg->major != FUSE_KERNEL_VERSION)
@@ -1402,7 +1406,7 @@ static void process_init_reply(struct fuse_args *args, int error)
ok = false;
}
if (flags & FUSE_OVER_IO_URING && fuse_uring_enabled())
fuse_chan_io_uring_enable(fc->chan);
io_uring_enabled = true;
if (flags & FUSE_REQUEST_TIMEOUT)
timeout = arg->request_timeout;
@@ -1416,6 +1420,7 @@ static void process_init_reply(struct fuse_args *args, int error)
fm->sb->s_bdi->ra_pages =
min(fm->sb->s_bdi->ra_pages, ra_pages);
fm->sb->s_bdi->io_pages = fc->max_pages;
fc->minor = arg->minor;
fc->max_write = arg->minor < 5 ? 4096 : arg->max_write;
fc->max_write = max_t(unsigned, 4096, fc->max_write);
@@ -1432,6 +1437,7 @@ static void process_init_reply(struct fuse_args *args, int error)
.minor = fc->minor,
.max_write = fc->max_write,
.max_pages = fc->max_pages,
.io_uring_enabled = io_uring_enabled,
};
fuse_chan_set_initialized(fc->chan, &cp);
}
@@ -1474,12 +1480,8 @@ static struct fuse_init_args *fuse_new_init(struct fuse_mount *fm)
if (IS_ENABLED(CONFIG_FUSE_PASSTHROUGH))
flags |= FUSE_PASSTHROUGH;
/*
* This is just an information flag for fuse server. No need to check
* the reply - server is either sending IORING_OP_URING_CMD or not.
*/
if (fuse_uring_enabled())
flags |= FUSE_OVER_IO_URING;
flags |= FUSE_OVER_IO_URING | FUSE_HAS_IO_URING_BUFPOOL;
ia->in.flags = flags;
ia->in.flags2 = flags >> 32;
@@ -1639,6 +1641,8 @@ static int fuse_fill_super_submount(struct super_block *sb,
fuse_fill_attr_from_inode(&root_attr, parent_fi);
root = fuse_iget(sb, parent_fi->nodeid, 0, &root_attr, 0, 0,
fuse_get_evict_ctr(fm->fc));
if (!root)
return -ENOMEM;
/*
* This inode is just a duplicate, so it is not looked up and
* its nlookup should not be incremented. fuse_iget() does

2
fs/fuse/readdir.c
View File

@@ -260,7 +260,7 @@ static int fuse_direntplus_link(struct file *file,
}
if (fc->readdirplus_auto)
set_bit(FUSE_I_INIT_RDPLUS, &get_fuse_inode(inode)->state);
dentry->d_time = epoch;
fuse_dentry_set_epoch(dentry, epoch);
fuse_change_entry_timeout(dentry, o);
dput(dentry);

12
fs/fuse/req_timeout.c
View File

@@ -128,18 +128,12 @@ static void set_request_timeout(struct fuse_chan *fch, unsigned int timeout)
void fuse_init_server_timeout(struct fuse_chan *fch, unsigned int timeout)
{
if (!timeout && !fuse_max_req_timeout && !fuse_default_req_timeout)
return;
if (!timeout)
timeout = fuse_default_req_timeout;
if (fuse_max_req_timeout) {
if (timeout)
timeout = min(fuse_max_req_timeout, timeout);
else
timeout = fuse_max_req_timeout;
}
timeout = min_not_zero(timeout, fuse_max_req_timeout);
if (!timeout)
return;
timeout = max(FUSE_TIMEOUT_TIMER_FREQ, timeout);

38
include/linux/io_uring/cmd.h
View File

@@ -91,6 +91,15 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd,
bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,
struct io_br_sel *sel, unsigned int issue_flags);
int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq,
void (*release)(void *), unsigned int index,
unsigned int issue_flags);
int io_buffer_register_bvec(struct io_uring_cmd *cmd, const struct bio_vec *bvs,
unsigned int nr_bvecs, void (*release)(void *),
void *priv, u8 dir, unsigned int index,
unsigned int issue_flags);
int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index,
unsigned int issue_flags);
#else
static inline int
io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -133,6 +142,29 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,
{
return true;
}
static inline int io_buffer_register_request(struct io_uring_cmd *cmd,
struct request *rq,
void (*release)(void *),
unsigned int index,
unsigned int issue_flags)
{
return -EOPNOTSUPP;
}
static inline int io_buffer_register_bvec(struct io_uring_cmd *cmd,
const struct bio_vec *bvs,
unsigned int nr_bvecs,
void (*release)(void *), void *priv,
u8 dir, unsigned int index,
unsigned int issue_flags)
{
return -EOPNOTSUPP;
}
static inline int io_buffer_unregister(struct io_uring_cmd *cmd,
unsigned int index,
unsigned int issue_flags)
{
return -EOPNOTSUPP;
}
#endif
static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
@@ -182,10 +214,4 @@ static inline void io_uring_cmd_done32(struct io_uring_cmd *ioucmd, s32 ret,
return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true);
}
int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq,
void (*release)(void *), unsigned int index,
unsigned int issue_flags);
int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,
unsigned int issue_flags);
#endif /* _LINUX_IO_URING_CMD_H */

5
include/linux/io_uring_types.h
View File

@@ -53,6 +53,11 @@ enum io_uring_cmd_flags {
IO_URING_F_COMPAT = (1 << 12),
};
enum {
IO_BUF_DEST = 1 << ITER_DEST,
IO_BUF_SOURCE = 1 << ITER_SOURCE,
};
struct iou_loop_params;
struct io_wq_work_node {

63
include/uapi/linux/fuse.h
View File

@@ -240,6 +240,14 @@
* - add FUSE_COPY_FILE_RANGE_64
* - add struct fuse_copy_file_range_out
* - add FUSE_NOTIFY_PRUNE
*
* 7.46
* - add FUSE_IO_URING_CMD_ADD_QUEUE
* - add FUSE_HAS_IO_URING_BUFPOOL
* - add fuse_uring_cmd_req bufpool struct
* - add bufpool offset field to fuse_uring_ent_in_out struct
* - add FUSE_URING_ZERO_COPY, FUSE_URING_ENT_ZERO_COPY, and
* FOPEN_IO_URING_ZERO_COPY flag
*/
#ifndef _LINUX_FUSE_H
@@ -275,7 +283,7 @@
#define FUSE_KERNEL_VERSION 7
/** Minor version number of this interface */
#define FUSE_KERNEL_MINOR_VERSION 45
#define FUSE_KERNEL_MINOR_VERSION 46
/** The node ID of the root inode */
#define FUSE_ROOT_ID 1
@@ -383,6 +391,12 @@ struct fuse_file_lock {
* FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE)
* FOPEN_PARALLEL_DIRECT_WRITES: Allow concurrent direct writes on the same inode
* FOPEN_PASSTHROUGH: passthrough read/write io for this open file
* FOPEN_IO_URING_ZERO_COPY: use io-uring zero-copy for reads/writes on this
* open file. Honored only when the serving io-uring
* queue was set up for zero-copy
* (FUSE_URING_ZERO_COPY) and the request carries page
* payload. Otherwise reads/writes fall back to
* copying.
*/
#define FOPEN_DIRECT_IO (1 << 0)
#define FOPEN_KEEP_CACHE (1 << 1)
@@ -392,6 +406,7 @@ struct fuse_file_lock {
#define FOPEN_NOFLUSH (1 << 5)
#define FOPEN_PARALLEL_DIRECT_WRITES (1 << 6)
#define FOPEN_PASSTHROUGH (1 << 7)
#define FOPEN_IO_URING_ZERO_COPY (1 << 8)
/**
* INIT request/reply flags
@@ -448,6 +463,7 @@ struct fuse_file_lock {
* FUSE_OVER_IO_URING: Indicate that client supports io-uring
* FUSE_REQUEST_TIMEOUT: kernel supports timing out requests.
* init_out.request_timeout contains the timeout (in secs)
* FUSE_HAS_IO_URING_BUFPOOL: kernel supports io-uring buffer pools
*/
#define FUSE_ASYNC_READ (1 << 0)
#define FUSE_POSIX_LOCKS (1 << 1)
@@ -495,6 +511,7 @@ struct fuse_file_lock {
#define FUSE_ALLOW_IDMAP (1ULL << 40)
#define FUSE_OVER_IO_URING (1ULL << 41)
#define FUSE_REQUEST_TIMEOUT (1ULL << 42)
#define FUSE_HAS_IO_URING_BUFPOOL (1ULL << 43)
/**
* CUSE INIT request/reply flags
@@ -1251,6 +1268,13 @@ struct fuse_supp_groups {
#define FUSE_URING_IN_OUT_HEADER_SZ 128
#define FUSE_URING_OP_IN_OUT_SZ 128
/**
* fuse_uring_ent_in_out flags
*
* FUSE_URING_ENT_ZERO_COPY: Set if the ent's payload is zero-copied
*/
#define FUSE_URING_ENT_ZERO_COPY (1 << 0)
/* Used as part of the fuse_uring_req_header */
struct fuse_uring_ent_in_out {
uint64_t flags;
@@ -1263,7 +1287,9 @@ struct fuse_uring_ent_in_out {
/* size of user payload buffer */
uint32_t payload_sz;
uint32_t padding;
/* Offset into the bufpool, if bufpools are used */
uint32_t offset;
uint64_t reserved;
};
@@ -1292,8 +1318,22 @@ enum fuse_uring_cmd {
/* commit fuse request result and fetch next request */
FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2,
/* add a queue */
FUSE_IO_URING_CMD_ADD_QUEUE = 3,
/* add a bufpool to a queue */
FUSE_IO_URING_CMD_ADD_BUFPOOL = 4,
};
/*
* fuse_uring_cmd_req flags for FUSE_IO_URING_CMD_ADD_QUEUE
*
* FUSE_URING_ZERO_COPY is only supported for queues with bufpools on privileged
* servers
*/
#define FUSE_URING_ZERO_COPY (1 << 0)
/**
* In the 80B command area of the SQE.
*/
@@ -1306,6 +1346,25 @@ struct fuse_uring_cmd_req {
/* queue the command is for (queue index) */
uint16_t qid;
uint8_t padding[6];
union {
struct {
/* base address of bufpool */
uint64_t uaddr;
uint32_t len;
uint32_t reserved;
} bufpool;
/*
* Index of this entry's slot in the server's io_uring
* registered buffer table, where the kernel registers the
* request's pages for zero-copy. Set for
* FUSE_IO_URING_CMD_REGISTER cmds only, and only on queues
* created with FUSE_URING_ZERO_COPY. On a non-zero-copy queue
* this must be 0
*/
uint16_t ent_zero_copy_buf_index;
};
};
#endif /* _LINUX_FUSE_H */

2
io_uring/io_uring.c
View File

@@ -3236,7 +3236,7 @@ static int __init io_uring_init(void)
io_uring_optable_init();
/* imu->dir is u8 */
BUILD_BUG_ON((IO_IMU_DEST | IO_IMU_SOURCE) > U8_MAX);
BUILD_BUG_ON((IO_BUF_DEST | IO_BUF_SOURCE) > U8_MAX);
/*
* Allow user copy in the per-command field, which starts after the

131
io_uring/rsrc.c
View File

@@ -912,7 +912,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
imu->release = io_release_ubuf;
imu->priv = imu;
imu->flags = 0;
imu->dir = IO_IMU_DEST | IO_IMU_SOURCE;
imu->dir = IO_BUF_DEST | IO_BUF_SOURCE;
if (coalesced)
imu->folio_shift = data.folio_shift;
refcount_set(&imu->refs, 1);
@@ -1015,71 +1015,124 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,
return ret;
}
int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq,
void (*release)(void *), unsigned int index,
unsigned int issue_flags)
static struct io_mapped_ubuf *io_kernel_buffer_init(struct io_ring_ctx *ctx,
unsigned int nr_bvecs,
unsigned int total_bytes,
u8 dir,
void (*release)(void *),
void *priv,
unsigned int index)
{
struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
struct io_rsrc_data *data = &ctx->buf_table;
struct req_iterator rq_iter;
struct io_mapped_ubuf *imu;
struct io_rsrc_node *node;
struct bio_vec bv;
unsigned int nr_bvecs = 0;
int ret = 0;
io_ring_submit_lock(ctx, issue_flags);
if (index >= data->nr) {
ret = -EINVAL;
goto unlock;
}
if (index >= data->nr)
return ERR_PTR(-EINVAL);
index = array_index_nospec(index, data->nr);
if (data->nodes[index]) {
ret = -EBUSY;
goto unlock;
}
if (data->nodes[index])
return ERR_PTR(-EBUSY);
node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);
if (!node) {
ret = -ENOMEM;
goto unlock;
if (!node)
return ERR_PTR(-ENOMEM);
imu = io_alloc_imu(ctx, nr_bvecs);
if (!imu) {
io_cache_free(&ctx->node_cache, node);
return ERR_PTR(-ENOMEM);
}
imu->ubuf = 0;
imu->len = total_bytes;
imu->folio_shift = PAGE_SHIFT;
imu->nr_bvecs = nr_bvecs;
refcount_set(&imu->refs, 1);
imu->release = release;
imu->priv = priv;
imu->dir = dir;
imu->flags = IO_REGBUF_F_KBUF;
node->buf = imu;
data->nodes[index] = node;
return imu;
}
int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq,
void (*release)(void *), unsigned int index,
unsigned int issue_flags)
{
struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
struct req_iterator rq_iter;
struct io_mapped_ubuf *imu;
struct bio_vec bv;
/*
* blk_rq_nr_phys_segments() may overestimate the number of bvecs
* but avoids needing to iterate over the bvecs
*/
imu = io_alloc_imu(ctx, blk_rq_nr_phys_segments(rq));
if (!imu) {
io_cache_free(&ctx->node_cache, node);
ret = -ENOMEM;
unsigned int nr_bvecs = blk_rq_nr_phys_segments(rq);
unsigned int total_bytes = blk_rq_bytes(rq);
int ret = 0;
io_ring_submit_lock(ctx, issue_flags);
imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes,
1 << rq_data_dir(rq), release, rq, index);
if (IS_ERR(imu)) {
ret = PTR_ERR(imu);
goto unlock;
}
imu->ubuf = 0;
imu->len = blk_rq_bytes(rq);
imu->folio_shift = PAGE_SHIFT;
refcount_set(&imu->refs, 1);
imu->release = release;
imu->priv = rq;
imu->flags = IO_REGBUF_F_KBUF;
imu->dir = 1 << rq_data_dir(rq);
nr_bvecs = 0;
rq_for_each_bvec(bv, rq, rq_iter)
imu->bvec[nr_bvecs++] = bv;
imu->nr_bvecs = nr_bvecs;
node->buf = imu;
data->nodes[index] = node;
unlock:
io_ring_submit_unlock(ctx, issue_flags);
return ret;
}
EXPORT_SYMBOL_GPL(io_buffer_register_request);
/*
* bvs is copied internally. caller may free it on return.
*/
int io_buffer_register_bvec(struct io_uring_cmd *cmd, const struct bio_vec *bvs,
unsigned int nr_bvecs, void (*release)(void *),
void *priv, u8 dir, unsigned int index,
unsigned int issue_flags)
{
struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
struct io_mapped_ubuf *imu;
struct bio_vec *bvec;
unsigned int i, total_bytes = 0;
int ret = 0;
for (i = 0; i < nr_bvecs; i++)
total_bytes += bvs[i].bv_len;
io_ring_submit_lock(ctx, issue_flags);
imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes, dir, release,
priv, index);
if (IS_ERR(imu)) {
ret = PTR_ERR(imu);
goto unlock;
}
bvec = imu->bvec;
for (i = 0; i < nr_bvecs; i++)
bvec[i] = bvs[i];
unlock:
io_ring_submit_unlock(ctx, issue_flags);
return ret;
}
EXPORT_SYMBOL_GPL(io_buffer_register_bvec);
int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,
unsigned int issue_flags)
int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index,
unsigned int issue_flags)
{
struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
struct io_rsrc_data *data = &ctx->buf_table;
@@ -1109,7 +1162,7 @@ int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,
io_ring_submit_unlock(ctx, issue_flags);
return ret;
}
EXPORT_SYMBOL_GPL(io_buffer_unregister_bvec);
EXPORT_SYMBOL_GPL(io_buffer_unregister);
static int validate_fixed_range(u64 buf_addr, size_t len,
const struct io_mapped_ubuf *imu)

5
io_uring/rsrc.h
View File

@@ -25,11 +25,6 @@ struct io_rsrc_node {
};
};
enum {
IO_IMU_DEST = 1 << ITER_DEST,
IO_IMU_SOURCE = 1 << ITER_SOURCE,
};
enum {
IO_REGBUF_F_KBUF = 1,
};

1
tools/testing/selftests/filesystems/fuse/.gitignore vendored
View File

@@ -1,3 +1,4 @@
# SPDX-License-Identifier: GPL-2.0-only
fuse_mnt
fusectl_test
write_extend_eof_test

3
tools/testing/selftests/filesystems/fuse/Makefile
View File

@@ -3,6 +3,7 @@
CFLAGS += -Wall -O2 -g $(KHDR_INCLUDES)
TEST_GEN_PROGS := fusectl_test
TEST_GEN_PROGS += write_extend_eof_test
TEST_GEN_FILES := fuse_mnt
# fuse_acl_cache_test requires libfuse3; add it only when the library is present.
@@ -14,6 +15,8 @@ endif
include ../../lib.mk
$(OUTPUT)/write_extend_eof_test: LDLIBS += -lpthread
VAR_CFLAGS := $(shell pkg-config fuse --cflags 2>/dev/null)
ifeq ($(VAR_CFLAGS),)
VAR_CFLAGS := -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse

368
tools/testing/selftests/filesystems/fuse/write_extend_eof_test.c Normal file
View File

@@ -0,0 +1,368 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Regression test for the fuse write-extend partial-EOF-page zeroing bug.
*
* A buffered write that extends i_size past a non-page-aligned EOF must zero
* the tail of the old last page. If an application has mmap'd that page and
* stored into the post-EOF region (undefined until the file grows), the
* now-in-bounds tail must read back as zero, not as the stale stored bytes.
*
* The bug is exposed on a non-writeback_cache server that keeps the page cache
* across the write (FOPEN_KEEP_CACHE without FOPEN_DIRECT_IO). This test is a
* raw /dev/fuse server in that mode; the backing data is always zero in the
* hole, so any non-zero byte a read sees is stale page-cache data.
*
* Requires root to mount fuse.
*/
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <linux/falloc.h>
#include <pthread.h>
#include <stdint.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/uio.h>
#include <linux/fuse.h>
#include "../../kselftest_harness.h"
#define FUSE_ROOT_ID 1
#define FILE_INO 2
#define MAX_WRITE (128 * 1024)
#define BACKING_SIZE (4 * 1024 * 1024)
#define POLLUTE 0xee
/* Server-side state, shared with the responder thread. */
struct server {
int fd;
unsigned char backing[BACKING_SIZE]; /* authoritative bytes */
uint64_t size;
};
static void reply(int fd, uint64_t unique, int error, void *data, size_t len)
{
struct fuse_out_header oh = {
.len = sizeof(oh) + (data ? len : 0),
.error = error,
.unique = unique,
};
struct iovec iov[2] = { { &oh, sizeof(oh) }, { data, len } };
/* Errors here are teardown races (device closed on unmount); ignore. */
if (writev(fd, iov, data ? 2 : 1) < 0)
return;
}
static void fill_attr(struct fuse_attr *a, uint64_t ino, uint32_t mode,
uint64_t size)
{
memset(a, 0, sizeof(*a));
a->ino = ino;
a->mode = mode;
a->nlink = 1;
a->size = size;
a->blksize = sysconf(_SC_PAGESIZE);
}
static void *server_thread(void *arg)
{
struct server *s = arg;
static char buf[MAX_WRITE + 4096];
for (;;) {
ssize_t n = read(s->fd, buf, sizeof(buf));
struct fuse_in_header *ih = (void *)buf;
if (n < 0) {
if (errno == EINTR || errno == EAGAIN)
continue;
return NULL; /* device closed on unmount */
}
if (n < (ssize_t)sizeof(*ih))
continue;
switch (ih->opcode) {
case FUSE_INIT: {
struct fuse_init_in *in = (void *)(ih + 1);
struct fuse_init_out out = {0};
/* No FUSE_WRITEBACK_CACHE: the exposed configuration. */
out.major = FUSE_KERNEL_VERSION;
out.minor = FUSE_KERNEL_MINOR_VERSION;
out.max_readahead = in->max_readahead;
out.max_write = MAX_WRITE;
out.max_background = 16;
out.congestion_threshold = 12;
out.flags = FUSE_MAX_PAGES;
out.max_pages = MAX_WRITE / sysconf(_SC_PAGESIZE);
reply(s->fd, ih->unique, 0, &out, sizeof(out));
break;
}
case FUSE_GETATTR: {
struct fuse_attr_out out = {0};
int root = ih->nodeid == FUSE_ROOT_ID;
out.attr_valid = 3600;
fill_attr(&out.attr, ih->nodeid,
root ? (S_IFDIR | 0755) : (S_IFREG | 0644),
root ? 0 : s->size);
reply(s->fd, ih->unique, 0, &out, sizeof(out));
break;
}
case FUSE_LOOKUP: {
struct fuse_entry_out out = {0};
out.nodeid = FILE_INO;
out.attr_valid = 3600;
out.entry_valid = 3600;
fill_attr(&out.attr, FILE_INO, S_IFREG | 0644, s->size);
reply(s->fd, ih->unique, 0, &out, sizeof(out));
break;
}
case FUSE_OPEN:
case FUSE_OPENDIR: {
struct fuse_open_out out = {0};
/* Keep the cache across the write, but not direct I/O. */
out.open_flags = FOPEN_KEEP_CACHE;
reply(s->fd, ih->unique, 0, &out, sizeof(out));
break;
}
case FUSE_READ: {
struct fuse_read_in *in = (void *)(ih + 1);
uint64_t off = in->offset;
uint32_t size = in->size;
if (off >= BACKING_SIZE)
size = 0;
else if (off + size > BACKING_SIZE)
size = BACKING_SIZE - off;
reply(s->fd, ih->unique, 0, s->backing + off, size);
break;
}
case FUSE_WRITE: {
struct fuse_write_in *in = (void *)(ih + 1);
struct fuse_write_out out = {0};
uint64_t off = in->offset;
uint32_t size = in->size;
if (off < BACKING_SIZE) {
uint32_t c = size;
if (off + c > BACKING_SIZE)
c = BACKING_SIZE - off;
memcpy(s->backing + off, in + 1, c);
if (off + c > s->size)
s->size = off + c;
}
out.size = size;
reply(s->fd, ih->unique, 0, &out, sizeof(out));
break;
}
case FUSE_SETATTR: {
struct fuse_setattr_in *in = (void *)(ih + 1);
struct fuse_attr_out out = {0};
if ((in->valid & FATTR_SIZE) && in->size <= BACKING_SIZE) {
if (in->size > s->size)
memset(s->backing + s->size, 0,
in->size - s->size);
s->size = in->size;
}
out.attr_valid = 3600;
fill_attr(&out.attr, ih->nodeid, S_IFREG | 0644, s->size);
reply(s->fd, ih->unique, 0, &out, sizeof(out));
break;
}
case FUSE_FALLOCATE: {
struct fuse_fallocate_in *in = (void *)(ih + 1);
uint64_t end = in->offset + in->length;
/* Only plain (size-extending) fallocate is used here. */
if (!(in->mode & FALLOC_FL_KEEP_SIZE) &&
end <= BACKING_SIZE && end > s->size) {
memset(s->backing + s->size, 0, end - s->size);
s->size = end;
}
reply(s->fd, ih->unique, 0, NULL, 0);
break;
}
case FUSE_FLUSH:
case FUSE_RELEASE:
case FUSE_RELEASEDIR:
case FUSE_FSYNC:
case FUSE_ACCESS:
reply(s->fd, ih->unique, 0, NULL, 0);
break;
case FUSE_FORGET:
break;
default:
reply(s->fd, ih->unique, -EOPNOTSUPP, NULL, 0);
break;
}
}
}
FIXTURE(fuse)
{
struct server *srv;
pthread_t thread;
char dir[64];
long page; /* runtime page size */
off_t eof; /* mid-page EOF, page-relative */
int fd; /* open test file */
char *map; /* mmap of the EOF page */
int mounted;
};
FIXTURE_SETUP(fuse)
{
char opts[128];
pthread_t t;
if (geteuid() != 0)
SKIP(return, "need root to mount fuse");
self->page = sysconf(_SC_PAGESIZE);
self->fd = -1;
self->map = MAP_FAILED;
self->srv = mmap(NULL, sizeof(*self->srv), PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_ANONYMOUS, -1, 0);
ASSERT_NE(MAP_FAILED, self->srv);
self->srv->fd = open("/dev/fuse", O_RDWR);
ASSERT_GE(self->srv->fd, 0);
strcpy(self->dir, "/tmp/fuse_weof_XXXXXX");
ASSERT_NE(NULL, mkdtemp(self->dir));
snprintf(opts, sizeof(opts),
"fd=%d,rootmode=40000,user_id=0,group_id=0",
self->srv->fd);
ASSERT_EQ(0, mount("fuse", self->dir, "fuse", 0, opts));
self->mounted = 1;
ASSERT_EQ(0, pthread_create(&t, NULL, server_thread, self->srv));
self->thread = t;
}
FIXTURE_TEARDOWN(fuse)
{
if (self->map != MAP_FAILED)
munmap(self->map, self->page);
if (self->fd >= 0)
close(self->fd);
if (self->mounted)
umount2(self->dir, MNT_DETACH);
if (self->srv && self->srv != MAP_FAILED) {
if (self->srv->fd > 0)
close(self->srv->fd);
munmap(self->srv, sizeof(*self->srv));
}
if (self->dir[0])
rmdir(self->dir);
}
/*
* Create the test file with a mid-page EOF and mmap-store POLLUTE into its
* post-EOF tail (a legal store, undefined until the file grows). Leaves the
* file open and the EOF page mapped in the fixture for the caller to extend.
*/
static void pollute_eof_tail(struct __test_metadata *_metadata,
FIXTURE_DATA(fuse) * self)
{
off_t eof = 2 * self->page + self->page / 4;
char path[128];
char *buf;
snprintf(path, sizeof(path), "%s/file", self->dir);
self->fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644);
ASSERT_GE(self->fd, 0);
self->eof = eof;
buf = malloc(eof);
ASSERT_NE(NULL, buf);
memset(buf, 'A', eof);
ASSERT_EQ(eof, pwrite(self->fd, buf, eof, 0));
free(buf);
self->map = mmap(NULL, self->page, PROT_READ | PROT_WRITE, MAP_SHARED,
self->fd, eof & ~(self->page - 1));
ASSERT_NE(MAP_FAILED, self->map);
memset(self->map + (eof & (self->page - 1)), POLLUTE,
self->page - (eof & (self->page - 1)));
}
/* Assert the old post-EOF tail [eof, end of its page) now reads back as zero. */
static void assert_tail_zeroed(struct __test_metadata *_metadata,
FIXTURE_DATA(fuse) * self)
{
off_t base = self->eof & ~(self->page - 1);
char *tail = malloc(self->page);
int i;
ASSERT_NE(NULL, tail);
ASSERT_EQ(self->page, pread(self->fd, tail, self->page, base));
for (i = self->eof & (self->page - 1); i < self->page; i++)
ASSERT_EQ(0, tail[i]);
free(tail);
}
/* Basic: pollute the post-EOF tail, extend past it by a later write. */
TEST_F(fuse, write_extend)
{
pollute_eof_tail(_metadata, self);
ASSERT_EQ(4, pwrite(self->fd, "data", 4, 5 * self->page + self->page / 3));
assert_tail_zeroed(_metadata, self);
}
/* Extend via ftruncate() rather than a write. */
TEST_F(fuse, ftruncate_extend)
{
pollute_eof_tail(_metadata, self);
ASSERT_EQ(0, ftruncate(self->fd, 8 * self->page));
assert_tail_zeroed(_metadata, self);
}
/* Extend via fallocate() starting at the old EOF. */
TEST_F(fuse, fallocate_extend)
{
pollute_eof_tail(_metadata, self);
ASSERT_EQ(0, fallocate(self->fd, 0, self->eof, 4 * self->page));
assert_tail_zeroed(_metadata, self);
}
/* A write landing inside the old EOF page must not clobber its own data. */
TEST_F(fuse, extend_into_eof_page_preserves_data)
{
off_t base, wr;
char *buf, *rd;
int i;
pollute_eof_tail(_metadata, self);
base = self->eof & ~(self->page - 1);
wr = base + 3 * self->page / 4; /* starts in the EOF page */
buf = malloc(2 * self->page);
ASSERT_NE(NULL, buf);
memset(buf, 'B', 2 * self->page);
ASSERT_EQ(2 * self->page, pwrite(self->fd, buf, 2 * self->page, wr));
free(buf);
rd = malloc(self->page);
ASSERT_NE(NULL, rd);
ASSERT_EQ(self->page, pread(self->fd, rd, self->page, base));
/* [eof, wr) is hole -> zero; [wr, page) is written data -> 'B'. */
for (i = self->eof & (self->page - 1); i < wr - base; i++)
ASSERT_EQ(0, rd[i]);
for (i = wr - base; i < self->page; i++)
ASSERT_EQ('B', rd[i]);
free(rd);
}
TEST_HARNESS_MAIN