From 485e38995cc3b39f1aa91c1d340cc66affcea39a Mon Sep 17 00:00:00 2001 From: Simon Schippers Date: Mon, 3 Aug 2026 20:36:37 +0200 Subject: [PATCH 1/5] tun/tap: add IFF_BACKPRESSURE flag Add the IFF_BACKPRESSURE flag to the UAPI header and to its tools/ copy. The flag has no effect yet, it is the opt-in switch for the qdisc backpressure logic added by the following patches. It is added to TUN_FEATURES only in the last patch of the series, once the implementation is complete. Until then TUNSETIFF silently masks it off, as it does for any flag outside TUN_FEATURES. Keeping the flag and its users in separate patches would either leave a window where backpressure is unconditional, or make the opt-in a later add-on. Adding the flag first lets every following patch be a no-op unless it is set. Signed-off-by: Simon Schippers Link: https://patch.msgid.link/20260803183641.96882-2-simon.schippers@tu-dortmund.de Signed-off-by: Jakub Kicinski --- include/uapi/linux/if_tun.h | 4 ++++ tools/include/uapi/linux/if_tun.h | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/include/uapi/linux/if_tun.h b/include/uapi/linux/if_tun.h index 79d53c7a1ebd..a0ddc50a7534 100644 --- a/include/uapi/linux/if_tun.h +++ b/include/uapi/linux/if_tun.h @@ -69,6 +69,10 @@ #define IFF_NAPI_FRAGS 0x0020 /* Used in TUNSETIFF to bring up tun/tap without carrier */ #define IFF_NO_CARRIER 0x0040 +/* Stop the queue instead of dropping when the internal ring is full, so an + * attached qdisc applies backpressure instead of being bypassed. + */ +#define IFF_BACKPRESSURE 0x0080 #define IFF_NO_PI 0x1000 /* This flag has no real effect */ #define IFF_ONE_QUEUE 0x2000 diff --git a/tools/include/uapi/linux/if_tun.h b/tools/include/uapi/linux/if_tun.h index 2ec07de1d73b..2c85525704c1 100644 --- a/tools/include/uapi/linux/if_tun.h +++ b/tools/include/uapi/linux/if_tun.h @@ -67,6 +67,10 @@ #define IFF_TAP 0x0002 #define IFF_NAPI 0x0010 #define IFF_NAPI_FRAGS 0x0020 +/* Stop the queue instead of dropping when the internal ring is full, so an + * attached qdisc applies backpressure instead of being bypassed. + */ +#define IFF_BACKPRESSURE 0x0080 #define IFF_NO_PI 0x1000 /* This flag has no real effect */ #define IFF_ONE_QUEUE 0x2000 From 9b990ae358b38e52f61336f6c42191be567606b3 Mon Sep 17 00:00:00 2001 From: Simon Schippers Date: Mon, 3 Aug 2026 20:36:38 +0200 Subject: [PATCH 2/5] tun/tap: add ptr_ring consume helper with netdev queue wakeup Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls __tun_wake_queue(). The latter wakes the stopped netdev subqueue once half of the ring capacity has been consumed, tracked via the new cons_cnt field in tun_file. As a safety net, the queue is also woken on the last consumed entry if it leaves the ring empty. The point is to allow the queue to be stopped when it gets full, which is required for traffic shaping, implemented by the following "stop tail-drop when IFF_BACKPRESSURE is set". __tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a tun/tap device that does not opt in only the added check on the consume path remains. Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running() under a ring lock that tun_net_close() takes, so that none of them undoes its stop. The core sets it before it calls ndo_open() and clears it before it calls ndo_stop(), so it is false for exactly as long as the device is down. IFF_UP would not do, it is only cleared after ndo_stop() returns. Some implementation details: - tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume() to properly wake the queue. - __tun_wake_queue() returns early for a device that is not running, so a stop from tun_net_close() is not mistaken for backpressure, and it only wakes if the tfile still owns its slot in tun->tfiles[]. A detached tfile keeps its queue_index, which __tun_detach() may already have handed to the tfile that took over the slot. - lockdep_assert_held() enforces the documented consumer_lock precondition of __tun_wake_queue(). - __tun_detach() locks the tx_ring.consumer_lock to avoid races with the consumer on the queue_index, and that of tfile across the hand-over of the slot, which makes the ownership check above exact. - The ptr_ring_consume() call in tun_queue_purge() is not replaced with tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for the ntfile taking it over, to avoid a possible stall. The queue is only woken if the ring of the ntfile is empty, as otherwise the consumer wakes it after consuming the remaining entries. This does not matter for tun_detach_all(), as it is called during device teardown and no tfile takes over any queue. - That wake sits after synchronize_net() and tun_queue_purge(), so it can not be undone by a concurrent tun_net_xmit() or __tun_wake_queue(). - Ensure detached queues are woken on re-attach by calling the new tun_force_wake_queue() helper from tun_attach(), and reuse it across the existing wake paths. Unlike __tun_wake_queue() it ignores IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is cleared. It does honour netif_running(), but it always clears cons_cnt, so no old count is left over when the queue is stopped again. - tun_net_close() takes and releases both ring locks of every tfile before netif_tx_stop_all_queues(), so that its stop is the last write to __QUEUE_STATE_DRV_XOFF. - The aforementioned upcoming patch explains the pairing of the smp_mb() of __tun_wake_queue(). Co-developed-by: Tim Gebauer Signed-off-by: Tim Gebauer Signed-off-by: Simon Schippers Link: https://patch.msgid.link/20260803183641.96882-3-simon.schippers@tu-dortmund.de Signed-off-by: Jakub Kicinski --- drivers/net/tun.c | 118 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 114 insertions(+), 4 deletions(-) diff --git a/drivers/net/tun.c b/drivers/net/tun.c index 51e80000bd0e..d49b6bfd104d 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -145,6 +145,8 @@ struct tun_file { struct list_head next; struct tun_struct *detached; struct ptr_ring tx_ring; + /* Protected by tx_ring.consumer_lock */ + int cons_cnt; struct xdp_rxq_info xdp_rxq; }; @@ -585,11 +587,16 @@ static void __tun_detach(struct tun_file *tfile, bool clean) u16 index = tfile->queue_index; BUG_ON(index >= tun->numqueues); + spin_lock(&tfile->tx_ring.consumer_lock); rcu_assign_pointer(tun->tfiles[index], tun->tfiles[tun->numqueues - 1]); + spin_unlock(&tfile->tx_ring.consumer_lock); ntfile = rtnl_dereference(tun->tfiles[index]); + spin_lock(&ntfile->tx_ring.consumer_lock); ntfile->queue_index = index; ntfile->xdp_rxq.queue_index = index; + ntfile->cons_cnt = 0; + spin_unlock(&ntfile->tx_ring.consumer_lock); rcu_assign_pointer(tun->tfiles[tun->numqueues - 1], NULL); @@ -606,6 +613,14 @@ static void __tun_detach(struct tun_file *tfile, bool clean) tun_flow_delete_by_queue(tun, tun->numqueues + 1); /* Drop read queue */ tun_queue_purge(tfile); + spin_lock_bh(&ntfile->tx_ring.consumer_lock); + spin_lock(&ntfile->tx_ring.producer_lock); + ntfile->cons_cnt = 0; + if (netif_running(tun->dev) && + __ptr_ring_empty(&ntfile->tx_ring)) + netif_wake_subqueue(tun->dev, index); + spin_unlock(&ntfile->tx_ring.producer_lock); + spin_unlock_bh(&ntfile->tx_ring.consumer_lock); tun_set_real_num_queues(tun); } else if (tfile->detached && clean) { tun = tun_enable_queue(tfile); @@ -687,6 +702,25 @@ static void tun_detach_all(struct net_device *dev) module_put(THIS_MODULE); } +static void tun_force_wake_queue(struct tun_struct *tun, + struct tun_file *tfile) +{ + /* Ensure that the producer can not stop the + * queue concurrently by taking locks. + */ + spin_lock_bh(&tfile->tx_ring.consumer_lock); + spin_lock(&tfile->tx_ring.producer_lock); + tfile->cons_cnt = 0; + /* Tested under the locks that tun_net_close() takes, so this can not + * undo its stop. tun_net_open() wakes the queues of a device that + * comes back up. + */ + if (netif_running(tun->dev)) + netif_wake_subqueue(tun->dev, tfile->queue_index); + spin_unlock(&tfile->tx_ring.producer_lock); + spin_unlock_bh(&tfile->tx_ring.consumer_lock); +} + static int tun_attach(struct tun_struct *tun, struct file *file, bool skip_filter, bool napi, bool napi_frags, bool publish_tun) @@ -730,8 +764,11 @@ static int tun_attach(struct tun_struct *tun, struct file *file, goto out; } + spin_lock(&tfile->tx_ring.consumer_lock); tfile->queue_index = tun->numqueues; + spin_unlock(&tfile->tx_ring.consumer_lock); tfile->socket.sk->sk_shutdown &= ~RCV_SHUTDOWN; + tun_force_wake_queue(tun, tfile); if (tfile->detached) { /* Re-attach detached tfile, updating XDP queue_index */ @@ -964,6 +1001,24 @@ static int tun_net_open(struct net_device *dev) /* Net device close. */ static int tun_net_close(struct net_device *dev) { + struct tun_struct *tun = netdev_priv(dev); + struct tun_file *tfile; + int i; + + /* netif_running() is already false: take both ring locks to keep the + * wake sites out, so the stop below is the last write to + * __QUEUE_STATE_DRV_XOFF. + */ + for (i = 0; i < tun->numqueues; i++) { + tfile = rtnl_dereference(tun->tfiles[i]); + + spin_lock_bh(&tfile->tx_ring.consumer_lock); + spin_lock(&tfile->tx_ring.producer_lock); + tfile->cons_cnt = 0; + spin_unlock(&tfile->tx_ring.producer_lock); + spin_unlock_bh(&tfile->tx_ring.consumer_lock); + } + netif_tx_stop_all_queues(dev); return 0; } @@ -2116,13 +2171,61 @@ static ssize_t tun_put_user(struct tun_struct *tun, return total; } -static void *tun_ring_recv(struct tun_file *tfile, int noblock, int *err) +/* Callers must hold ring.consumer_lock */ +static void __tun_wake_queue(struct tun_struct *tun, + struct tun_file *tfile, int consumed) +{ + u16 queue_index = tfile->queue_index; + struct netdev_queue *txq; + + lockdep_assert_held(&tfile->tx_ring.consumer_lock); + + if (!(tun->flags & IFF_BACKPRESSURE)) + return; + + /* A stop from tun_net_close() is not backpressure, leave it alone. */ + if (unlikely(!netif_running(tun->dev))) + return; + + /* Only the current owner of the slot may wake its subqueue. */ + if (unlikely(rcu_access_pointer(tun->tfiles[queue_index]) != tfile)) + return; + + txq = netdev_get_tx_queue(tun->dev, queue_index); + + /* Paired with smp_mb__after_atomic() in tun_net_xmit() */ + smp_mb(); + if (netif_tx_queue_stopped(txq)) { + tfile->cons_cnt += consumed; + if (tfile->cons_cnt >= tfile->tx_ring.size / 2 || + __ptr_ring_empty(&tfile->tx_ring)) { + netif_tx_wake_queue(txq); + tfile->cons_cnt = 0; + } + } +} + +static void *tun_ring_consume(struct tun_struct *tun, struct tun_file *tfile) +{ + void *ptr; + + spin_lock(&tfile->tx_ring.consumer_lock); + ptr = __ptr_ring_consume(&tfile->tx_ring); + if (ptr) + __tun_wake_queue(tun, tfile, 1); + + spin_unlock(&tfile->tx_ring.consumer_lock); + return ptr; +} + +static void *tun_ring_recv(struct tun_struct *tun, struct tun_file *tfile, + int noblock, int *err) { DECLARE_WAITQUEUE(wait, current); void *ptr = NULL; int error = 0; - ptr = ptr_ring_consume(&tfile->tx_ring); + ptr = tun_ring_consume(tun, tfile); if (ptr) goto out; if (noblock) { @@ -2134,7 +2237,7 @@ static void *tun_ring_recv(struct tun_file *tfile, int noblock, int *err) while (1) { set_current_state(TASK_INTERRUPTIBLE); - ptr = ptr_ring_consume(&tfile->tx_ring); + ptr = tun_ring_consume(tun, tfile); if (ptr) break; if (signal_pending(current)) { @@ -2171,7 +2274,7 @@ static ssize_t tun_do_read(struct tun_struct *tun, struct tun_file *tfile, if (!ptr) { /* Read frames from ring */ - ptr = tun_ring_recv(tfile, noblock, &err); + ptr = tun_ring_recv(tun, tfile, noblock, &err); if (!ptr) return err; } @@ -3630,6 +3733,13 @@ static int tun_queue_resize(struct tun_struct *tun) dev->tx_queue_len, GFP_KERNEL, tun_ptr_free); + if (!ret) { + for (i = 0; i < tun->numqueues; i++) { + tfile = rtnl_dereference(tun->tfiles[i]); + tun_force_wake_queue(tun, tfile); + } + } + kfree(rings); return ret; } From f65c1fb427aa5726cfe7a1b903f733f3a27099bb Mon Sep 17 00:00:00 2001 From: Simon Schippers Date: Mon, 3 Aug 2026 20:36:39 +0200 Subject: [PATCH 3/5] vhost-net: wake queue of tun/tap after ptr_ring consume Add tun_wake_queue() to tun.c and export it for use by vhost-net. The function validates that the file belongs to a device implemented by drivers/net/tun.c, in IFF_TUN as well as in IFF_TAP mode, and that the tfile exists, dereferences the tun_struct under RCU, and delegates to __tun_wake_queue(). vhost_net_buf_produce() now calls tun_wake_queue() after a successful batched consume of the ring to allow the netdev subqueue to be woken up. The point is to allow the queue to be stopped when it gets full, which is required for traffic shaping, implemented by the following "stop tail-drop when IFF_BACKPRESSURE is set". As __tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, a tun/tap device that does not opt in only pays for the added check. macvtap and ipvtap rings, which get_tap_ptr_ring() accepts too, are unaffected: their producer is the tap_handle_frame() rx_handler and not ndo_start_xmit, so stopping a netdev TX queue would not hold it back. drivers/net/tap.c has no netdev_ops of its own either. No tap_wake_queue() is needed. cons_cnt and the wake decision are best-effort and are not reverted by ptr_ring_unconsume(), so vhost_net_buf_unproduce() can leave the subqueue woken over a full ring. The producer re-stops it on the next packet, and that path only runs from vhost_net_stop_vq() and vhost_net_set_backend(), when the consumer is going away, so a stopped queue is the correct end state rather than a stall. Co-developed-by: Tim Gebauer Signed-off-by: Tim Gebauer Signed-off-by: Simon Schippers Link: https://patch.msgid.link/20260803183641.96882-4-simon.schippers@tu-dortmund.de Signed-off-by: Jakub Kicinski --- drivers/net/tun.c | 25 +++++++++++++++++++++++++ drivers/vhost/net.c | 21 +++++++++++++++------ include/linux/if_tun.h | 4 ++++ 3 files changed, 44 insertions(+), 6 deletions(-) diff --git a/drivers/net/tun.c b/drivers/net/tun.c index d49b6bfd104d..dc32566588d1 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -3848,6 +3848,31 @@ struct ptr_ring *tun_get_tx_ring(struct file *file) } EXPORT_SYMBOL_GPL(tun_get_tx_ring); +/* Callers must hold ring.consumer_lock */ +void tun_wake_queue(struct file *file, int consumed) +{ + struct tun_file *tfile; + struct tun_struct *tun; + + if (file->f_op != &tun_fops) + return; + + tfile = file->private_data; + if (!tfile) + return; + + lockdep_assert_held(&tfile->tx_ring.consumer_lock); + + rcu_read_lock(); + + tun = rcu_dereference(tfile->tun); + if (tun) + __tun_wake_queue(tun, tfile, consumed); + + rcu_read_unlock(); +} +EXPORT_SYMBOL_GPL(tun_wake_queue); + module_init(tun_init); module_exit(tun_cleanup); MODULE_DESCRIPTION(DRV_DESCRIPTION); diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 6949b704166d..3e72b9c6af0c 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -176,13 +176,21 @@ static void *vhost_net_buf_consume(struct vhost_net_buf *rxq) return ret; } -static int vhost_net_buf_produce(struct vhost_net_virtqueue *nvq) +static int vhost_net_buf_produce(struct sock *sk, + struct vhost_net_virtqueue *nvq) { + struct file *file = sk->sk_socket->file; struct vhost_net_buf *rxq = &nvq->rxq; rxq->head = 0; - rxq->tail = ptr_ring_consume_batched(nvq->rx_ring, rxq->queue, - VHOST_NET_BATCH); + spin_lock(&nvq->rx_ring->consumer_lock); + rxq->tail = __ptr_ring_consume_batched(nvq->rx_ring, rxq->queue, + VHOST_NET_BATCH); + + if (rxq->tail) + tun_wake_queue(file, rxq->tail); + + spin_unlock(&nvq->rx_ring->consumer_lock); return rxq->tail; } @@ -209,14 +217,15 @@ static int vhost_net_buf_peek_len(void *ptr) return __skb_array_len_with_tag(ptr); } -static int vhost_net_buf_peek(struct vhost_net_virtqueue *nvq) +static int vhost_net_buf_peek(struct sock *sk, + struct vhost_net_virtqueue *nvq) { struct vhost_net_buf *rxq = &nvq->rxq; if (!vhost_net_buf_is_empty(rxq)) goto out; - if (!vhost_net_buf_produce(nvq)) + if (!vhost_net_buf_produce(sk, nvq)) return 0; out: @@ -1004,7 +1013,7 @@ static int peek_head_len(struct vhost_net_virtqueue *rvq, struct sock *sk) unsigned long flags; if (rvq->rx_ring) - return vhost_net_buf_peek(rvq); + return vhost_net_buf_peek(sk, rvq); spin_lock_irqsave(&sk->sk_receive_queue.lock, flags); head = skb_peek(&sk->sk_receive_queue); diff --git a/include/linux/if_tun.h b/include/linux/if_tun.h index 80166eb62f41..eeb9ed3c5a23 100644 --- a/include/linux/if_tun.h +++ b/include/linux/if_tun.h @@ -22,6 +22,8 @@ struct tun_msg_ctl { #if defined(CONFIG_TUN) || defined(CONFIG_TUN_MODULE) struct socket *tun_get_socket(struct file *); struct ptr_ring *tun_get_tx_ring(struct file *file); +/* Callers must hold the consumer_lock of the ring of file */ +void tun_wake_queue(struct file *file, int consumed); static inline bool tun_is_xdp_frame(void *ptr) { @@ -55,6 +57,8 @@ static inline struct ptr_ring *tun_get_tx_ring(struct file *f) return ERR_PTR(-EINVAL); } +static inline void tun_wake_queue(struct file *f, int consumed) {} + static inline bool tun_is_xdp_frame(void *ptr) { return false; From 43be21ec2efdeb53d5e46ae77b3f7ce91539a5d9 Mon Sep 17 00:00:00 2001 From: Simon Schippers Date: Mon, 3 Aug 2026 20:36:40 +0200 Subject: [PATCH 4/5] ptr_ring: move free-space check into separate helper This patch moves the check for available free space for a new entry into a separate function. Existing callers that only check for a non-zero return value are unaffected. __ptr_ring_produce() now returns -EINVAL for a zero-size ring and -ENOSPC when full, whereas before both cases returned -ENOSPC. The new helper allows callers to determine in advance whether a single subsequent __ptr_ring_produce() call will succeed. This information can, for example, be used to temporarily stop producing until __ptr_ring_check_produce() indicates that space is available again. The return values are documented above the helper, as a caller that waits for space must distinguish the transient -ENOSPC from the permanent -EINVAL. Co-developed-by: Tim Gebauer Signed-off-by: Tim Gebauer Signed-off-by: Simon Schippers Link: https://patch.msgid.link/20260803183641.96882-5-simon.schippers@tu-dortmund.de Signed-off-by: Jakub Kicinski --- include/linux/ptr_ring.h | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/include/linux/ptr_ring.h b/include/linux/ptr_ring.h index d2c3629bbe45..631c43fde440 100644 --- a/include/linux/ptr_ring.h +++ b/include/linux/ptr_ring.h @@ -96,6 +96,26 @@ static inline bool ptr_ring_full_bh(struct ptr_ring *r) return ret; } +/* Report whether the next __ptr_ring_produce() has room for one entry: + * 0 means the single slot at r->queue[r->producer] is free, -ENOSPC means + * the ring is full, which is transient, and -EINVAL means r->size is 0, + * which is permanent. A caller that stops producing and waits for space + * must therefore do so only for -ENOSPC. + * + * Note: callers invoking this in a loop must use a compiler barrier, + * for example cpu_relax(). Callers must hold producer_lock. + */ +static inline int __ptr_ring_check_produce(struct ptr_ring *r) +{ + if (unlikely(!r->size)) + return -EINVAL; + + if (data_race(r->queue[r->producer])) + return -ENOSPC; + + return 0; +} + /* Note: callers invoking this in a loop must use a compiler barrier, * for example cpu_relax(). Callers must hold producer_lock. * Callers are responsible for making sure pointer that is being queued @@ -103,8 +123,10 @@ static inline bool ptr_ring_full_bh(struct ptr_ring *r) */ static inline int __ptr_ring_produce(struct ptr_ring *r, void *ptr) { - if (unlikely(!r->size) || data_race(r->queue[r->producer])) - return -ENOSPC; + int ret = __ptr_ring_check_produce(r); + + if (ret) + return ret; /* Make sure the pointer we are storing points to a valid data. */ /* Pairs with the dependency ordering in __ptr_ring_consume. */ From d00c7369ef24ac8e0383de0fd8ef3384de20bcfa Mon Sep 17 00:00:00 2001 From: Simon Schippers Date: Mon, 3 Aug 2026 20:36:41 +0200 Subject: [PATCH 5/5] tun/tap & vhost-net: stop tail-drop when IFF_BACKPRESSURE is set This commit prevents tail-drop when IFF_BACKPRESSURE is set, a qdisc is present and the ptr_ring becomes full. Once the ring reaches capacity after a produce attempt, the netdev queue is stopped instead of dropping subsequent packets. Without the flag, or if no qdisc is present, the previous tail-drop behavior is preserved. IFF_BACKPRESSURE is added to TUN_FEATURES here and not in the patch that defines it, so that TUNSETIFF honours the flag only once the implementation behind it is complete. The unconditional version of this behavior was reverted because it caused a significant throughput drop in an IPv6 multicast testcase on Brett Sheffield's librecast testbed [1]: with 8 iperf3 TCP threads sending, the throughput dropped from 13.5 Gbit/s to 9.13 Gbit/s. This is why the queue stopping is now gated on IFF_BACKPRESSURE. If producing an entry fails anyway due to a race, tun_net_xmit() drops the packet. Such rare races are expected because LLTX is enabled and the transmit path operates without the usual locking. The queue state is only touched while the device is running. The stop itself would be harmless during teardown, as tun_net_close() sets the same bit, but the re-check below it wakes the queue again and must not clear that stop. A later TUNSETIFF can clear the flag again while the device has at most one queue. Past that point tun_set_iff() returns before it writes tun->flags, which is how it already treats every other TUN_FEATURES bit. For the case where the flag does change, tun_set_iff() calls tun_force_wake_queue() for the attached tfiles, so that no queue stays stopped without a consumer that would wake it. The __tun_wake_queue() function of the consumer races with the producer for waking/stopping the netdev queue, which could result in a stalled queue. Therefore, an smp_mb__after_atomic() is introduced that pairs with the smp_mb() of the consumer. It follows the principle of store buffering described in tools/memory-model/Documentation/recipes.txt: - The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF, followed by an smp_mb__after_atomic() (= smp_mb()), and then reads the ring with __ptr_ring_check_produce(). - The consumer in __tun_wake_queue() first writes zero to the ring in __ptr_ring_consume(), followed by an smp_mb(), and then reads the queue status with netif_tx_queue_stopped(). => Following the aforementioned principle, it is impossible for the producer to see a full ring (and therefore not wake the queue on the re-check) while the consumer simultaneously fails to see a stopped queue (and therefore also does not wake it). tun_net_xmit() holds only the producer_lock and can not reset cons_cnt, which the consumer_lock protects, so the wake on the re-check leaves stale credit behind. That is accepted as best-effort, the re-check rarely succeeds and the next drain corrects the count. The documentation in tuntap.rst is updated accordingly. Benchmarks: My own benchmarks show a slight regression in raw transmission performance when using two sending threads. Packet loss also occurs only in the two-thread sending case; no packet loss was observed with a single sending thread. Test setup: AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads; Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2 mitigations disabled. Note for tap+vhost-net: XDP drop program active in VM -> ~2.5x faster; slower for tap due to more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf) +--------------------------+--------------+----------------+----------+ | 1 thread | Stock | Patched with | diff | | sending | | fq_codel qdisc | | +------------+-------------+--------------+----------------+----------+ | TAP | Received | 1.132 Mpps | 1.123 Mpps | -0.8% | | +-------------+--------------+----------------+----------+ | | Lost/s | 3.765 Mpps | 0 pps | | +------------+-------------+--------------+----------------+----------+ | TAP | Received | 3.857 Mpps | 3.901 Mpps | +1.1% | | +-------------+--------------+----------------+----------+ | +vhost-net | Lost/s | 0.802 Mpps | 0 pps | | +------------+-------------+--------------+----------------+----------+ +--------------------------+--------------+----------------+----------+ | 2 threads | Stock | Patched with | diff | | sending | | fq_codel qdisc | | +------------+-------------+--------------+----------------+----------+ | TAP | Received | 1.115 Mpps | 1.081 Mpps | -3.0% | | +-------------+--------------+----------------+----------+ | | Lost/s | 8.490 Mpps | 391 pps | | +------------+-------------+--------------+----------------+----------+ | TAP | Received | 3.664 Mpps | 3.555 Mpps | -3.0% | | +-------------+--------------+----------------+----------+ | +vhost-net | Lost/s | 5.330 Mpps | 938 pps | | +------------+-------------+--------------+----------------+----------+ [1] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/ Co-developed-by: Tim Gebauer Signed-off-by: Tim Gebauer Signed-off-by: Simon Schippers Link: https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/ Link: https://patch.msgid.link/20260803183641.96882-6-simon.schippers@tu-dortmund.de Signed-off-by: Jakub Kicinski --- Documentation/networking/tuntap.rst | 32 +++++++++++++++++++++++ drivers/net/tun.c | 39 ++++++++++++++++++++++++----- 2 files changed, 65 insertions(+), 6 deletions(-) diff --git a/Documentation/networking/tuntap.rst b/Documentation/networking/tuntap.rst index 4d7087f727be..56c9dc7b96af 100644 --- a/Documentation/networking/tuntap.rst +++ b/Documentation/networking/tuntap.rst @@ -206,6 +206,38 @@ enable is true we enable it, otherwise we disable it:: return ioctl(fd, TUNSETQUEUE, (void *)&ifr); } +3.4 qdisc backpressure +---------------------- + +IFF_BACKPRESSURE can be set to enable qdisc backpressure. Without it, TX +drops occur when the internal ring buffer is full, so any attached qdisc +is effectively bypassed and applications only learn about congestion +through those drops. + +With it, the kernel stops the queue instead, letting the qdisc hold and +schedule packets, so its AQM, shaping and fairness actually apply. This +helps protocols like TCP, which cut throughput in reaction to packet +drops. With IFF_BACKPRESSURE, drops then only occur as a rare race. +Backpressure requires a qdisc to be attached and has no effect with +noqueue. + +The flag is a property of the TUN/TAP device rather than of the file +descriptor it was set on, so it applies to all queues of the device, +regardless of which process opened which queue. + +The flag can only be changed while the device has at most one queue. On a +multiqueue device that already has a second queue attached or detached, a +later TUNSETIFF succeeds but leaves the flag as it is. All the other +TUNSETIFF flags behave the same way. + +The txqueuelen can be reduced alongside this flag to further shift +buffering into the qdisc and reduce bufferbloat, at a possible +performance cost. + +When running multiple network streams in parallel through a single +TUN/TAP queue, the flag may reduce performance due to the extra overhead +of the backpressure mechanism. + Universal TUN/TAP device driver Frequently Asked Question ========================================================= diff --git a/drivers/net/tun.c b/drivers/net/tun.c index dc32566588d1..ec90fef4a42f 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -98,7 +98,8 @@ static void tun_default_link_ksettings(struct net_device *dev, #define TUN_FASYNC IFF_ATTACH_QUEUE #define TUN_FEATURES (IFF_NO_PI | IFF_ONE_QUEUE | IFF_VNET_HDR | \ - IFF_MULTI_QUEUE | IFF_NAPI | IFF_NAPI_FRAGS) + IFF_MULTI_QUEUE | IFF_NAPI | IFF_NAPI_FRAGS | \ + IFF_BACKPRESSURE) #define GOODCOPY_LEN 128 @@ -1063,6 +1064,7 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev) struct netdev_queue *queue; struct tun_file *tfile; int len = skb->len; + int ret; rcu_read_lock(); tfile = rcu_dereference(tun->tfiles[txq]); @@ -1117,13 +1119,35 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev) nf_reset_ct(skb); - if (ptr_ring_produce(&tfile->tx_ring, skb)) { + queue = netdev_get_tx_queue(dev, txq); + + spin_lock(&tfile->tx_ring.producer_lock); + ret = __ptr_ring_produce(&tfile->tx_ring, skb); + /* Do not touch the queue state of a device that is going down. */ + if ((tun->flags & IFF_BACKPRESSURE) && netif_running(dev) && + !qdisc_txq_has_no_queue(queue) && + __ptr_ring_check_produce(&tfile->tx_ring) == -ENOSPC) { + netif_tx_stop_queue(queue); + /* Paired with smp_mb() in __tun_wake_queue() */ + smp_mb__after_atomic(); + if (!__ptr_ring_check_produce(&tfile->tx_ring)) + netif_tx_wake_queue(queue); + } + spin_unlock(&tfile->tx_ring.producer_lock); + + if (ret) { + /* This should be a rare case if IFF_BACKPRESSURE is enabled and + * a qdisc is present, but can happen due to lltx. + * Since skb_tx_timestamp(), skb_orphan(), + * run_ebpf_filter() and pskb_trim() could have tinkered + * with the SKB, returning NETDEV_TX_BUSY is unsafe and + * we must drop instead. + */ drop_reason = SKB_DROP_REASON_FULL_RING; goto drop; } /* dev->lltx requires to do our own update of trans_start */ - queue = netdev_get_tx_queue(dev, txq); txq_trans_cond_update(queue); /* Notify and wake up reader process */ @@ -2806,8 +2830,9 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr) { struct tun_struct *tun; struct tun_file *tfile = file->private_data; + struct tun_file *ntfile; struct net_device *dev; - int err; + int err, i; if (tfile->detached) return -EINVAL; @@ -2936,8 +2961,10 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr) /* Make sure persistent devices do not get stuck in * xoff state. */ - if (netif_running(tun->dev)) - netif_tx_wake_all_queues(tun->dev); + for (i = 0; i < tun->numqueues; i++) { + ntfile = rtnl_dereference(tun->tfiles[i]); + tun_force_wake_queue(tun, ntfile); + } strscpy(ifr->ifr_name, tun->dev->name); return 0;