Liam R. Howlett
a2ac9935d3
maple_tree: remove unnecessary assignment of orig_l index
...
The index value is already a copy of the maple state so there is no need
to set it again.
Link: https://lkml.kernel.org/r/20260130205935.2559335-5-Liam.Howlett@oracle.com
Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com >
Cc: Alice Ryhl <aliceryhl@google.com >
Cc: Andrew Ballance <andrewjballance@gmail.com >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: Christian Kujau <lists@nerdbynature.de >
Cc: Geert Uytterhoeven <geert@linux-m68k.org >
Cc: Kuninori Morimoto <kuninori.morimoto.gx@renesas.com >
Cc: Matthew Wilcox (Oracle) <willy@infradead.org >
Cc: SeongJae Park <sj@kernel.org >
Cc: Sidhartha Kumar <sidhartha.kumar@oracle.com >
Cc: Suren Baghdasaryan <surenb@google.com >
Cc: Vlastimil Babka <vbabka@suse.cz >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-05 13:52:53 -07:00
Liam R. Howlett
df11f9ee8f
maple_tree: extract use of big node from mas_wr_spanning_store()
...
Isolate big node to use in its own function.
No functional changes intended.
Link: https://lkml.kernel.org/r/20260130205935.2559335-4-Liam.Howlett@oracle.com
Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com >
Cc: Alice Ryhl <aliceryhl@google.com >
Cc: Andrew Ballance <andrewjballance@gmail.com >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: Christian Kujau <lists@nerdbynature.de >
Cc: Geert Uytterhoeven <geert@linux-m68k.org >
Cc: Kuninori Morimoto <kuninori.morimoto.gx@renesas.com >
Cc: Matthew Wilcox (Oracle) <willy@infradead.org >
Cc: SeongJae Park <sj@kernel.org >
Cc: Sidhartha Kumar <sidhartha.kumar@oracle.com >
Cc: Suren Baghdasaryan <surenb@google.com >
Cc: Vlastimil Babka <vbabka@suse.cz >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-05 13:52:53 -07:00
Liam R. Howlett
3e302560b9
maple_tree: move mas_spanning_rebalance loop to function
...
Move the loop over the tree levels to its own function.
No intended functional changes.
Link: https://lkml.kernel.org/r/20260130205935.2559335-3-Liam.Howlett@oracle.com
Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com >
Cc: Alice Ryhl <aliceryhl@google.com >
Cc: Andrew Ballance <andrewjballance@gmail.com >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: Christian Kujau <lists@nerdbynature.de >
Cc: Geert Uytterhoeven <geert@linux-m68k.org >
Cc: Kuninori Morimoto <kuninori.morimoto.gx@renesas.com >
Cc: Matthew Wilcox (Oracle) <willy@infradead.org >
Cc: SeongJae Park <sj@kernel.org >
Cc: Sidhartha Kumar <sidhartha.kumar@oracle.com >
Cc: Suren Baghdasaryan <surenb@google.com >
Cc: Vlastimil Babka <vbabka@suse.cz >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-05 13:52:53 -07:00
Liam R. Howlett
6884832472
maple_tree: fix mas_dup_alloc() sparse warning
...
Patch series "maple_tree: Replace big node with maple copy", v3.
The big node struct was created for simplicity of splitting, rebalancing,
and spanning store operations by using a copy buffer to create the data
necessary prior to breaking it up into 256B nodes. Certain operations
were rather tricky due to the restriction of keeping NULL entries together
and never at the end of a node (except the right-most node).
The big node struct is incompatible with future features that are
currently in development. Specifically different node types and different
data type sizes for pivots.
The big node struct was also a stack variable, which caused issues with
certain configurations of kernel build.
This series removes big node by introducing another node type which will
never be written to the tree: maple_copy. The maple copy node operates
more like a scatter/gather operation with a number of sources and
destinations of allocated nodes.
The sources are copied to the destinations, in turn, until the sources are
exhausted. The destination is changed if it is filled or the split
location is reached prior to the source data end.
New data is inserted by using the maple copy node itself as a source with
up to 3 slots and pivots. The data in the maple copy node is the data
being written to the tree along with any fragment of the range(s) being
overwritten.
As with all nodes, the maple copy node is of size 256B. Using a node type
allows for the copy operation to treat the new data stored in the maple
copy node the same as any other source node.
Analysis of the runtime shows no regression or benefit of removing the
larger stack structure. The motivation is the ground work to use new node
types and to help those with odd configurations that have had issues.
The change was tested by myself using mm_tests on amd64 and by Suren on
android (arm64). Limited testing on s390 qemu was also performed using
stress-ng on the virtual memory, which should cover many corner cases.
This patch (of 30):
Use RCU_INIT_POINTER to initialize an rcu pointer to an initial value
since there are no readers within the tree being created during
duplication. There is no risk of readers seeing the initialized or
uninitialized value until after the synchronization call in
mas_dup_buld().
Link: https://lkml.kernel.org/r/20260130205935.2559335-1-Liam.Howlett@oracle.com
Link: https://lkml.kernel.org/r/20260130205935.2559335-2-Liam.Howlett@oracle.com
Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com >
Cc: Alice Ryhl <aliceryhl@google.com >
Cc: Andrew Ballance <andrewjballance@gmail.com >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: Christian Kujau <lists@nerdbynature.de >
Cc: Geert Uytterhoeven <geert@linux-m68k.org >
Cc: Kuninori Morimoto <kuninori.morimoto.gx@renesas.com >
Cc: Matthew Wilcox (Oracle) <willy@infradead.org >
Cc: SeongJae Park <sj@kernel.org >
Cc: Sidhartha Kumar <sidhartha.kumar@oracle.com >
Cc: Suren Baghdasaryan <surenb@google.com >
Cc: Vlastimil Babka <vbabka@suse.cz >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-05 13:52:53 -07:00
Feng Jiang
e73bcb3708
lib/string_kunit: extend benchmarks to strnlen() and chr searches
...
Extend the string benchmarking suite to include strnlen(), strchr(),
and strrchr().
For character search functions strchr() and strrchr(), the benchmark
targets the NUL character. This ensures the entire string is scanned,
providing a consistent measure of full-length processing efficiency
comparable to strlen().
Suggested-by: Andy Shevchenko <andy@kernel.org >
Suggested-by: Eric Biggers <ebiggers@kernel.org >
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn >
Acked-by: Andy Shevchenko <andy@kernel.org >
Reviewed-by: Kees Cook <kees@kernel.org >
Link: https://patch.msgid.link/20260130025018.172925-6-jiangfeng@kylinos.cn
Signed-off-by: Paul Walmsley <pjw@kernel.org >
2026-04-04 18:42:44 -06:00
Feng Jiang
0020240a43
lib/string_kunit: add performance benchmark for strlen()
...
Introduce a benchmarking framework to the string_kunit test suite to
measure the execution efficiency of string functions.
The implementation is inspired by crc_benchmark(), measuring throughput
(MB/s) and latency (ns/call) across a range of string lengths. It
includes a warm-up phase, disables preemption during measurement, and
uses a fixed seed for reproducible results.
This framework allows for comparing different implementations (e.g.,
generic C vs. architecture-optimized assembly) within the KUnit
environment.
Initially, provide a benchmark for strlen().
Suggested-by: Andy Shevchenko <andy@kernel.org >
Suggested-by: Eric Biggers <ebiggers@kernel.org >
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn >
Reviewed-by: Kees Cook <kees@kernel.org >
Link: https://patch.msgid.link/20260130025018.172925-5-jiangfeng@kylinos.cn
[pjw@kernel.org: fixed a checkpatch issue]
Signed-off-by: Paul Walmsley <pjw@kernel.org>
2026-04-04 18:42:44 -06:00
Feng Jiang
27b2810a4a
lib/string_kunit: add correctness test for strrchr()
...
Add a KUnit test for strrchr() to verify correctness across
different string lengths and memory alignments. Use vmalloc()
to place the NUL character at the page boundary to ensure
over-reads are detected.
Suggested-by: Kees Cook <kees@kernel.org >
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn >
Reviewed-by: Kees Cook <kees@kernel.org >
Link: https://patch.msgid.link/20260130025018.172925-4-jiangfeng@kylinos.cn
Signed-off-by: Paul Walmsley <pjw@kernel.org >
2026-04-04 18:42:44 -06:00
Feng Jiang
263dca234e
lib/string_kunit: add correctness test for strnlen()
...
Add a KUnit test for strnlen() to verify correctness across
different string lengths and memory alignments. Use vmalloc()
to place the NUL character at the page boundary to ensure
over-reads are detected.
Suggested-by: Andy Shevchenko <andy@kernel.org >
Suggested-by: Kees Cook <kees@kernel.org >
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn >
Reviewed-by: Kees Cook <kees@kernel.org >
Link: https://patch.msgid.link/20260130025018.172925-3-jiangfeng@kylinos.cn
Signed-off-by: Paul Walmsley <pjw@kernel.org >
2026-04-04 18:42:44 -06:00
Feng Jiang
ae45f896a4
lib/string_kunit: add correctness test for strlen()
...
Add a KUnit test for strlen() to verify correctness across
different string lengths and memory alignments. Use vmalloc()
to place the NUL character at the page boundary to ensure
over-reads are detected.
Suggested-by: Kees Cook <kees@kernel.org >
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn >
Reviewed-by: Kees Cook <kees@kernel.org >
Link: https://patch.msgid.link/20260130025018.172925-2-jiangfeng@kylinos.cn
Signed-off-by: Paul Walmsley <pjw@kernel.org >
2026-04-04 18:42:44 -06:00
Arnd Bergmann
c6890f36fc
workqueue: avoid unguarded 64-bit division
...
The printk() requires a division that is not allowed on 32-bit architectures:
x86_64-linux-ld: lib/test_workqueue.o: in function `test_workqueue_init':
test_workqueue.c:(.init.text+0x36f): undefined reference to `__udivdi3'
Use div_u64() to print the resulting elapsed microseconds.
Fixes: 24b2e73f97arnd@arndb.de >
Signed-off-by: Tejun Heo <tj@kernel.org >
2026-04-03 07:14:24 -10:00
Inseob Kim
d01684a2f0
lib: parser: fix match_wildcard to correctly handle trailing stars
...
This fixes a bug in match_wildcard that incorrectly handles trailing
asterisks. For example, `match_wildcard("abc**", "abc")` must return
true, but it returns false.
Link: https://lkml.kernel.org/r/20260326020630.4139520-1-inseob@google.com
Signed-off-by: Inseob Kim <inseob@google.com >
Cc: Changbin Du <changbin.du@gmail.com >
Cc: Jason Baron <jbaron@akamai.com >
Cc: Joe Perches <joe@perches.com >
Cc: Josh Law <objecting@objecting.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:24 -07:00
Christian A. Ehrhardt
0913b75547
lib: kunit_iov_iter: add tests for extract_iter_to_sg
...
Add test cases that test extract_iter_to_sg.
For each iterator type an iterator is loaded with a suitable buffer. The
iterator is then extracted to a scatterlist with multiple calls to
extract_iter_to_sg. The final scatterlist is copied into a scratch
buffer.
The test passes if the scratch buffer contains the same data as the
original buffer.
The new tests demonstrate bugs in extract_iter_to_sg for kvec and user
iterators that are fixed by the previous commits.
Link: https://lkml.kernel.org/r/20260326214905.818170-6-lk@c--e.de
Signed-off-by: Christian A. Ehrhardt <lk@c--e.de >
Cc: David Howells <dhowells@redhat.com >
Cc: David Gow <davidgow@google.com >
Cc: Kees Cook <kees@kernel.org >
Cc: Petr Mladek <pmladek@suse.com >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:23 -07:00
Christian A. Ehrhardt
7278aa840b
lib: kunit_iov_iter: improve error detection
...
In the kunit_iov_iter test prevent the kernel buffer from being a single
physically contiguous region.
Additionally, make sure that the test pattern written to a page in the
buffer depends on the offset of the page within the buffer.
Link: https://lkml.kernel.org/r/20260326214905.818170-5-lk@c--e.de
Signed-off-by: Christian A. Ehrhardt <lk@c--e.de >
Cc: David Howells <dhowells@redhat.com >
Cc: David Gow <davidgow@google.com >
Cc: Kees Cook <kees@kernel.org >
Cc: Petr Mladek <pmladek@suse.com >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:23 -07:00
Christian A. Ehrhardt
0b49c7d0ae
lib: kunit_iov_iter: fix memory leaks
...
Use vfree() instead of vunmap() to free the buffer allocated by
iov_kunit_create_buffer() because vunmap() does not honour
VM_MAP_PUT_PAGES. In order for this to work the page array itself must
not be managed by kunit.
Remove the folio_put() when destroying a folioq. This is handled by
vfree(), now.
Pointed out by sashiko.dev on a previous iteration of this series.
Tested by running the kunit test 10000 times in a loop.
Link: https://lkml.kernel.org/r/20260326214905.818170-4-lk@c--e.de
Fixes: 2d71340ff1lk@c--e.de >
Cc: David Howells <dhowells@redhat.com >
Cc: David Gow <davidgow@google.com >
Cc: Kees Cook <kees@kernel.org >
Cc: Petr Mladek <pmladek@suse.com >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:23 -07:00
Christian A. Ehrhardt
118cf3f559
lib/scatterlist: fix temp buffer in extract_user_to_sg()
...
Instead of allocating a temporary buffer for extracted user pages
extract_user_to_sg() uses the end of the to be filled scatterlist as a
temporary buffer.
Fix the calculation of the start address if the scatterlist already
contains elements. The unused space starts at sgtable->sgl +
sgtable->nents not directly at sgtable->nents and the temporary buffer is
placed at the end of this unused space.
A subsequent commit will add kunit test cases that demonstrate that the
patch is necessary.
Pointed out by sashiko.dev on a previous iteration of this series.
Link: https://lkml.kernel.org/r/20260326214905.818170-3-lk@c--e.de
Fixes: 0185846975lk@c--e.de >
Cc: David Howells <dhowells@redhat.com >
Cc: David Gow <davidgow@google.com >
Cc: Kees Cook <kees@kernel.org >
Cc: Petr Mladek <pmladek@suse.com >
Cc: <stable@vger.kernel.org > [v6.5+]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:23 -07:00
Christian A. Ehrhardt
07b7d66e65
lib/scatterlist: fix length calculations in extract_kvec_to_sg
...
Patch series "Fix bugs in extract_iter_to_sg()", v3.
Fix bugs in the kvec and user variants of extract_iter_to_sg. This series
is growing due to useful remarks made by sashiko.dev.
The main bugs are:
- The length for an sglist entry when extracting from
a kvec can exceed the number of bytes in the page. This
is obviously not intended.
- When extracting a user buffer the sglist is temporarily
used as a scratch buffer for extracted page pointers.
If the sglist already contains some elements this scratch
buffer could overlap with existing entries in the sglist.
The series adds test cases to the kunit_iov_iter test that demonstrate all
of these bugs. Additionally, there is a memory leak fix for the test
itself.
The bugs were orignally introduced into kernel v6.3 where the function
lived in fs/netfs/iterator.c. It was later moved to lib/scatterlist.c in
v6.5. Thus the actual fix is only marked for backports to v6.5+.
This patch (of 5):
When extracting from a kvec to a scatterlist, do not cross page
boundaries. The required length was already calculated but not used as
intended.
Adjust the copied length if the loop runs out of sglist entries without
extracting everything.
While there, return immediately from extract_iter_to_sg if there are no
sglist entries at all.
A subsequent commit will add kunit test cases that demonstrate that the
patch is necessary.
Link: https://lkml.kernel.org/r/20260326214905.818170-1-lk@c--e.de
Link: https://lkml.kernel.org/r/20260326214905.818170-2-lk@c--e.de
Fixes: 0185846975lk@c--e.de >
Cc: David Gow <davidgow@google.com >
Cc: David Howells <dhowells@redhat.com >
Cc: Kees Cook <kees@kernel.org >
Cc: Petr Mladek <pmladek@suse.com >
Cc: <stable@vger.kernel.org > [v6.5+]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:22 -07:00
Kuan-Wei Chiu
86bda539fb
lib/list_sort: remove dummy cmp() calls to speed up merge_final()
...
Historically, list_sort() implemented a hack in merge_final():
if (unlikely(!++count))
cmp(priv, b, b);
This was introduced 16 years ago in commit 835cc0c847https://lkml.kernel.org/r/20260320180938.1827148-3-visitorckw@gmail.com
Signed-off-by: Kuan-Wei Chiu <visitorckw@gmail.com >
Reviewed-by: Christoph Hellwig <hch@lst.de >
Cc: Ching-Chun (Jim) Huang <jserv@ccns.ncku.edu.tw >
Cc: Mars Cheng <marscheng@google.com >
Cc: Richard Weinberger <richard@nod.at >
Cc: Yu-Chun Lin <eleanor15x@gmail.com >
Cc: Zhihao Cheng <chengzhihao1@huawei.com >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:22 -07:00
Christoph Hellwig
af53e85ef7
xor: add a kunit test case
...
Add a test case for the XOR routines loosely based on the CRC kunit
test.
Link: https://lkml.kernel.org/r/20260327061704.3707577-29-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:22 -07:00
Christoph Hellwig
a21921dd02
xor: use static_call for xor_gen
...
Avoid the indirect call for xor_generation by using a static_call.
Link: https://lkml.kernel.org/r/20260327061704.3707577-28-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:21 -07:00
Christoph Hellwig
80dcf0a783
xor: pass the entire operation to the low-level ops
...
Currently the high-level xor code chunks up all operations into small
units for only up to 1 + 4 vectors, and passes it to four different
methods. This means the FPU/vector context is entered and left a lot for
wide stripes, and a lot of indirect expensive indirect calls are
performed. Switch to passing the entire gen_xor request to the low-level
ops, and provide a macro to dispatch it to the existing helper.
This reduce the number of indirect calls and FPU/vector context switches
by a factor approaching nr_stripes / 4, and also reduces source and binary
code size.
Link: https://lkml.kernel.org/r/20260327061704.3707577-27-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:21 -07:00
Christoph Hellwig
e420f0a88b
xor: add a better public API
...
xor_blocks is very annoying to use, because it is limited to 4 + 1 sources
/ destinations, has an odd argument order and is completely undocumented.
Lift the code that loops around it from btrfs and async_tx/async_xor into
common code under the name xor_gen and properly document it.
[hch@lst.de: make xor_blocks less annoying to use]
Link: https://lkml.kernel.org/r/20260327061704.3707577-24-hch@lst.de
Link: https://lkml.kernel.org/r/20260327061704.3707577-23-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:20 -07:00
Christoph Hellwig
e20043b476
xor: make xor.ko self-contained in lib/raid/
...
Move the asm/xor.h headers to lib/raid/xor/$(SRCARCH)/xor_arch.h and
include/linux/raid/xor_impl.h to lib/raid/xor/xor_impl.h so that the
xor.ko module implementation is self-contained in lib/raid/.
As this remove the asm-generic mechanism a new kconfig symbol is added to
indicate that a architecture-specific implementations exists, and
xor_arch.h should be included.
Link: https://lkml.kernel.org/r/20260327061704.3707577-22-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:20 -07:00
Christoph Hellwig
352ebd066b
xor: avoid indirect calls for arm64-optimized ops
...
Remove the inner xor_block_templates, and instead have two separate actual
template that call into the neon-enabled compilation unit.
Link: https://lkml.kernel.org/r/20260327061704.3707577-21-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:20 -07:00
Christoph Hellwig
77fd47e57a
x86: move the XOR code to lib/raid/
...
Move the optimized XOR code out of line into lib/raid.
Link: https://lkml.kernel.org/r/20260327061704.3707577-20-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:19 -07:00
Christoph Hellwig
95c104cc55
s390: move the XOR code to lib/raid/
...
Move the optimized XOR into lib/raid and include it it in xor.ko instead
of unconditionally building it into the main kernel image.
Link: https://lkml.kernel.org/r/20260327061704.3707577-19-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Acked-by: Heiko Carstens <hca@linux.ibm.com >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:19 -07:00
Christoph Hellwig
7f96362396
sparc: move the XOR code to lib/raid/
...
Move the optimized XOR into lib/raid and include it it in xor.ko instead
of always building it into the main kernel image.
The code should probably be split into separate files for the two
implementations, but for now this just does the trivial move.
Link: https://lkml.kernel.org/r/20260327061704.3707577-18-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:19 -07:00
Christoph Hellwig
5265d55b21
riscv: move the XOR code to lib/raid/
...
Move the optimized XOR into lib/raid and include it it in xor.ko instead
of always building it into the main kernel image.
Link: https://lkml.kernel.org/r/20260327061704.3707577-17-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:19 -07:00
Christoph Hellwig
3f276cece4
powerpc: move the XOR code to lib/raid/
...
Move the optimized XOR into lib/raid and include it it in xor.ko instead
of always building it into the main kernel image.
Link: https://lkml.kernel.org/r/20260327061704.3707577-16-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:18 -07:00
Christoph Hellwig
033bee3e49
loongarch: move the XOR code to lib/raid/
...
Move the optimized XOR into lib/raid and include it it in xor.ko instead
of always building it into the main kernel image.
Link: https://lkml.kernel.org/r/20260327061704.3707577-15-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:18 -07:00
Christoph Hellwig
3786f2ad00
arm64: move the XOR code to lib/raid/
...
Move the optimized XOR into lib/raid and include it it in the main xor.ko
instead of building a separate module for it.
Note that this drops the CONFIG_KERNEL_MODE_NEON dependency, as that is
always set for arm64.
Link: https://lkml.kernel.org/r/20260327061704.3707577-14-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:18 -07:00
Christoph Hellwig
0d64a24ec0
arm: move the XOR code to lib/raid/
...
Move the optimized XOR into lib/raid and include it it in the main xor.ko
instead of building a separate module for it.
Link: https://lkml.kernel.org/r/20260327061704.3707577-13-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:18 -07:00
Christoph Hellwig
503793b134
alpha: move the XOR code to lib/raid/
...
Move the optimized XOR code out of line into lib/raid.
Note that the giant inline assembly block might be better off as a
separate assembly source file now, but I'll leave that to the alpha
maintainers.
Link: https://lkml.kernel.org/r/20260327061704.3707577-12-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Acked-by: Magnus Lindholm <linmag7@gmail.com >
Tested-by: Magnus Lindholm <linmag7@gmail.com >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:17 -07:00
Christoph Hellwig
c46928fdcf
xor: move generic implementations out of asm-generic/xor.h
...
Move the generic implementations from asm-generic/xor.h to
per-implementaion .c files in lib/raid. This will build them
unconditionally even when an architecture forces a specific
implementation, but as we'll need at least one generic version for the
static_call optimization later on we'll pay that price.
Note that this would cause the second xor_block_8regs instance created by
arch/arm/lib/xor-neon.c to be generated instead of discarded as dead code,
so add a NO_TEMPLATE symbol to disable it for this case.
Link: https://lkml.kernel.org/r/20260327061704.3707577-11-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:17 -07:00
Christoph Hellwig
35ebc4de10
xor: remove macro abuse for XOR implementation registrations
...
Drop the pretty confusing historic XOR_TRY_TEMPLATES and
XOR_SELECT_TEMPLATE, and instead let the architectures provide a
arch_xor_init that calls either xor_register to register candidates or
xor_force to force a specific implementation.
Link: https://lkml.kernel.org/r/20260327061704.3707577-10-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:17 -07:00
Christoph Hellwig
54e20be48f
xor: split xor.h
...
Keep xor.h for the public API, and split the struct xor_block_template
definition that is only needed by the xor.ko core and
architecture-specific optimizations into a separate xor_impl.h header.
Link: https://lkml.kernel.org/r/20260327061704.3707577-9-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:17 -07:00
Christoph Hellwig
0471415f3f
xor: cleanup registration and probing
...
Originally, the XOR code benchmarked all algorithms at load time, but it
has since then been hacked multiple times to allow forcing an algorithm,
and then commit 524ccdbdfbhttps://lkml.kernel.org/r/20260327061704.3707577-8-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:16 -07:00
Christoph Hellwig
7c6e6b2b48
xor: small cleanups
...
Update the to of file comment to be correct and non-redundant, and drop
the unused BH_TRACE define.
Link: https://lkml.kernel.org/r/20260327061704.3707577-7-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:16 -07:00
Christoph Hellwig
9e229025e2
xor: move to lib/raid/
...
Move the RAID XOR code to lib/raid/ as it has nothing to do with the
crypto API.
Link: https://lkml.kernel.org/r/20260327061704.3707577-6-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de >
Reviewed-by: Eric Biggers <ebiggers@kernel.org >
Tested-by: Eric Biggers <ebiggers@kernel.org >
Cc: Albert Ou <aou@eecs.berkeley.edu >
Cc: Alexander Gordeev <agordeev@linux.ibm.com >
Cc: Alexandre Ghiti <alex@ghiti.fr >
Cc: Andreas Larsson <andreas@gaisler.com >
Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com >
Cc: Ard Biesheuvel <ardb@kernel.org >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: "Borislav Petkov (AMD)" <bp@alien8.de >
Cc: Catalin Marinas <catalin.marinas@arm.com >
Cc: Chris Mason <clm@fb.com >
Cc: Christian Borntraeger <borntraeger@linux.ibm.com >
Cc: Dan Williams <dan.j.williams@intel.com >
Cc: David S. Miller <davem@davemloft.net >
Cc: David Sterba <dsterba@suse.com >
Cc: Heiko Carstens <hca@linux.ibm.com >
Cc: Herbert Xu <herbert@gondor.apana.org.au >
Cc: "H. Peter Anvin" <hpa@zytor.com >
Cc: Huacai Chen <chenhuacai@kernel.org >
Cc: Ingo Molnar <mingo@redhat.com >
Cc: Jason A. Donenfeld <jason@zx2c4.com >
Cc: Johannes Berg <johannes@sipsolutions.net >
Cc: Li Nan <linan122@huawei.com >
Cc: Madhavan Srinivasan <maddy@linux.ibm.com >
Cc: Magnus Lindholm <linmag7@gmail.com >
Cc: Matt Turner <mattst88@gmail.com >
Cc: Michael Ellerman <mpe@ellerman.id.au >
Cc: Nicholas Piggin <npiggin@gmail.com >
Cc: Palmer Dabbelt <palmer@dabbelt.com >
Cc: Richard Henderson <richard.henderson@linaro.org >
Cc: Richard Weinberger <richard@nod.at >
Cc: Russell King <linux@armlinux.org.uk >
Cc: Song Liu <song@kernel.org >
Cc: Sven Schnelle <svens@linux.ibm.com >
Cc: Ted Ts'o <tytso@mit.edu >
Cc: Vasily Gorbik <gor@linux.ibm.com >
Cc: WANG Xuerui <kernel@xen0n.name >
Cc: Will Deacon <will@kernel.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2026-04-02 23:36:16 -07:00
Ard Biesheuvel
8fdef85d60
lib/crc: arm64: Simplify intrinsics implementation
...
NEON intrinsics are useful because they remove the need for manual
register allocation, and the resulting code can be re-compiled and
optimized for different micro-architectures, and shared between arm64
and 32-bit ARM.
However, the strong typing of the vector variables can lead to
incomprehensible gibberish, as is the case with the new CRC64
implementation. To address this, let's repaint all variables as
uint64x2_t to minimize the number of vreinterpretq_xxx() calls, and to
be able to rely on the ^ operator for exclusive OR operations. This
makes the code much more concise and readable.
While at it, wrap the calls to vmull_p64() et al in order to have a more
consistent calling convention, and encapsulate any remaining
vreinterpret() calls that are still needed.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org >
Link: https://lore.kernel.org/r/20260330144630.33026-11-ardb@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org >
2026-04-02 16:14:53 -07:00
Ard Biesheuvel
f956dc8131
lib/crc: arm64: Use existing macros for kernel-mode FPU cflags
...
Use the existing CC_FPU_CFLAGS and CC_NO_FPU_CFLAGS to pass the
appropriate compiler command line options for building kernel mode NEON
intrinsics code. This is tidier, and will make it easier to reuse the
code for 32-bit ARM.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org >
Link: https://lore.kernel.org/r/20260330144630.33026-9-ardb@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org >
2026-04-02 16:14:53 -07:00
Ard Biesheuvel
e0718ed60d
lib/crc: arm64: Drop unnecessary chunking logic from crc64
...
On arm64, kernel mode NEON executes with preemption enabled, so there is
no need to chunk the input by hand.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org >
Link: https://lore.kernel.org/r/20260330144630.33026-8-ardb@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org >
2026-04-02 16:14:53 -07:00
Eric Biggers
5276ea17a2
lib/crc: arm64: Assume a little-endian kernel
...
Since support for big-endian arm64 kernels was removed, the CPU_LE()
macro now unconditionally emits the code it is passed, and the CPU_BE()
macro now unconditionally discards the code it is passed.
Simplify the assembly code in lib/crc/arm64/ accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org >
Link: https://lore.kernel.org/r/20260401004431.151432-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org >
2026-04-02 16:13:18 -07:00
Yury Norov
d57e74f104
bitmap: introduce bitmap_weighted_xor()
...
The function helps to XOR bitmaps and calculate Hamming weight of
the result in one pass.
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com >
Reviewed-by: Jacob Keller <jacob.e.keller@intel.com >
Signed-off-by: Yury Norov <ynorov@nvidia.com >
2026-04-01 20:03:07 -04:00
Breno Leitao
24b2e73f97
workqueue: add test_workqueue benchmark module
...
Add a kernel module that benchmarks queue_work() throughput on an
unbound workqueue to measure pool->lock contention under different
affinity scope configurations (cache vs cache_shard).
The module spawns N kthreads (default: num_online_cpus()), each bound
to a different CPU. All threads start simultaneously and queue work
items, measuring the latency of each queue_work() call. Results are
reported as p50/p90/p95 latencies for each affinity scope.
The affinity scope is switched between runs via the workqueue's sysfs
affinity_scope attribute (WQ_SYSFS), avoiding the need for any new
exported symbols.
The module runs as __init-only, returning -EAGAIN to auto-unload,
and can be re-run via insmod.
Example of the output:
running 50 threads, 50000 items/thread
cpu 6806017 items/sec p50=2574 p90=5068 p95=5818 ns
smt 6821040 items/sec p50=2624 p90=5168 p95=5949 ns
cache_shard 1633653 items/sec p50=5337 p90=9694 p95=11207 ns
cache 286069 items/sec p50=72509 p90=82304 p95=85009 ns
numa 319403 items/sec p50=63745 p90=73480 p95=76505 ns
system 308461 items/sec p50=66561 p90=75714 p95=78048 ns
Signed-off-by: Breno Leitao <leitao@debian.org >
Signed-off-by: Tejun Heo <tj@kernel.org >
2026-04-01 10:24:18 -10:00
Eric Biggers
12b11e47f1
lib/crypto: arm64: Assume a little-endian kernel
...
Since support for big-endian arm64 kernels was removed, the CPU_LE()
macro now unconditionally emits the code it is passed, and the CPU_BE()
macro now unconditionally discards the code it is passed.
Simplify the assembly code in lib/crypto/arm64/ accordingly.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org >
Link: https://lore.kernel.org/r/20260401003331.144065-1-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org >
2026-04-01 13:02:15 -07:00
Eric Biggers
6d575f11c7
lib/crypto: arm64/sha3: Remove obsolete chunking logic
...
Since commit aefbab8e777dadeaa6e8ardb@kernel.org >
Link: https://lore.kernel.org/r/20260401000548.133151-9-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org >
2026-04-01 13:02:10 -07:00
Eric Biggers
7116418f6b
lib/crypto: arm64/sha512: Remove obsolete chunking logic
...
Since commit aefbab8e777dadeaa6e8ardb@kernel.org >
Link: https://lore.kernel.org/r/20260401000548.133151-8-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org >
2026-04-01 13:02:10 -07:00
Eric Biggers
fe1233c2eb
lib/crypto: arm64/sha256: Remove obsolete chunking logic
...
Since commit aefbab8e777dadeaa6e8ardb@kernel.org >
Link: https://lore.kernel.org/r/20260401000548.133151-7-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org >
2026-04-01 13:02:10 -07:00
Eric Biggers
fd5017138c
lib/crypto: arm64/sha1: Remove obsolete chunking logic
...
Since commit aefbab8e777dadeaa6e8ardb@kernel.org >
Link: https://lore.kernel.org/r/20260401000548.133151-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org >
2026-04-01 13:02:10 -07:00
Eric Biggers
dec1061f0a
lib/crypto: arm64/poly1305: Remove obsolete chunking logic
...
Since commit aefbab8e777dadeaa6e8ardb@kernel.org >
Link: https://lore.kernel.org/r/20260401000548.133151-5-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org >
2026-04-01 13:02:10 -07:00
