Commit Graph

1088658 Commits

Author SHA1 Message Date
Kai Ye
5bfabd50c6 crypto: hisilicon/zip - support last word dumping
1. Add some debugging registers.
2. Add last word dumping function during zip engine controller reset.

Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:29 +08:00
Kai Ye
42123e81fd crypto: hisilicon/hpre - support last word dumping
1. Add some debugging registers.
2. Add last word dumping function during hpre engine controller reset.

Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:29 +08:00
Kai Ye
8a88d09145 crypto: hisilicon/sec - support last word dumping
Add last word dumping function during sec engine controller reset.

Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:29 +08:00
Kai Ye
a888ccd6c6 crypto: hisilicon/qm - add last word dumping for ACC
Add last word dumping function during acc engines controller reset.
The last words are reported to the printed information during the
reset. The dmesg information included qm debugging registers and
engine debugging registers. It can help to improve debugging
capability.

Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:29 +08:00
Kai Ye
30169c5b55 Documentation: update debugfs doc for Hisilicon ZIP
Update documentation describing DebugFS that could help
to check the change of register values.

Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:29 +08:00
Kai Ye
73e3b46e90 Documentation: update debugfs doc for Hisilicon SEC
Update documentation describing DebugFS that could help
to check the change of register values.

Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:29 +08:00
Kai Ye
a7dbdfda0c Documentation: update debugfs doc for Hisilicon HPRE
Update documentation describing DebugFS that could help
to check the change of register values.

Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:29 +08:00
Kai Ye
9b0c97dfc2 crypto: hisilicon/zip - support register checking
The value of the register is changed after the task running. A debugfs
file node is added to help users to check the change of register values.

Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:28 +08:00
Kai Ye
16175030bb crypto: hisilicon/sec - support register checking
The value of the register is changed after the task running. A debugfs
file node is added to help users to check the change of register values.

Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:28 +08:00
Kai Ye
9210bdaa0d crypto: hisilicon/hpre - support register checking
The value of the register is changed after the task running. A debugfs
file node is added to help users to check the change of register values.

Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:28 +08:00
Kai Ye
f1724d397c crypto: hisilicon/qm - add register checking for ACC
Add register detection function to accelerator. Provided a tool that
user can checking differential register through Debugfs.
e.g.
    cd /sys/kernel/debug/hisi_zip/<bdf>/zip_dfx
    cat diff_regs

Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Signed-off-by: Kai Ye <yekai13@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:28 +08:00
Yihao Han
10299073bc crypto: ux500/hash - simplify if-if to if-else
Replace `if (!req_ctx->updated)` with `else` for simplification
and add curly brackets according to the kernel coding style:

"Do not unnecessarily use braces where a single statement will do."

...

"This does not apply if only one branch of a conditional statement is
a single statement; in the latter case use braces in both branches"

Please refer to:
https://www.kernel.org/doc/html/v5.17-rc8/process/coding-style.html

Signed-off-by: Yihao Han <hanyihao@vivo.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:28 +08:00
Conor Dooley
716a757c83 hwrng: mpfs - add polarfire soc hwrng support
Add a driver to access the hardware random number generator on the
Polarfire SoC. The hwrng can only be accessed via the system controller,
so use the mailbox interface the system controller exposes to access the
hwrng.

Signed-off-by: Conor Dooley <conor.dooley@microchip.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:28 +08:00
Marco Chiappero
ebd26229a7 crypto: qat - remove line wrapping for pfvf_ops functions
Remove unnecessary line wrapping for the
adf_enable_vf2pf_interrupts() function, and harmonize pfvf_ops text.

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:27 +08:00
Marco Chiappero
fa37495483 crypto: qat - use u32 variables in all GEN4 pfvf_ops
Change adf_gen4_enable_vf2pf_interrupts() to use a u32 variable,
consistently with both other GEN4 pfvf_ops and pfvf_ops of other
generations.

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:27 +08:00
Marco Chiappero
e3e668fc77 crypto: qat - replace disable_vf2pf_interrupts()
As a consequence of the refactored VF2PF interrupt handling logic, a
function that disables specific VF2PF interrupts is no longer needed.
Instead, a simpler function that disables all the interrupts, also
hiding the device specific amount of VFs to be disabled from the
pfvf_ops users, would be sufficient.

This patch replaces disable_vf2pf_interrupts() with the new
disable_all_vf2pf_interrupts(), which doesn't need any argument and
disables all the VF2PF interrupts.

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:27 +08:00
Marco Chiappero
8314ae8f53 crypto: qat - leverage the GEN2 VF mask definiton
Replace hard coded VF masks in adf_gen2_pfvf.c with the recently
introduced ADF_GEN2_VF_MSK.

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:27 +08:00
Marco Chiappero
c690c7f631 crypto: qat - rework the VF2PF interrupt handling logic
Change the VF2PF interrupt handler in the PF ISR and the definition of
the internal PFVF API to correct the current implementation, which can
result in missed interrupts.

More specifically, current HW generations consider a write to the mask
register, regardless of the value, as an acknowledge of any pending
VF2PF interrupt. Therefore, if there is an interrupt between the source
register read and the mask register write, such interrupt will not be
delivered and silently acknowledged, resulting in a lost VF2PF message.

To work around the problem, rather than disabling specific interrupts,
disable all the interrupts and re-enable only the ones that we are not
serving (excluding the already disabled ones too). This will force any
other pending interrupt to be triggered and be serviced by a subsequent
ISR.

This new approach requires, however, changes to the interrupt related
pfvf_ops functions. In particular, get_vf2pf_sources() has now been
removed in favor of disable_pending_vf2pf_interrupts(), which not only
retrieves and returns the pending (and enabled) sources, but also
disables them.
As a consequence, introduce the adf_disable_pending_vf2pf_interrupts()
utility in place of adf_disable_vf2pf_interrupts_irq(), which is no
longer needed.

Cc: stable@vger.kernel.org
Fixes: 993161d ("crypto: qat - fix handling of VF to PF interrupts")
Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Co-developed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:27 +08:00
Marco Chiappero
dd3d081b7e crypto: qat - fix off-by-one error in PFVF debug print
PFVF Block Message requests for CRC use 0-based values to indicate
amounts, which have to be remapped to 1-based values on the receiving
side.

This patch fixes one debug print which was however using the wire value.

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:27 +08:00
Marco Chiappero
2ca1e0a7fa crypto: qat - fix wording and formatting in code comment
Remove an unintentional extra space and improve the readability of a
PFVF related code comment.

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:26 +08:00
Marco Chiappero
27c0f3a14f crypto: qat - test PFVF registers for spurious interrupts on GEN4
Spurious PFVF interrupts can happen when either the ISR is invoked
without a valid source being set or, otherwise, when no interrupt bit
is set in the PFVF register containing the message.

The latter test was present for GEN2 devices but missing for GEN4, this
patch fills the gap.

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:26 +08:00
Wojciech Ziemba
4b61d2bd34 crypto: qat - add check for invalid PFVF protocol version 0
PFVF protocol version 0 is not a valid version, but PF drivers
currently would report any such version from VFs as compatible.
This patch adds an extra check for the invalid PFVF protocol
version 0.

Signed-off-by: Wojciech Ziemba <wojciech.ziemba@intel.com>
Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Marco Chiappero <marco.chiappero@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:26 +08:00
Marco Chiappero
569b462e66 crypto: qat - add missing restarting event notification in VFs
VF drivers are notified via PFVF of the VFs being disabled, but
such notification was not propagated within the VF driver.
Dispatch the ADF_EVENT_RESTARTING event by adding a missing call
to adf_dev_restarting_notify().

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:26 +08:00
Marco Chiappero
f9f8f2b741 crypto: qat - remove unnecessary tests to detect PFVF support
Previously, the GEN4 host driver supported SR-IOV but had no working
implementation of the PFVF protocol to communicate with VF drivers.
Since all the host drivers for QAT devices now support both SR-IOV and
PFVF, remove the old and unnecessary checks to test PFVF support.

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:26 +08:00
Giovanni Cabiddu
80280aeb2d crypto: qat - remove unused PFVF stubs
The functions adf_enable_pf2vf_interrupts(), adf_flush_vf_wq() and
adf_disable_pf2vf_interrupts() are not referenced when the driver is
compiled with CONFIG_PCI_IOV=n. This patch removes these unused stubs.

Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Marco Chiappero <marco.chiappero@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:26 +08:00
Marco Chiappero
992ec1fa86 crypto: qat - remove unneeded braces
Remove unnecessary braces around a single statement in a for loop.

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:26 +08:00
Marco Chiappero
9ff9139b5d crypto: qat - fix ETR sources enabled by default on GEN2 devices
When the driver starts the device, it enables all the necessary
interrupts. However interrupts associated to host rings are enabled by
default on all GEN2 devices (except for dh895x) even when SR-IOV is
active. Fix this behaviour by checking if data structures associated to
VFs have been allocated to determine whether to enable such interrupts
or not.

Since the logic for the fix is the same across GEN2 devices, replace
the function to be fixed (adf_enable_ints()) with a single one
(adf_gen2_enable_ints()) in the common GEN2 code in adf_gen2_hw_data.c.
Likewise, remove the unnecessary duplication of defines too.

Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:26 +08:00
Giovanni Cabiddu
0eaa515432 crypto: qat - set COMPRESSION capability for DH895XCC
The capability detection logic clears bits for the features that are
disabled in a certain SKU. For example, if the bit associate to
compression is not present in the LEGFUSE register, the correspondent
bit is cleared in the capability mask.
This change adds the compression capability to the mask as this was
missing in the commit that enhanced the capability detection logic.

Fixes: cfe4894ecc ("crypto: qat - set COMPRESSION capability for QAT GEN2")
Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Marco Chiappero <marco.chiappero@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:25 +08:00
Giovanni Cabiddu
6a23804cb8 crypto: qat - set CIPHER capability for DH895XCC
Set the CIPHER capability for QAT DH895XCC devices if the hardware supports
it. This is done if both the CIPHER and the AUTHENTICATION engines are
available on the device.

Fixes: ad1332aa67 ("crypto: qat - add support for capability detection")
Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Marco Chiappero <marco.chiappero@intel.com>
Reviewed-by: Marco Chiappero <marco.chiappero@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:25 +08:00
Gilad Ben-Yossef
a260436c98 crypto: ccree - use fine grained DMA mapping dir
Use a fine grained specification of DMA mapping directions
in certain cases, allowing both a more optimized operation
as well as shushing out a harmless, though persky
dma-debug warning.

Signed-off-by: Gilad Ben-Yossef <gilad@benyossef.com>
Reported-by: Corentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:25 +08:00
Gilad Ben-Yossef
476c9ab759 crypto: ccree - rearrange init calls to avoid race
Rearrange init calls to avoid the rare race condition of
the cipher algs being registered and used while we still
init the hash code which uses the HW without proper lock.

Signed-off-by: Gilad Ben-Yossef <gilad@benyossef.com>
Reported-by: Dung Nguyen <dung.nguyen.zy@renesas.com>
Tested-by: Jing Dan <jing.dan.nx@renesas.com>
Tested-by: Dung Nguyen <dung.nguyen.zy@renesas.com>
Fixes: 63893811b0fc("crypto: ccree - add ahash support")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:25 +08:00
Robin Murphy
c79c09ad31 crypto: qat - stop using iommu_present()
Even if an IOMMU might be present for some PCI segment in the system,
that doesn't necessarily mean it provides translation for the device
we care about. Replace iommu_present() with a more appropriate check.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Acked-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-15 16:34:25 +08:00
Jakob Koschel
7b2206d880 crypto: cavium/nitrox - remove check of list iterator against head past the loop body
When list_for_each_entry() completes the iteration over the whole list
without breaking the loop, the iterator value will be a bogus pointer
computed based on the head element.

While it is safe to use the pointer to determine if it was computed
based on the head element, either with list_entry_is_head() or
&pos->member == head, using the iterator variable after the loop should
be avoided.

In preparation to limit the scope of a list iterator to the list
traversal loop, use a dedicated pointer to point to the found element [1].

Link: https://lore.kernel.org/all/CAHk-=wgRr_D8CB-D9Kg-c=EHreAsk5SqXPwr9Y7k9sA6cWXJ6w@mail.gmail.com/ [1]
Signed-off-by: Jakob Koschel <jakobkoschel@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:26:43 +08:00
Mario Limonciello
4e2c87949f crypto: ccp - When TSME and SME both detected notify user
CC_ATTR_HOST_MEM_ENCRYPT is used to relay that memory encryption has been
activated by the kernel.

As it's technically possible to enable both SME and TSME at the same time,
detect this scenario and notify the user that enabling TSME and SME at the
same time is unnecessary.

Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Acked-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:26:43 +08:00
Mario Limonciello
84ee393b1e crypto: ccp - Allow PSP driver to load without SEV/TEE support
Previously the PSP probe routine would fail if both SEV and TEE were
missing.  This is possibly the case for some client parts.

As capabilities can now be accessed from userspace, it may still be
useful to have the PSP driver finish loading so that those capabilities
can be read.

Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Acked-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:26:43 +08:00
Mario Limonciello
50c4decc1b crypto: ccp - Export PSP security bits to userspace
The PSP sets several pre-defined bits in the capabilities
register to indicate that security attributes of the platform.

Export these attributes into userspace for administrators to
confirm platform is properly locked down.

Acked-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:26:43 +08:00
Mario Limonciello
cac32cd4f1 crypto: ccp - cache capability into psp device
The results of the capability register will be used by future
code at runtime rather than just initialization.

Acked-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:25:19 +08:00
Eric Biggers
f17f9e9069 crypto: testmgr - test in-place en/decryption with two sglists
As was established in the thread
https://lore.kernel.org/linux-crypto/20220223080400.139367-1-gilad@benyossef.com/T/#u,
many crypto API users doing in-place en/decryption don't use the same
scatterlist pointers for the source and destination, but rather use
separate scatterlists that point to the same memory.  This case isn't
tested by the self-tests, resulting in bugs.

This is the natural usage of the crypto API in some cases, so requiring
API users to avoid this usage is not reasonable.

Therefore, update the self-tests to start testing this case.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:25:19 +08:00
Randy Dunlap
f16a005cde crypto: x86 - eliminate anonymous module_init & module_exit
Eliminate anonymous module_init() and module_exit(), which can lead to
confusion or ambiguity when reading System.map, crashes/oops/bugs,
or an initcall_debug log.

Give each of these init and exit functions unique driver-specific
names to eliminate the anonymous names.

Example 1: (System.map)
 ffffffff832fc78c t init
 ffffffff832fc79e t init
 ffffffff832fc8f8 t init

Example 2: (initcall_debug log)
 calling  init+0x0/0x12 @ 1
 initcall init+0x0/0x12 returned 0 after 15 usecs
 calling  init+0x0/0x60 @ 1
 initcall init+0x0/0x60 returned 0 after 2 usecs
 calling  init+0x0/0x9a @ 1
 initcall init+0x0/0x9a returned 0 after 74 usecs

Fixes: 64b94ceae8 ("crypto: blowfish - add x86_64 assembly implementation")
Fixes: 676a38046f ("crypto: camellia-x86_64 - module init/exit functions should be static")
Fixes: 0b95ec56ae ("crypto: camellia - add assembler implementation for x86_64")
Fixes: 56d76c96a9 ("crypto: serpent - add AVX2/x86_64 assembler implementation of serpent cipher")
Fixes: b9f535ffe3 ("[CRYPTO] twofish: i586 assembly version")
Fixes: ff0a70fe05 ("crypto: twofish-x86_64-3way - module init/exit functions should be static")
Fixes: 8280daad43 ("crypto: twofish - add 3-way parallel x86_64 assembler implemention")
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Cc: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Cc: Joachim Fritschi <jfritschi@freenet.de>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: linux-crypto@vger.kernel.org
Cc: x86@kernel.org
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:13:31 +08:00
Hui Tang
4cda2f4a0e crypto: hisilicon/qm - optimize the barrier operation
A 'dma_wmb' barrier is enough to guarantee previous writes
before accessing by acc device in the outer shareable domain.

A 'smp_wmb' barrier is enough to guarantee previous writes
before accessing by other cpus in the inner shareble domain.

Signed-off-by: Hui Tang <tanghui20@huawei.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:13:30 +08:00
Yang Li
d5db91d26a crypto: engine - Add parameter description in crypto_transfer_request() kernel-doc comment
Add the description of @need_pump in crypto_transfer_request() kernel-doc
comment to remove warning found by running scripts/kernel-doc, which is
caused by using 'make W=1'.

crypto/crypto_engine.c:260: warning: Function parameter or member
'need_pump' not described in 'crypto_transfer_request'

Reported-by: Abaci Robot <abaci@linux.alibaba.com>
Signed-off-by: Yang Li <yang.lee@linux.alibaba.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:13:30 +08:00
Tianjia Zhang
5b33e0ec88 crypto: arm64/sm4 - add ARMv8 Crypto Extensions implementation
This adds ARMv8 implementations of SM4 in ECB, CBC, CFB and CTR
modes using Crypto Extensions, also includes key expansion operations
because the Crypto Extensions instruction is much faster than software
implementations.

The Crypto Extensions for SM4 can only run on ARMv8 implementations
that have support for these optional extensions.

Benchmark on T-Head Yitian-710 2.75 GHz, the data comes from the 218
mode of tcrypt. The abscissas are blocks of different lengths. The
data is tabulated and the unit is Mb/s:

sm4-generic |     16       64      128      256     1024     1420     4096
    ECB enc |  80.05    91.42    93.66    94.77    95.69    95.77    95.86
    ECB dec |  79.98    91.41    93.64    94.76    95.66    95.77    95.85
    CBC enc |  78.55    86.50    88.02    88.77    89.36    89.42    89.48
    CBC dec |  76.82    89.06    91.52    92.77    93.75    93.83    93.96
    CFB enc |  77.64    86.13    87.62    88.42    89.08    88.83    89.18
    CFB dec |  77.57    88.34    90.36    91.45    92.34    92.00    92.44
    CTR enc |  77.80    88.28    90.23    91.22    92.11    91.81    92.25
    CTR dec |  77.83    88.22    90.22    91.22    92.04    91.82    92.28
sm4-neon
    ECB enc |  28.31   112.77   203.03   209.89   215.49   202.11   210.59
    ECB dec |  28.36   113.45   203.23   210.00   215.52   202.13   210.65
    CBC enc |  79.32    87.02    88.51    89.28    89.85    89.89    89.97
    CBC dec |  28.29   112.20   203.30   209.82   214.99   201.51   209.95
    CFB enc |  79.59    87.16    88.54    89.30    89.83    89.62    89.92
    CFB dec |  28.12   111.05   202.47   209.02   214.21   210.90   209.12
    CTR enc |  28.04   108.81   200.62   206.65   211.78   208.78   206.74
    CTR dec |  28.02   108.82   200.45   206.62   211.78   208.74   206.70
sm4-ce-cipher
    ECB enc | 336.79   587.13   682.70   747.37   803.75   811.52   818.06
    ECB dec | 339.18   584.52   679.72   743.68   798.82   803.83   811.54
    CBC enc | 316.63   521.47   597.00   647.14   690.82   695.21   700.55
    CBC dec | 291.80   503.79   585.66   640.82   689.86   695.16   701.72
    CFB enc | 294.79   482.31   552.13   594.71   631.60   628.91   638.92
    CFB dec | 293.09   466.44   526.56   563.17   594.41   592.26   601.97
    CTR enc | 309.61   506.13   576.86   620.47   656.38   654.51   665.10
    CTR dec | 306.69   505.57   576.84   620.18   657.09   654.52   665.32
sm4-ce
    ECB enc | 366.96  1329.81  2024.29  2755.50  3790.07  3861.91  4051.40
    ECB dec | 367.30  1323.93  2018.72  2747.43  3787.39  3862.55  4052.62
    CBC enc | 358.09   682.68   807.24   885.35   958.29   963.60   973.73
    CBC dec | 366.51  1303.63  1978.64  2667.93  3624.53  3683.41  3856.08
    CFB enc | 351.51   681.26   807.81   893.10   968.54   969.17   985.83
    CFB dec | 354.98  1266.61  1929.63  2634.81  3614.23  3611.59  3841.68
    CTR enc | 324.23  1121.25  1689.44  2256.70  2981.90  3007.79  3060.74
    CTR dec | 324.18  1120.44  1694.31  2258.32  2982.01  3010.09  3060.99

Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:13:29 +08:00
Tianjia Zhang
4f1aef9b80 crypto: arm64/sm4 - add ARMv8 NEON implementation
This adds ARMv8 NEON implementations of SM4 in ECB, CBC, CFB and CTR
modes. This implementation uses the plain NEON instruction set, All
S-BOX substitutions uses the tbl/tbx instructions of ARMv8, combined
with the out-of-order execution in CPU, this optimization supports
encryption of up to 8 blocks at the same time.

The performance of encrypting one block is not as good as software
implementation, so the encryption operations of CBC and CFB still
use pure software algorithms.

Benchmark on T-Head Yitian-710 2.75 GHz, the data comes from the 218
mode of tcrypt. The abscissas are blocks of different lengths. The
data is tabulated and the unit is Mb/s:

sm4-generic |     16       64      128      256     1024     1420     4096
    ECB enc |  80.05    91.42    93.66    94.77    95.69    95.77    95.86
    ECB dec |  79.98    91.41    93.64    94.76    95.66    95.77    95.85
    CBC enc |  78.55    86.50    88.02    88.77    89.36    89.42    89.48
    CBC dec |  76.82    89.06    91.52    92.77    93.75    93.83    93.96
    CFB enc |  77.64    86.13    87.62    88.42    89.08    88.83    89.18
    CFB dec |  77.57    88.34    90.36    91.45    92.34    92.00    92.44
    CTR enc |  77.80    88.28    90.23    91.22    92.11    91.81    92.25
    CTR dec |  77.83    88.22    90.22    91.22    92.04    91.82    92.28
sm4-neon
    ECB enc |  28.31   112.77   203.03   209.89   215.49   202.11   210.59
    ECB dec |  28.36   113.45   203.23   210.00   215.52   202.13   210.65
    CBC enc |  79.32    87.02    88.51    89.28    89.85    89.89    89.97
    CBC dec |  28.29   112.20   203.30   209.82   214.99   201.51   209.95
    CFB enc |  79.59    87.16    88.54    89.30    89.83    89.62    89.92
    CFB dec |  28.12   111.05   202.47   209.02   214.21   210.90   209.12
    CTR enc |  28.04   108.81   200.62   206.65   211.78   208.78   206.74
    CTR dec |  28.02   108.82   200.45   206.62   211.78   208.74   206.70

Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:13:13 +08:00
Tianjia Zhang
02436762f5 crypto: arm64/sm4-ce - rename to sm4-ce-cipher
The subsequent patches of the series will have an implementation
of SM4-ECB/CBC/CFB/CTR accelerated by the CE instruction set, which
conflicts with the current module name. In order to keep the naming
rules of the AES algorithm consistent, the sm4-ce algorithm is
renamed to sm4-ce-cipher.

In addition, the speed of sm4-ce-cipher is better than that of SM4
NEON. By the way, the priority of the algorithm is adjusted to 300,
which is also to leave room for the priority of SM4 NEON.

Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:12:47 +08:00
Tianjia Zhang
73c919d314 crypto: sm4 - export sm4 constant arrays
Export the constant arrays fk, ck, sbox of the SM4 algorithm, and
add the 'crypto_sm4_' prefix, where sbox is used in the SM4 NEON
implementation for the tbl/tbx instruction to replace the S-BOX,
and the fk, ck arrays are used in the SM4 CE implementation. Use
the sm4ekey instruction to speed up key expansion operations.

Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:12:46 +08:00
Jason A. Donenfeld
d2825fa936 crypto: sm3,sm4 - move into crypto directory
The lib/crypto libraries live in lib because they are used by various
drivers of the kernel. In contrast, the various helper functions in
crypto are there because they're used exclusively by the crypto API. The
SM3 and SM4 helper functions were erroniously moved into lib/crypto/
instead of crypto/, even though there are no in-kernel users outside of
the crypto API of those functions. This commit moves them into crypto/.

Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
Cc: Eric Biggers <ebiggers@kernel.org>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2022-04-08 16:11:48 +08:00
Linus Torvalds
3123109284 Linux 5.18-rc1 v5.18-rc1 2022-04-03 14:08:21 -07:00
Linus Torvalds
09bb8856d4 Merge tag 'trace-v5.18-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace
Pull more tracing updates from Steven Rostedt:

 - Rename the staging files to give them some meaning. Just
   stage1,stag2,etc, does not show what they are for

 - Check for NULL from allocation in bootconfig

 - Hold event mutex for dyn_event call in user events

 - Mark user events to broken (to work on the API)

 - Remove eBPF updates from user events

 - Remove user events from uapi header to keep it from being installed.

 - Move ftrace_graph_is_dead() into inline as it is called from hot
   paths and also convert it into a static branch.

* tag 'trace-v5.18-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace:
  tracing: Move user_events.h temporarily out of include/uapi
  ftrace: Make ftrace_graph_is_dead() a static branch
  tracing: Set user_events to BROKEN
  tracing/user_events: Remove eBPF interfaces
  tracing/user_events: Hold event_mutex during dyn_event_add
  proc: bootconfig: Add null pointer check
  tracing: Rename the staging files for trace_events
2022-04-03 12:26:01 -07:00
Linus Torvalds
34a53ff911 Merge tag 'clk-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux
Pull clk fix from Stephen Boyd:
 "A single revert to fix a boot regression seen when clk_put() started
  dropping rate range requests. It's best to keep various systems
  booting so we'll kick this out and try again next time"

* tag 'clk-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux:
  Revert "clk: Drop the rate range on clk_put()"
2022-04-03 12:21:14 -07:00
Linus Torvalds
8b5656bc4e Merge tag 'x86-urgent-2022-04-03' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 fixes from Thomas Gleixner:
 "A set of x86 fixes and updates:

   - Make the prctl() for enabling dynamic XSTATE components correct so
     it adds the newly requested feature to the permission bitmap
     instead of overwriting it. Add a selftest which validates that.

   - Unroll string MMIO for encrypted SEV guests as the hypervisor
     cannot emulate it.

   - Handle supervisor states correctly in the FPU/XSTATE code so it
     takes the feature set of the fpstate buffer into account. The
     feature sets can differ between host and guest buffers. Guest
     buffers do not contain supervisor states. So far this was not an
     issue, but with enabling PASID it needs to be handled in the buffer
     offset calculation and in the permission bitmaps.

   - Avoid a gazillion of repeated CPUID invocations in by caching the
     values early in the FPU/XSTATE code.

   - Enable CONFIG_WERROR in x86 defconfig.

   - Make the X86 defconfigs more useful by adapting them to Y2022
     reality"

* tag 'x86-urgent-2022-04-03' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/fpu/xstate: Consolidate size calculations
  x86/fpu/xstate: Handle supervisor states in XSTATE permissions
  x86/fpu/xsave: Handle compacted offsets correctly with supervisor states
  x86/fpu: Cache xfeature flags from CPUID
  x86/fpu/xsave: Initialize offset/size cache early
  x86/fpu: Remove unused supervisor only offsets
  x86/fpu: Remove redundant XCOMP_BV initialization
  x86/sev: Unroll string mmio with CC_ATTR_GUEST_UNROLL_STRING_IO
  x86/config: Make the x86 defconfigs a bit more usable
  x86/defconfig: Enable WERROR
  selftests/x86/amx: Update the ARCH_REQ_XCOMP_PERM test
  x86/fpu/xstate: Fix the ARCH_REQ_XCOMP_PERM implementation
2022-04-03 12:15:47 -07:00