Fix a security vulnerability where malicious VCE command streams
with oversized dimensions (e.g. 65536×65536) cause 32-bit integer
overflow, wrapping the calculated buffer size to 0. This bypasses
validation and allows GPU firmware to perform out-of-bound memory
access.
The fix uses 64-bit arithmetic to detect overflow and rejects
invalid dimensions before they reach the hardware.
V2: remove redundant check
V3: modify max height value
V4: remove size64
Signed-off-by: Boyuan Zhang <boyuan.zhang@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit cbe408dba5)
Cc: stable@vger.kernel.org
186bfdc4e2