Commit Graph

113516 Commits

Author SHA1 Message Date
Ville Syrjälä
e6001c930a drm/i915: Move dbuf_state->active_pipes into skl_wm_get_hw_state()
Move the dbuf_state readout parts into skl_wm_get_hw_state()
so that the details are better hidden from sight.

This will stop updating this on pre-skl, but that's what we want
since the dbuf state is only used on skl+.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-18-ville.syrjala@linux.intel.com
2025-03-07 19:11:35 +02:00
Ville Syrjälä
14968c207a drm/i915: Do wm readout ealier for skl+
Move the wm readout to happen earlier. This is needed because
the bw_state readout will need ddb information populated by
the wm readout.

For now limit this to skl+ as I've not really analyzed the
implications of doing this on other platforms.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-17-ville.syrjala@linux.intel.com
2025-03-07 19:02:58 +02:00
Ville Syrjälä
dbf1b1ed55 drm/i915: Split wm sanitize from readout
I'll need to move the wm readout to an earlier point in the
sequence (since the bw state readout will need ddb information
from the wm readout). But (at least for now) the wm sanitation
will need to stay put as it needs to also sanitize things for
any pipes/planes we disable later during the hw state takeover.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-16-ville.syrjala@linux.intel.com
2025-03-07 19:02:55 +02:00
Ville Syrjälä
003ec4ce42 drm/i915: Simplify cdclk_disable_noatomic()
Instead of hand rolling the cdclk state disabling for a
pipe in noatomic() let's just recompute the whole thing
from scratch. Less code we have to remember to keep in sync.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-15-ville.syrjala@linux.intel.com
2025-03-07 19:02:52 +02:00
Ville Syrjälä
e45b8192c7 sem/i915: Simplify intel_cdclk_update_hw_state()
intel_crtc_calculate_min_cdclk() can't return an error
(since commit 5ac860cc52 ("drm/i915: Fix DBUF bandwidth vs.
cdclk handling")) so there is no point in checking for one.

Also we can just call it unconditionally since it itself
checks crtc_state->hw.enabled. We are currently checking
crtc_state->hw.active in the readout path, but active==enabled
during readout, and arguably enabled is the more correct thing
to check anyway.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-14-ville.syrjala@linux.intel.com
2025-03-07 19:02:49 +02:00
Ville Syrjälä
ff79a131d3 drm/i915: Skip some bw_state readout on pre-icl
We only compute bw_state->data_rate and bw_state->num_active_planes
on icl+. Do the same during readout so that we don't leave random
junk inside the state.

v2: Skip the whole intel_bw_crtc_update() (Vinod)

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-13-ville.syrjala@linux.intel.com
2025-03-07 19:02:46 +02:00
Ville Syrjälä
7b65b24a4e drm/i915: Update bw_state->active_pipes during readout
Update bw_state->active_pipes during readout.

This was completely missing from the current readout code.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-12-ville.syrjala@linux.intel.com
2025-03-07 19:02:44 +02:00
Ville Syrjälä
ad2d834f88 drm/i915: Extract intel_bw_update_hw_state()
Hoist the bw stuff into a separate function from
intel_modeset_readout_hw_state() so that the details
are better hidden inside intel_bw.c.

We can also skip the whole thing on pre-skl since the dbuf state
isn't actually used on those platforms.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-11-ville.syrjala@linux.intel.com
2025-03-07 19:02:41 +02:00
Ville Syrjälä
bd17fc4b6b drm/i915: Extract intel_cdclk_update_hw_state()
Hoist the cdclk stuff into a separate function from
intel_modeset_readout_hw_state() so that the details
are better hidden inside intel_cdclk.c.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-10-ville.syrjala@linux.intel.com
2025-03-07 19:02:37 +02:00
Ville Syrjälä
1ec9686095 drm/i915: Extract intel_bw_crtc_disable_noatomic()
Hoist the bw stuff into a separate function from
intel_crtc_disable_noatomic_complete() so that the details
are better hidden inside intel_bw.c.

We can also skip the whole thing on pre-skl since the dbuf state
isn't actually used on those platforms.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-9-ville.syrjala@linux.intel.com
2025-03-07 19:02:35 +02:00
Ville Syrjälä
d34b59d5ba drm/i915: Add skl_wm_plane_disable_noatomic()
Add skl_wm_plane_disable_noatomic() which will clear out all
the ddb and wm state for the plane. And let's do this _before_
we call plane->disable_arm() so that it'll actually clear out
the state in the hardware as well.

Currently this won't do anything new for most of the
intel_plane_disable_noatomic() calls since those are done before
wm readout, and thus everything wm/ddb related in the state
will still be zeroed anyway. The only difference will be for
skl_dbuf_sanitize() is happens after wm readout. But I'll be
reordering thigns so that wm readout happens earlier and at that
point this will guarantee that we still clear out the old
wm/ddb junk from the state.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-8-ville.syrjala@linux.intel.com
2025-03-07 19:02:32 +02:00
Ville Syrjälä
1dc6076c67 drm/i915: clean up pipe's ddb usage in intel_crtc_disable_noatomic()
Update the ddb tracking information when we disable a pipe
during sanitization. Avoids leaving stale junk in the states.

Currently this doesn't do anything as we haven't read out this
state yet when we do the sanitization, but that will change soon.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-7-ville.syrjala@linux.intel.com
2025-03-07 19:02:30 +02:00
Ville Syrjälä
d66fbd662d drm/i915: Extract skl_wm_crtc_disable_noatomic()
Hoist the dbuf stuff into a separate function from
intel_crtc_disable_noatomic_complete() so that the details
are better hidden inside skl_watermark.c.

We can also skip the whole thing on pre-skl since the dbuf state
isn't actually used on those platforms. The readout path does
still fill dbuf_state->active_pipes but we'll remedy that later.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-6-ville.syrjala@linux.intel.com
2025-03-07 19:02:27 +02:00
Ville Syrjälä
082ed1ffc2 drm/i915: Extract intel_cdclk_crtc_disable_noatomic()
Hoist the cdclk stuff into a separate function from
intel_crtc_disable_noatomic_complete() so that the details
are better hidden inside intel_cdclk.c.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-5-ville.syrjala@linux.intel.com
2025-03-07 19:02:24 +02:00
Ville Syrjälä
cd5693cf5e drm/i915: Use intel_plane_set_invisible() in intel_plane_disable_noatomic()
Reuse intel_plane_set_invisible() in intel_plane_disable_noatomic()
instead of hand rolling the same stuff.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-4-ville.syrjala@linux.intel.com
2025-03-07 19:02:21 +02:00
Ville Syrjälä
0851227d62 drm/i915: Don't clobber crtc_state->cpu_transcoder for inactive crtcs
Inactive crtcs are supposed to have their crtc_state completely
cleared. Currently we are clobbering crtc_state->cpu_transcoder
before determining whether it's actually enabled or not. Don't
do that.

I want to rework the inherited flag handling for inactive crtcs
a bit, and having a bogus cpu_transcoder in the crtc state can
then cause confusing fastset mismatches even when the crtc never
changes state during the commit.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-3-ville.syrjala@linux.intel.com
2025-03-07 19:02:18 +02:00
Ville Syrjälä
72eea9175e drm/i915: Drop redundant shared_dpll=NULL assignments
The crtc state is expected to be fully cleared before readout,
so there is no need to clear the shared_dpll pointers by hand.

Reviewed-by: Vinod Govindapillai <vinod.govindapillai@intel.com>
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306163420.3961-2-ville.syrjala@linux.intel.com
2025-03-07 19:02:13 +02:00
Ville Syrjälä
68b97b12c4 drm/i915: Program CURSOR_PROGRAM and COEFF_POLARITY for icl+ combo PHYs
Bspec asks us to clear the CURSOR_PROGRAM and COEFF_POLARITY
bits in PORT_TX_DW5 on icl+ combo PHYs. Make it so.

Bspec: 21257, 49291
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250303123952.5669-2-ville.syrjala@linux.intel.com
Reviewed-by: Chaitanya Kumar Borah <chaitanya.kumar.borah@intel.com>
2025-03-07 19:01:26 +02:00
Jani Nikula
30a6be2c20 drm/i915/plane: convert intel_atomic_plane.[ch] to struct intel_display
Going forward, struct intel_display is the main display device data
pointer. Convert intel_atomic_plane.[ch] to struct intel_display.

Reviewed-by: Nemesa Garg <nemesa.garg@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/d7e28ad43f67d92e54fb7e14373872b5e561038c.1741192597.git.jani.nikula@intel.com
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-03-07 11:36:37 +02:00
Jani Nikula
2fff55d499 drm/xe/compat: refactor compat i915_drv.h
The compat i915_drv.h contains things that aren't there in the original
i915_drv.h. Split out gem/i915_gem_object.h and i915_scheduler_types.h,
moving the corresponding pieces out, including FORCEWAKE_ALL to
intel_uncore.h.

Technically I915_PRIORITY_DISPLAY should be in i915_priolist_types.h,
but it's a bit overkill to split out another file just for
that. i915_scheduler_types.h shall do.

With this, the compat i915_drv.h becomes a strict subset of the
original.

Reviewed-by: Nemesa Garg <nemesa.garg@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/d6bd95bf52aa37f48ddec3e675b7a3cc66829eef.1741192597.git.jani.nikula@intel.com
[Jani: fix i915_gem_object.h header guard while applying]
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
2025-03-07 11:36:07 +02:00
Jiapeng Chong
2060a2db12 drm/msm/dpu: Remove duplicate dpu_hw_cwb.h header
./drivers/gpu/drm/msm/disp/dpu1/dpu_encoder.c: dpu_hw_cwb.h is included more than once.

Reported-by: Abaci Robot <abaci@linux.alibaba.com>
Closes: https://bugzilla.openanolis.cn/show_bug.cgi?id=19239
Signed-off-by: Jiapeng Chong <jiapeng.chong@linux.alibaba.com>
Reported-by: kernel test robot <lkp@intel.com>
Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Fixes: dd331404ac ("drm/msm/dpu: Configure CWB in writeback encoder")
Patchwork: https://patchwork.freedesktop.org/patch/641543/
Link: https://lore.kernel.org/r/20250307015030.86282-1-jiapeng.chong@linux.alibaba.com
Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
2025-03-07 06:05:25 +02:00
Jessica Zhang
1cf5cd92a2 drm/msm/dpu: Adjust CDM_MUX to support CWB PINGPONG
Similar to WB_MUX, CDM_MUX also needs to be adjusted to support
dedicated CWB PINGPONGs

Signed-off-by: Jessica Zhang <quic_jesszhan@quicinc.com>
Reviewed-by: Abhinav Kumar <quic_abhinavk@quicinc.com>
Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Patchwork: https://patchwork.freedesktop.org/patch/641272/
Link: https://lore.kernel.org/r/20250305-cdm-cwb-mux-fix-v1-1-16148ca6e4d2@quicinc.com
Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
2025-03-07 06:02:13 +02:00
Dave Airlie
d65a27f95f Merge tag 'drm-misc-next-2025-03-06' of https://gitlab.freedesktop.org/drm/misc/kernel into drm-next
drm-misc-next for v6.15:

Cross-subsystem Changes:

base:
- component: Provide helper to query bound status

fbdev:
- fbtft: Remove access to page->index

Core Changes:

- Fix usage of logging macros in several places

gem:
- Add test function for imported dma-bufs and use it in core and helpers
- Avoid struct drm_gem_object.import_attach

tests:
- Fix lockdep warnings

ttm:
- Add helpers for TTM shrinker

Driver Changes:

adp:
- Add support for Apple Touch Bar displays on M1/M2

amdxdna:
- Fix interrupt handling

appletbdrm:
- Add support for Apple Touch Bar displays on x86

bridge:
- synopsys: Add HDMI audio support
- ti-sn65dsi83: Support negative DE polarity

ipu-v3:
- Remove unused code

nouveau:
- Avoid multiple -Wflex-array-member-not-at-end warnings

panthor:
- Fix CS_STATUS_ defines
- Improve locking

rockchip:
- analogix_dp: Add eDP support
- lvds: Improve logging
- vop2: Improve HDMI mode handling; Add support for RK3576
- Fix shutdown
- Support rk3562-mali

xe:
- Use TTM shrinker

Signed-off-by: Dave Airlie <airlied@redhat.com>

From: Thomas Zimmermann <tzimmermann@suse.de>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306130700.GA485504@linux.fritz.box
2025-03-07 09:55:50 +10:00
Dave Airlie
c8bc66206a Merge tag 'amd-drm-fixes-6.14-2025-03-06' of https://gitlab.freedesktop.org/agd5f/linux into drm-fixes
amd-drm-fixes-6.14-2025-03-06:

amdgpu:
- Fix NULL check in DC code
- SMU 14 fix

amdkfd:
- Fix NULL check in queue validation

radeon:
- RS400 HyperZ fix

Signed-off-by: Dave Airlie <airlied@redhat.com>

From: Alex Deucher <alexander.deucher@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306193424.27413-1-alexander.deucher@amd.com
2025-03-07 09:54:43 +10:00
Dave Airlie
eea6520c22 Merge tag 'drm-xe-fixes-2025-03-06' of https://gitlab.freedesktop.org/drm/xe/kernel into drm-fixes
- Remove double page flip on initial plane (Maarten)
- Properly setup userptr pfn_flags_mask (Auld)
- Fix GT "for each engine" workarounds (Tvrtko)
- Fix userptr races and missed validations (Thomas, Brost)
- Userptr invalid page access fixes (Thomas)

Signed-off-by: Dave Airlie <airlied@redhat.com>

From: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/Z8ni6w3tskCFL11O@intel.com
2025-03-07 09:03:18 +10:00
Dave Airlie
019899b551 Merge tag 'drm-intel-fixes-2025-03-06' of https://gitlab.freedesktop.org/drm/i915/kernel into drm-fixes
- DP MST fix (Jani)

Signed-off-by: Dave Airlie <airlied@redhat.com>

From: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/Z8ng8NjmRGiVcb5t@intel.com
2025-03-07 08:25:11 +10:00
Dave Airlie
ac3a75bd42 Merge tag 'drm-misc-fixes-2025-03-06' of https://gitlab.freedesktop.org/drm/misc/kernel into drm-fixes
A Kconfig fix for nouveau, locking and timestamp fixes for imagination,
a header guard fix for sched and a DPMS regression fix for bochs.

Signed-off-by: Dave Airlie <airlied@redhat.com>

From: Maxime Ripard <mripard@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306-antelope-of-imminent-anger-bca19e@houat
2025-03-07 07:06:34 +10:00
Matthew Brost
c56904f6cc drm/xe: Add always_migrate_to_vram modparam
Used to show we can bounce memory multiple times which will happen once
a real migration policy is implemented. Can be removed once migration
policy is implemented.

v3:
 - Pull some changes into the previous patch (Thomas)
 - Better commit message (Thomas)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-32-matthew.brost@intel.com
2025-03-06 11:36:57 -08:00
Matthew Brost
8e5a5dc056 drm/xe: Add modparam for SVM notifier size
Useful to experiment with notifier size and how it affects performance.

v3:
 - Pull missing changes including in following patch (Thomas)
v5:
 - Spell out power of 2 (Thomas)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-31-matthew.brost@intel.com
2025-03-06 11:36:57 -08:00
Matthew Brost
d92eabb370 drm/xe: Add SVM debug
Add some useful SVM debug logging fro SVM range which prints the range's
state.

v2:
 - Update logging with latest structure layout
v3:
 - Better commit message (Thomas)
 - New range structure (Thomas)
 - s/COLLECTOT/s/COLLECTOR (Thomas)
v4:
 - Drop partial evict message (Thomas)
 - Use %p for pointers print (Thomas)
v6:
 - Cast dma_addr to u64 (CI)
 - Only compile if CONFIG_DRM_GPUSVM selected (CI, Lucas)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-30-matthew.brost@intel.com
2025-03-06 11:36:57 -08:00
Matthew Brost
3ca608dc75 drm/xe: Basic SVM BO eviction
Wire xe_bo_move to GPU SVM migration via new helper xe_svm_bo_evict.

v2:
 - Use xe_svm_bo_evict
 - Drop bo->range
v3:
 - Kernel doc (Thomas)
v4:
 - Add missing xe_bo.c code
v5:
 - Add XE_BO_FLAG_CPU_ADDR_MIRROR flag in this patch (Thomas)
 - Add message on eviction failure
v6:
 - Only compile if CONFIG_DRM_GPUSVM selected (CI, Lucas)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-29-matthew.brost@intel.com
2025-03-06 11:36:57 -08:00
Matthew Brost
2f118c9491 drm/xe: Add SVM VRAM migration
Migration is implemented with range granularity, with VRAM backing being
a VM private TTM BO (i.e., shares dma-resv with VM). The lifetime of the
TTM BO is limited to when the SVM range is in VRAM (i.e., when a VRAM
SVM range is migrated to SRAM, the TTM BO is destroyed).

The design choice for using TTM BO for VRAM backing store, as opposed to
direct buddy allocation, is as follows:

- DRM buddy allocations are not at page granularity, offering no
  advantage over a BO.
- Unified eviction is required (SVM VRAM and TTM BOs need to be able to
  evict each other).
- For exhaustive eviction [1], SVM VRAM allocations will almost certainly
  require a dma-resv.
- Likely allocation size is 2M which makes of size of BO (872)
  acceptable per allocation (872 / 2M == .0004158).

With this, using TTM BO for VRAM backing store seems to be an obvious
choice as it allows leveraging of the TTM eviction code.

Current migration policy is migrate any SVM range greater than or equal
to 64k once.

[1] https://patchwork.freedesktop.org/series/133643/

v2:
 - Rebase on latest GPU SVM
 - Retry page fault on get pages returning mixed allocation
 - Use drm_gpusvm_devmem
v3:
 - Use new BO flags
 - New range structure (Thomas)
 - Hide migration behind Kconfig
 - Kernel doc (Thomas)
 - Use check_pages_threshold
v4:
 - Don't evict partial unmaps in garbage collector (Thomas)
 - Use %pe to print errors (Thomas)
 - Use %p to print pointers (Thomas)
v5:
 - Use range size helper (Thomas)
 - Make BO external (Thomas)
 - Set tile to NULL for BO creation (Thomas)
 - Drop BO mirror flag (Thomas)
 - Hold BO dma-resv lock across migration (Auld, Thomas)
v6:
 - s/drm_info/drm_dbg (Thomas)
 - s/migrated/skip_migrate (Himal)
 - Better debug message on VRAM migration failure (Himal)
 - Drop return BO from VRAM allocation function (Thomas)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-28-matthew.brost@intel.com
2025-03-06 11:36:56 -08:00
Matthew Brost
5951fed85c drm/xe: Add Xe SVM devmem_release GPU SVM vfunc
Implement with a simple BO put which releases the device memory.

v2:
 - Use new drm_gpusvm_devmem_ops
v3:
 - Better commit message (Thomas)
v4:
 - Use xe_bo_put_async (Thomas)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-27-matthew.brost@intel.com
2025-03-06 11:36:56 -08:00
Matthew Brost
ecacec0f4a drm/xe: Add Xe SVM populate_devmem_pfn GPU SVM vfunc
Get device pfns from BO's buddy blocks. Used in migrate_* core MM
functions called in GPU SVM to migrate between device and system memory.

v2:
 - Use new drm_gpusvm_devmem_ops
v3:
 - Better commit message (Thomas)
v5:
 - s/xe_mem_region/xe_vram_region (Rebase)

Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-26-matthew.brost@intel.com
2025-03-06 11:36:56 -08:00
Matthew Brost
c5b3eb5a90 drm/xe: Add GPUSVM device memory copy vfunc functions
Add GPUSVM device memory copy vfunc functions and connect to migration
layer. Used for device memory migration.

v2:
 - Allow NULL device pages in xe_svm_copy
 - Use new drm_gpusvm_devmem_ops
v3:
 - Prefix defines with XE_ (Thomas)
 - Change copy chunk size to 8M
 - Add a bunch of comments to xe_svm_copy to clarify behavior (Thomas)
 - Better commit message (Thomas)
v5:
 - s/xe_mem_region/xe_vram_region (Rebase)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-25-matthew.brost@intel.com
2025-03-06 11:36:56 -08:00
Thomas Hellström
11bbe0d9aa drm/xe: Add drm_pagemap ops to SVM
Add support for mapping device pages to Xe SVM by attaching drm_pagemap
to a memory region, which is then linked to a GPU SVM devmem allocation.
This enables GPU SVM to derive the device page address.

v3:
 - Better commit message (Thomas)
 - New drm_pagemap.h location
v5:
 - s/xe_mem_region/xe_vram_region (Rebase)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-24-matthew.brost@intel.com
2025-03-06 11:36:54 -08:00
Matthew Brost
808c37ee39 drm/xe: Add drm_gpusvm_devmem to xe_bo
Add drm_gpusvm_devmem to xe_bo. Required to enable SVM migrations.

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-23-matthew.brost@intel.com
2025-03-06 11:35:56 -08:00
Matthew Brost
0c30c65473 drm/xe: Add SVM device memory mirroring
Add SVM device memory mirroring which enables device pages for
migration. Enabled via CONFIG_XE_DEVMEM_MIRROR Kconfig. Kconfig option
defaults to enabled. If not enabled, SVM will work sans migration and
KMD memory footprint will be less.

v3:
 - Add CONFIG_XE_DEVMEM_MIRROR
v4:
 - Fix Kconfig (Himal)
 - Use %pe to print errors (Thomas)
 - Fix alignment issue (Checkpatch)
v5:
 - s/xe_mem_region/xe_vram_region (Rebase)
v6:
 - Only compile if CONFIG_DRM_GPUSVM selected (CI, Lucas)
 - s/drm_info/drm_dbg/

Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
Signed-off-by: Oak Zeng <oak.zeng@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-22-matthew.brost@intel.com
2025-03-06 11:35:54 -08:00
Matthew Brost
9c44fd5f6e drm/xe: Add migrate layer functions for SVM support
Add functions which migrate to / from VRAM accepting a single DPA
argument (VRAM) and array of dma addresses (SRAM). Used for SVM
migrations.

v2:
 - Don't unlock job_mutex in error path of xe_migrate_vram
v3:
 - Kernel doc (Thomas)
 - Better commit message (Thomas)
 - s/dword/num_dword (Thomas)
 - Return error on to large of migration (Thomas)

Signed-off-by: Oak Zeng <oak.zeng@intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-21-matthew.brost@intel.com
2025-03-06 11:35:53 -08:00
Matthew Brost
77613a2e10 drm/xe/uapi: Add DRM_XE_QUERY_CONFIG_FLAG_HAS_CPU_ADDR_MIRROR
Add the DRM_XE_QUERY_CONFIG_FLAG_HAS_CPU_ADDR_MIRROR device query flag,
which indicates whether the device supports CPU address mirroring. The
intent is for UMDs to use this query to determine if a VM can be set up
with CPU address mirroring. This flag is implemented by checking if the
device supports GPU faults.

v7:
 - Only report enabled if CONFIG_DRM_GPUSVM is selected (CI)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-20-matthew.brost@intel.com
2025-03-06 11:35:52 -08:00
Matthew Brost
c73b2cbd10 drm/xe: Enable CPU address mirror uAPI
Support for CPU address mirror bindings in SRAM fully in place, enable the
implementation.

v3:
 - s/system allocator/CPU address mirror (Thomas)
v7:
 - Only enable uAPI if selected by GPU SVM (CI)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-19-matthew.brost@intel.com
2025-03-06 11:35:51 -08:00
Matthew Brost
f0e4238f6d drm/xe: Do not allow CPU address mirror VMA unbind if
uAPI is designed with the use case that only mapping a BO to a malloc'd
address will unbind a CPU-address mirror VMA. Therefore, allowing a
CPU-address mirror VMA to unbind when the GPU has bindings in the range
being unbound does not make much sense. This behavior is not supported,
as it simplifies the code. This decision can always be revisited if a
use case arises.

v3:
 - s/arrises/arises (Thomas)
 - s/system allocator/GPU address mirror (Thomas)
 - Kernel doc (Thomas)
 - Newline between function defs (Thomas)
v5:
 - Kernel doc (Thomas)
v6:
 - Only compile if CONFIG_DRM_GPUSVM selected (CI, Lucas)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-18-matthew.brost@intel.com
2025-03-06 11:35:49 -08:00
Matthew Brost
d1e6efdfab drm/xe: Add unbind to SVM garbage collector
Add unbind to SVM garbage collector. To facilitate add unbind support
function to VM layer which unbinds a SVM range. Also teach PT layer to
understand unbinds of SVM ranges.

v3:
 - s/INVALID_VMA/XE_INVALID_VMA (Thomas)
 - Kernel doc (Thomas)
 - New GPU SVM range structure (Thomas)
 - s/DRM_GPUVA_OP_USER/DRM_GPUVA_OP_DRIVER (Thomas)
v4:
 - Use xe_vma_op_unmap_range (Himal)
v5:
 - s/PY/PT (Thomas)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-17-matthew.brost@intel.com
2025-03-06 11:35:47 -08:00
Matthew Brost
63f6e480d1 drm/xe: Add SVM garbage collector
Add basic SVM garbage collector which destroy a SVM range upon a MMU
UNMAP event. The garbage collector runs on worker or in GPU fault
handler and is required as locks in the path of reclaim are required and
cannot be taken the notifier.

v2:
 - Flush garbage collector in xe_svm_close
v3:
 - Better commit message (Thomas)
 - Kernel doc (Thomas)
 - Use list_first_entry_or_null for garbage collector loop (Thomas)
 - Don't add to garbage collector if VM is closed (Thomas)
v4:
 - Use %pe to print error (Thomas)
v5:
 - s/visable/visible (Thomas)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-16-matthew.brost@intel.com
2025-03-06 11:35:46 -08:00
Matthew Brost
7d1d48fb17 drm/xe: Add (re)bind to SVM page fault handler
Add (re)bind to SVM page fault handler. To facilitate add support
function to VM layer which (re)binds a SVM range. Also teach PT layer to
understand (re)binds of SVM ranges.

v2:
 - Don't assert BO lock held for range binds
 - Use xe_svm_notifier_lock/unlock helper in xe_svm_close
 - Use drm_pagemap dma cursor
 - Take notifier lock in bind code to check range state
v3:
 - Use new GPU SVM range structure (Thomas)
 - Kernel doc (Thomas)
 - s/DRM_GPUVA_OP_USER/DRM_GPUVA_OP_DRIVER (Thomas)
v5:
 - Kernel doc (Thomas)
v6:
 - Only compile if CONFIG_DRM_GPUSVM selected (CI, Lucas)

Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Tested-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-15-matthew.brost@intel.com
2025-03-06 11:35:43 -08:00
Matthew Brost
ab498828fa drm/xe: Add SVM range invalidation and page fault
Add SVM range invalidation vfunc which invalidates PTEs. A new PT layer
function which accepts a SVM range is added to support this. In
addition, add the basic page fault handler which allocates a SVM range
which is used by SVM range invalidation vfunc.

v2:
 - Don't run invalidation if VM is closed
 - Cycle notifier lock in xe_svm_close
 - Drop xe_gt_tlb_invalidation_fence_fini
v3:
 - Better commit message (Thomas)
 - Add lockdep asserts (Thomas)
 - Add kernel doc (Thomas)
 - s/change/changed (Thomas)
 - Use new GPU SVM range / notifier structures
 - Ensure PTEs are zapped / dma mappings are unmapped on VM close (Thomas)
v4:
 - Fix macro (Checkpatch)
v5:
 - Use range start/end helpers (Thomas)
 - Use notifier start/end helpers (Thomas)
v6:
 - Use min/max helpers (Himal)
 - Only compile if CONFIG_DRM_GPUSVM selected (CI, Lucas)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-13-matthew.brost@intel.com
2025-03-06 11:35:40 -08:00
Matthew Brost
074e40d9c2 drm/xe: Nuke VM's mapping upon close
Clear root PT entry and invalidate entire VM's address space when
closing the VM. Will prevent the GPU from accessing any of the VM's
memory after closing.

v2:
 - s/vma/vm in kernel doc (CI)
 - Don't nuke migration VM as this occur at driver unload (CI)
v3:
 - Rebase and pull into SVM series (Thomas)
 - Wait for pending binds (Thomas)
v5:
 - Remove xe_gt_tlb_invalidation_fence_fini in error case (Matt Auld)
 - Drop local migration bool (Thomas)
v7:
 - Add drm_dev_enter/exit protecting invalidation (CI, Matt Auld)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-12-matthew.brost@intel.com
2025-03-06 11:35:38 -08:00
Thomas Hellström
85d4653354 drm/xe: Add dma_addr res cursor
Add dma_addr res cursor which walks an array of drm_pagemap_dma_addr.
Useful for SVM ranges and programing page tables.

v3:
 - Better commit message (Thomas)
 - Use new drm_pagemap.h location
v7:
 - Fix kernel doc (CI)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-11-matthew.brost@intel.com
2025-03-06 11:35:37 -08:00
Matthew Brost
6fd979c2f3 drm/xe: Add SVM init / close / fini to faulting VMs
Add SVM init / close / fini to faulting VMs. Minimual implementation
acting as a placeholder for follow on patches.

v2:
 - Add close function
v3:
 - Better commit message (Thomas)
 - Kernel doc (Thomas)
 - Update chunk array to be unsigned long (Thomas)
 - Use new drm_gpusvm.h header location (Thomas)
 - Newlines between functions in xe_svm.h (Thomas)
 - Call drm_gpusvm_driver_set_lock in init (Thomas)
v6:
 - Only compile if CONFIG_DRM_GPUSVM selected (CI, Lucas)
v7:
 - Only select CONFIG_DRM_GPUSVM if DEVICE_PRIVATE (CI)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-10-matthew.brost@intel.com
2025-03-06 11:35:35 -08:00
Matthew Brost
b43e864af0 drm/xe/uapi: Add DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR
Add the DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR flag, which is used to
create unpopulated virtual memory areas (VMAs) without memory backing or
GPU page tables. These VMAs are referred to as CPU address mirror VMAs.
The idea is that upon a page fault or prefetch, the memory backing and
GPU page tables will be populated.

CPU address mirror VMAs only update GPUVM state; they do not have an
internal page table (PT) state, nor do they have GPU mappings.

It is expected that CPU address mirror VMAs will be mixed with buffer
object (BO) VMAs within a single VM. In other words, system allocations
and runtime allocations can be mixed within a single user-mode driver
(UMD) program.

Expected usage:

- Bind the entire virtual address (VA) space upon program load using the
  DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR flag.
- If a buffer object (BO) requires GPU mapping (runtime allocation),
  allocate a CPU address using mmap(PROT_NONE), bind the BO to the
  mmapped address using existing bind IOCTLs. If a CPU map of the BO is
  needed, mmap it again to the same CPU address using mmap(MAP_FIXED)
- If a BO no longer requires GPU mapping, munmap it from the CPU address
  space and them bind the mapping address with the
  DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR flag.
- Any malloc'd or mmapped CPU address accessed by the GPU will be
  faulted in via the SVM implementation (system allocation).
- Upon freeing any mmapped or malloc'd data, the SVM implementation will
  remove GPU mappings.

Only supporting 1 to 1 mapping between user address space and GPU
address space at the moment as that is the expected use case. uAPI
defines interface for non 1 to 1 but enforces 1 to 1, this restriction
can be lifted if use cases arrise for non 1 to 1 mappings.

This patch essentially short-circuits the code in the existing VM bind
paths to avoid populating page tables when the
DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR flag is set.

v3:
 - Call vm_bind_ioctl_ops_fini on -ENODATA
 - Don't allow DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR on non-faulting VMs
 - s/DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR/DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR (Thomas)
 - Rework commit message for expected usage (Thomas)
 - Describe state of code after patch in commit message (Thomas)
v4:
 - Fix alignment (Checkpatch)

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20250306012657.3505757-9-matthew.brost@intel.com
2025-03-06 11:35:33 -08:00