mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-07-22 15:07:32 -04:00
dd29c017ae
RT migration is done aggressively. When a CPU schedules out a high
priority RT task for a lower priority task, it will look to see if there's
any RT tasks that are waiting to run on another CPU that is of higher
priority than the task this CPU is about to run. If it finds one, it will
pull that task over to the CPU and allow it to run there instead.
Normally, this pulling is done by looking at the RT overloaded mask (rto)
which contains all the CPUs in the scheduler domain with RT tasks that are
waiting to run due to a higher priority RT task currently running on their
CPU. The CPU that is about to schedule a lower priority task will grab the
rq lock of the overloaded CPU and move the RT task from that CPU's runqueue
to the local one and schedule the higher priority RT task.
This caused issues when a lot of CPUs would schedule a lower priority task
at the same time. They would all try to grab the same runqueue lock of
the CPU with the overloaded RT tasks. Only the first CPU that got in will
get that task. All the others would wait until they got the runqueue lock
and see there's nothing to pull and do nothing. On systems with lots of
CPUs, this caused a large latency (up to 500us) which is beyond what
PREEMPT_RT is to allow.
The solution to that was to create an RT_PUSH_IPI logic. When any CPU
wanted to pull a task, instead of grabbing the runqueue lock of the
overloaded CPU, it would start by sending an IPI to the overloaded CPU,
and that IPI handler would have the CPU with the waiting RT task do a push
instead. Then that handler would send an IPI to the next CPU with
overloaded RT tasks, and so on. Note, after the first CPU starts this
process, if another CPU wanted to do a pull, it would see that the process
has already begun and would only increment a counter to have the IPIs
continue again.
The RT_PUSH_IPI solved the latency problem with PREEMPT_RT but could cause
a new issue with non PREEMPT_RT. Namely, softirqs run in a threaded
context on PREEMPT_RT but they can run in an interrupt context in non-RT.
If an IPI lands on a CPU that has just woken up multiple RT tasks and the
current CPU is running a non RT or a low priority RT task, instead of
doing a push, it would simply do a schedule on that CPU. But if a softirq
was also executing on this CPU, the schedule would need to wait until the
softirq finished. Until then, the CPU would still be considered overloaded
as there are RT tasks still waiting to run on it.
A live lock occurred on a workload that was doing heavy networking traffic
on a large machine where the softirqs would run 500us out of 750us. And it
would also be waking up RT tasks, causing the RT pull logic to be
constantly executed.
When a softirq triggered on a CPU with RT tasks queued but not running
yet, and the other CPUs would see this CPU as being overloaded, they would
send an IPI over to it. The CPU would notice that the waiting RT tasks are
of higher priority than the currently running task and simply schedule
that CPU instead. But because the softirq was executing, before it could
schedule, it would receive another IPI to do the same. The amount of IPIs
would slow down the currently running softirq so much that before it could
return back to task context, it would execute another softirq never
allowing the CPU to schedule. This live locked that CPU.
As RT_PUSH_IPI was created to help PREEMPT_RT, make it default off if
PREEMPT_RT is not enabled.
Fixes: b6366f048e ("sched/rt: Use IPI to trigger RT task push migration instead of pulling")
Closes: https://lore.kernel.org/all/20260506235716.2530720-1-tj@kernel.org/
Reported-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260515103740.25ccbed8@gandalf.local.home
145 lines
3.7 KiB
C
145 lines
3.7 KiB
C
/* SPDX-License-Identifier: GPL-2.0 */
|
|
|
|
/*
|
|
* Using the avg_vruntime, do the right thing and preserve lag across
|
|
* sleep+wake cycles. EEVDF placement strategy #1, #2 if disabled.
|
|
*/
|
|
SCHED_FEAT(PLACE_LAG, true)
|
|
/*
|
|
* Give new tasks half a slice to ease into the competition.
|
|
*/
|
|
SCHED_FEAT(PLACE_DEADLINE_INITIAL, true)
|
|
/*
|
|
* Preserve relative virtual deadline on 'migration'.
|
|
*/
|
|
SCHED_FEAT(PLACE_REL_DEADLINE, true)
|
|
/*
|
|
* Inhibit (wakeup) preemption until the current task has either matched the
|
|
* 0-lag point or until is has exhausted it's slice.
|
|
*/
|
|
SCHED_FEAT(RUN_TO_PARITY, true)
|
|
/*
|
|
* Allow wakeup of tasks with a shorter slice to cancel RUN_TO_PARITY for
|
|
* current.
|
|
*/
|
|
SCHED_FEAT(PREEMPT_SHORT, true)
|
|
|
|
/*
|
|
* Prefer to schedule the task we woke last (assuming it failed
|
|
* wakeup-preemption), since its likely going to consume data we
|
|
* touched, increases cache locality.
|
|
*/
|
|
SCHED_FEAT(NEXT_BUDDY, false)
|
|
|
|
/*
|
|
* Allow completely ignoring cfs_rq->next; which can be set from various
|
|
* places:
|
|
* - NEXT_BUDDY (wakeup preemption)
|
|
* - yield_to_task()
|
|
* - cgroup dequeue / pick
|
|
*/
|
|
SCHED_FEAT(PICK_BUDDY, true)
|
|
|
|
/*
|
|
* Consider buddies to be cache hot, decreases the likeliness of a
|
|
* cache buddy being migrated away, increases cache locality.
|
|
*/
|
|
SCHED_FEAT(CACHE_HOT_BUDDY, true)
|
|
|
|
/*
|
|
* Delay dequeueing tasks until they get selected or woken.
|
|
*
|
|
* By delaying the dequeue for non-eligible tasks, they remain in the
|
|
* competition and can burn off their negative lag. When they get selected
|
|
* they'll have positive lag by definition.
|
|
*
|
|
* DELAY_ZERO clips the lag on dequeue (or wakeup) to 0.
|
|
*/
|
|
SCHED_FEAT(DELAY_DEQUEUE, true)
|
|
SCHED_FEAT(DELAY_ZERO, true)
|
|
|
|
SCHED_FEAT(PARANOID_AVG, false)
|
|
|
|
/*
|
|
* Allow wakeup-time preemption of the current task:
|
|
*/
|
|
SCHED_FEAT(WAKEUP_PREEMPTION, true)
|
|
|
|
#ifdef CONFIG_HRTIMER_REARM_DEFERRED
|
|
SCHED_FEAT(HRTICK, true)
|
|
SCHED_FEAT(HRTICK_DL, true)
|
|
#else
|
|
SCHED_FEAT(HRTICK, false)
|
|
SCHED_FEAT(HRTICK_DL, false)
|
|
#endif
|
|
|
|
/*
|
|
* Decrement CPU capacity based on time not spent running tasks
|
|
*/
|
|
SCHED_FEAT(NONTASK_CAPACITY, true)
|
|
|
|
#ifdef CONFIG_PREEMPT_RT
|
|
SCHED_FEAT(TTWU_QUEUE, false)
|
|
#else
|
|
|
|
/*
|
|
* Queue remote wakeups on the target CPU and process them
|
|
* using the scheduler IPI. Reduces rq->lock contention/bounces.
|
|
*/
|
|
SCHED_FEAT(TTWU_QUEUE, true)
|
|
#endif
|
|
|
|
/*
|
|
* When doing wakeups, attempt to limit superfluous scans of the LLC domain.
|
|
*/
|
|
SCHED_FEAT(SIS_UTIL, true)
|
|
|
|
/*
|
|
* Issue a WARN when we do multiple update_rq_clock() calls
|
|
* in a single rq->lock section. Default disabled because the
|
|
* annotations are not complete.
|
|
*/
|
|
SCHED_FEAT(WARN_DOUBLE_CLOCK, false)
|
|
|
|
#ifdef HAVE_RT_PUSH_IPI
|
|
/*
|
|
* In order to avoid a thundering herd attack of CPUs that are
|
|
* lowering their priorities at the same time, and there being
|
|
* a single CPU that has an RT task that can migrate and is waiting
|
|
* to run, where the other CPUs will try to take that CPUs
|
|
* rq lock and possibly create a large contention, sending an
|
|
* IPI to that CPU and let that CPU push the RT task to where
|
|
* it should go may be a better scenario.
|
|
*
|
|
* This is best for PREEMPT_RT, but for non-RT it can cause issues
|
|
* when preemption is disabled for long periods of time. Have
|
|
* it only default enabled for PREEMPT_RT.
|
|
*/
|
|
# ifdef CONFIG_PREEMPT_RT
|
|
SCHED_FEAT(RT_PUSH_IPI, true)
|
|
# else
|
|
SCHED_FEAT(RT_PUSH_IPI, false)
|
|
# endif
|
|
#endif
|
|
|
|
SCHED_FEAT(RT_RUNTIME_SHARE, false)
|
|
SCHED_FEAT(LB_MIN, false)
|
|
SCHED_FEAT(ATTACH_AGE_LOAD, true)
|
|
|
|
SCHED_FEAT(WA_IDLE, true)
|
|
SCHED_FEAT(WA_WEIGHT, true)
|
|
SCHED_FEAT(WA_BIAS, true)
|
|
|
|
/*
|
|
* UtilEstimation. Use estimated CPU utilization.
|
|
*/
|
|
SCHED_FEAT(UTIL_EST, true)
|
|
|
|
SCHED_FEAT(LATENCY_WARN, false)
|
|
|
|
/*
|
|
* Do newidle balancing proportional to its success rate using randomization.
|
|
*/
|
|
SCHED_FEAT(NI_RANDOM, true)
|
|
SCHED_FEAT(NI_RATE, true)
|