[PATCH v2] sched/cache: Honor asym packing over cache aware scheduling on hybrid systems

[PATCH v2] sched/cache: Honor asym packing over cache aware scheduling on hybrid systems

[Tim Chen]

A regression was reported on an AMD Ryzen AI HX 370 running a cache
intensive Clang full-LTO link. The little cores run at a much lower
frequency (3.3 GHz vs 5.1 GHz) and have only half of the L3 cache
(8 MB vs 16 MB), so pinning such a task to the little-core LLC
hurts twice, and full-LTO builds slow down dramatically compared to
pre-cache-aware-scheduling kernels.

Asym packing and cache aware scheduling express conflicting placement
strategies. Asym packing wants a task to run on the highest priority CPU,
whereas cache aware scheduling wants to co-locate the tasks of a process
on one LLC regardless of the priority of CPUs in that LLC.

When asym packing tries to migrate task to an idle core that has higher
priority than source cpu, let asym packing win.  Moving tasks to a higher
performing idle core will buy more performance than cache co-location.

Prioritize asym packing over LLC balancing for regular and active load
balancing.

Fixes: 23b2b5ccc45c ("sched/cache: Introduce helper functions to enforce LLC migration policy")
Reported-by: Klaus Kusche <klaus.kusche@computerix.info>
Closes: https://lore.kernel.org/lkml/2180ea5a-eb28-4152-8d4d-cd00b0c24b2e@computerix.info/
Suggested-by: Kayra Cizmeci <kayracizmeci@gmail.com>
Tested-by: Klaus Kusche <klaus.kusche@computerix.info>
Tested-by: Ricardo Neri <ricardo.neri@intel.com>
Cc: stable@vger.kernel.org # 7.2.x
Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
---

Notes:
    v2: Ensure asym packing condition of idle CPU is fulfilled in
    can_migrate_llc_task() when bypassing LLC check (Kayra Cizmeci).
    v2 tested by Ricardo and Klaus offline.
    v1 link: https://lore.kernel.org/lkml/221f8b0345328c4b26b65daff4d3eec56a32b06d.1790617047.git.tim.c.chen@linux.intel.com/

 kernel/sched/fair.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 57360f5cdde4f..0301e27be35f5 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10934,6 +10934,8 @@ static inline bool task_misfits_asym_cpu(struct lb_env *env, struct task_struct
 	return false;
 }
 
+static inline bool sched_asym(struct sched_domain *sd, int dst_cpu, int src_cpu);
+
 /*
  * Check if task p can migrate from source LLC to
  * destination LLC in terms of cache aware load balance.
@@ -10958,6 +10960,10 @@ static enum llc_mig can_migrate_llc_task(struct lb_env *env,
 	if (cpu < 0 || cpus_share_cache(src_cpu, dst_cpu))
 		return mig_unrestricted;
 
+	/* Prioritize asym packing to idle core over cache awareness */
+	if (env->idle && sched_asym(env->sd, dst_cpu, src_cpu))
+		return mig_unrestricted;
+
 	/* skip cache aware load balance for too many threads */
 	if (invalid_llc_nr(grp, p, dst_cpu) ||
 	    exceed_llc_capacity(grp, dst_cpu)) {
@@ -12154,6 +12160,15 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,
 	    sgs->group_misfit_task_load)
 		return false;
 
+	/*
+	 * On asym packing domains, if the destination CPU
+	 * has higher priority than all CPUs in the source group,
+	 * prioritize asym packing.
+	 */
+	if ((env->sd->flags & SD_ASYM_PACKING) &&
+	    sgs->group_asym_packing)
+		return false;
+
 	/*
 	 * Skip cache aware tagging if nr_balanced_failed is sufficiently high.
 	 * Threshold of cache_nice_tries is set to 1 higher than nr_balance_failed
@@ -13569,12 +13584,12 @@ static int need_active_balance(struct lb_env *env)
 {
 	struct sched_domain *sd = env->sd;
 
-	if (alb_break_llc(env))
-		return 0;
-
 	if (asym_active_balance(env))
 		return 1;
 
+	if (alb_break_llc(env))
+		return 0;
+
 	if (imbalanced_active_balance(env))
 		return 1;
 
-- 
2.32.0

Re: [PATCH v2] sched/cache: Honor asym packing over cache aware scheduling on hybrid systems

[Kayra Cizmeci]

Hi Tim,

> +	/*
> +	 * On asym packing domains, if the destination CPU
> +	 * has higher priority than all CPUs in the source group,
> +	 * prioritize asym packing.
> +	 */
> +	if ((env->sd->flags & SD_ASYM_PACKING) &&
> +	    sgs->group_asym_packing)
> +		return false;
> +

I think that check is redundant. group_asym_packing is set by sched_group_asym() and bunch of other stuff,
but sched_group_asym() calls sched_asym() and sched_asym() calls sched_use_asym_prio() that has SD_ASYM_PACKING check
inside of it.

Ah.. Hope I'm not missing something...

Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com>

Now, I'm going to grab a cup of tea. And sleep.

Re: [PATCH v2] sched/cache: Honor asym packing over cache aware scheduling on hybrid systems

[Chen Yu]

On Mon, Oct 05, 2026 at 11:29:53AM -0700, Tim Chen wrote:
> Date: Mon,  5 Oct 2026 11:29:53 -0700
> From: Tim Chen <tim.c.chen@linux.intel.com>
> To: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@redhat.com>
> Cc: Tim Chen <tim.c.chen@linux.intel.com>, Chen Yu <yu.c.chen@intel.com>,
>  Mario Limonciello <mario.limonciello@amd.com>, Vishal Badole
>  <Vishal.Badole@amd.com>, linux-kernel@vger.kernel.org, x86@kernel.org,
>  platform-driver-x86@vger.kernel.org, K Prateek Nayak
>  <KPrateek.Nayak@amd.com>, Ricardo Neri <ricardo.neri@intel.com>, Kayra
>  Cizmeci <kayracizmeci@gmail.com>, stable@vger.kernel.org, Vincent Guittot
>  <vincent.guittot@linaro.org>, Juri Lelli <juri.lelli@redhat.com>, Klaus
>  Kusche <klaus.kusche@computerix.info>
> Subject: [PATCH v2] sched/cache: Honor asym packing over cache aware
>  scheduling on hybrid systems
> X-Mailer: git-send-email 2.32.0
> 
> A regression was reported on an AMD Ryzen AI HX 370 running a cache
> intensive Clang full-LTO link. The little cores run at a much lower
> frequency (3.3 GHz vs 5.1 GHz) and have only half of the L3 cache
> (8 MB vs 16 MB), so pinning such a task to the little-core LLC
> hurts twice, and full-LTO builds slow down dramatically compared to
> pre-cache-aware-scheduling kernels.
> 
> Asym packing and cache aware scheduling express conflicting placement
> strategies. Asym packing wants a task to run on the highest priority CPU,
> whereas cache aware scheduling wants to co-locate the tasks of a process
> on one LLC regardless of the priority of CPUs in that LLC.
> 
> When asym packing tries to migrate task to an idle core that has higher
> priority than source cpu, let asym packing win.  Moving tasks to a higher
> performing idle core will buy more performance than cache co-location.
> 
> Prioritize asym packing over LLC balancing for regular and active load
> balancing.
> 
> Fixes: 23b2b5ccc45c ("sched/cache: Introduce helper functions to enforce LLC migration policy")
> Reported-by: Klaus Kusche <klaus.kusche@computerix.info>
> Closes: https://lore.kernel.org/lkml/2180ea5a-eb28-4152-8d4d-cd00b0c24b2e@computerix.info/
> Suggested-by: Kayra Cizmeci <kayracizmeci@gmail.com>
> Tested-by: Klaus Kusche <klaus.kusche@computerix.info>
> Tested-by: Ricardo Neri <ricardo.neri@intel.com>
> Cc: stable@vger.kernel.org # 7.2.x
> Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
> ---
>

I leveraged AI to test it on top of 7.3.0-rc2 using an emulated hybrid setup on a symmetric
AMD Ryzen 9 8945HX (Zen 4, 16C/32T, two 32 MB L3):

By hacking the AMD pstate driver and QOS to limit the L3 cache ways:

  - LLC0 "big/fast":   CPUs 0-7,16-23, max freq 5.46 GHz, 16 L3 ways (32 MB),
                       ITMT prefcore ranking 236
  - LLC1 "little/slow": CPUs 8-15,24-31, max freq 3.29 GHz, 8 L3 ways (16 MB),
                       ITMT prefcore ranking 100

Workload: an 8-thread pointer-chase ring (24 MB working set).

This patch works as expected:
Before the patch:
                 cache-aware ON       cache-aware OFF
  throughput     ~230 M/s             ~530 M/s           (-56.6%)
  placement      6 of 8 threads       8 threads LLC0
                 stuck on LLC1


After the patch:
                 cache-aware ON       cache-aware OFF
  throughput     520.7 +- 18.3 M/s    512.9 +- 16.6 M/s   (+1.5%, noise)
  placement      8 threads LLC0       8 threads LLC0


Tested-by: Chen Yu <yu.c.chen@intel.com>

thanks,
Chenyu 

Re: [PATCH v2] sched/cache: Honor asym packing over cache aware scheduling on hybrid systems

[Tim Chen]

On Tue, 2026-10-06 at 16:59 +0300, Kayra Cizmeci wrote:
> Hi Tim,
> 
> > +	/*
> > +	 * On asym packing domains, if the destination CPU
> > +	 * has higher priority than all CPUs in the source group,
> > +	 * prioritize asym packing.
> > +	 */
> > +	if ((env->sd->flags & SD_ASYM_PACKING) &&
> > +	    sgs->group_asym_packing)
> > +		return false;
> > +
> 
> I think that check is redundant. group_asym_packing is set by sched_group_asym() and bunch of other stuff,
> but sched_group_asym() calls sched_asym() and sched_asym() calls sched_use_asym_prio() that has SD_ASYM_PACKING check
> inside of it.


Thanks.  Yes, we can remove the SD_ASYM_PACKING check as the check was done when group_asym_packing was set.

Tim

> 
> Ah.. Hope I'm not missing something...
> 
> Reviewed-by: Kayra Cizmeci <kayracizmeci@gmail.com>
> 
> Now, I'm going to grab a cup of tea. And sleep.
> 
> 

Re: [PATCH v2] sched/cache: Honor asym packing over cache aware scheduling on hybrid systems

[Tim Chen]

On Tue, 2026-10-06 at 23:30 +0800, Chen Yu wrote:
> On Mon, Oct 05, 2026 at 11:29:53AM -0700, Tim Chen wrote:
> > Date: Mon,  5 Oct 2026 11:29:53 -0700
> > From: Tim Chen <tim.c.chen@linux.intel.com>
> > To: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@redhat.com>
> > Cc: Tim Chen <tim.c.chen@linux.intel.com>, Chen Yu <yu.c.chen@intel.com>,
> >  Mario Limonciello <mario.limonciello@amd.com>, Vishal Badole
> >  <Vishal.Badole@amd.com>, linux-kernel@vger.kernel.org, x86@kernel.org,
> >  platform-driver-x86@vger.kernel.org, K Prateek Nayak
> >  <KPrateek.Nayak@amd.com>, Ricardo Neri <ricardo.neri@intel.com>, Kayra
> >  Cizmeci <kayracizmeci@gmail.com>, stable@vger.kernel.org, Vincent Guittot
> >  <vincent.guittot@linaro.org>, Juri Lelli <juri.lelli@redhat.com>, Klaus
> >  Kusche <klaus.kusche@computerix.info>
> > Subject: [PATCH v2] sched/cache: Honor asym packing over cache aware
> >  scheduling on hybrid systems
> > X-Mailer: git-send-email 2.32.0
> > 
> > A regression was reported on an AMD Ryzen AI HX 370 running a cache
> > intensive Clang full-LTO link. The little cores run at a much lower
> > frequency (3.3 GHz vs 5.1 GHz) and have only half of the L3 cache
> > (8 MB vs 16 MB), so pinning such a task to the little-core LLC
> > hurts twice, and full-LTO builds slow down dramatically compared to
> > pre-cache-aware-scheduling kernels.
> > 
> > Asym packing and cache aware scheduling express conflicting placement
> > strategies. Asym packing wants a task to run on the highest priority CPU,
> > whereas cache aware scheduling wants to co-locate the tasks of a process
> > on one LLC regardless of the priority of CPUs in that LLC.
> > 
> > When asym packing tries to migrate task to an idle core that has higher
> > priority than source cpu, let asym packing win.  Moving tasks to a higher
> > performing idle core will buy more performance than cache co-location.
> > 
> > Prioritize asym packing over LLC balancing for regular and active load
> > balancing.
> > 
> > Fixes: 23b2b5ccc45c ("sched/cache: Introduce helper functions to enforce LLC migration policy")
> > Reported-by: Klaus Kusche <klaus.kusche@computerix.info>
> > Closes: https://lore.kernel.org/lkml/2180ea5a-eb28-4152-8d4d-cd00b0c24b2e@computerix.info/
> > Suggested-by: Kayra Cizmeci <kayracizmeci@gmail.com>
> > Tested-by: Klaus Kusche <klaus.kusche@computerix.info>
> > Tested-by: Ricardo Neri <ricardo.neri@intel.com>
> > Cc: stable@vger.kernel.org # 7.2.x
> > Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
> > ---
> > 
> 
> I leveraged AI to test it on top of 7.3.0-rc2 using an emulated hybrid setup on a symmetric
> AMD Ryzen 9 8945HX (Zen 4, 16C/32T, two 32 MB L3):
> 
> By hacking the AMD pstate driver and QOS to limit the L3 cache ways:
> 
>   - LLC0 "big/fast":   CPUs 0-7,16-23, max freq 5.46 GHz, 16 L3 ways (32 MB),
>                        ITMT prefcore ranking 236
>   - LLC1 "little/slow": CPUs 8-15,24-31, max freq 3.29 GHz, 8 L3 ways (16 MB),
>                        ITMT prefcore ranking 100
> 
> Workload: an 8-thread pointer-chase ring (24 MB working set).
> 
> This patch works as expected:
> Before the patch:
>                  cache-aware ON       cache-aware OFF
>   throughput     ~230 M/s             ~530 M/s           (-56.6%)
>   placement      6 of 8 threads       8 threads LLC0
>                  stuck on LLC1
> 
> 
> After the patch:
>                  cache-aware ON       cache-aware OFF
>   throughput     520.7 +- 18.3 M/s    512.9 +- 16.6 M/s   (+1.5%, noise)
>   placement      8 threads LLC0       8 threads LLC0
> 
> 
> Tested-by: Chen Yu <yu.c.chen@intel.com>

Thanks for validating the patch.

Tim

> 
> thanks,
> Chenyu