Re: [PATCH 06/19] sched/fair: Assign preferred LLC ID to processes

["Chen, Yu C" <yu.c.chen@intel.com>]

(Copied the question from Vern as the email seems to not reach LKML)

On 10/14/2025 2:09 AM, Tim Chen wrote:
 > On Mon, 2025-10-13 at 17:10 +0800, vernhao wrote:
 >>
 >> Tim Chen<tim.c.chen@linux.intel.com> wrote:
 >> With cache-aware scheduling enabled, each task is assigned a
 >> preferred LLC ID. This allows quick identification of the LLC domain
 >> where the task prefers to run, similar to numa_preferred_nid in
 >> NUMA balancing.
 >>
 >> Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>

[snip]

 >> +
 >> + if (mm->mm_sched_cpu != -1)
 >> + mm_sched_llc = per_cpu(sd_llc_id, mm->mm_sched_cpu);
 >>
 >> In high-concurrency multi-threaded scenarios, not all threads handle
 >> same events, so their hot data in the LLC is not completely shared.
 >> Therefore, if every thread's preferred LLC is migrated to the LLC
 >> pointed to by mm->mm_sched_cpu, this would lead to the incorrect
 >> assumption that all threads prefer the same LLC, thereby intensifying
 >> competition between LLCs.
 >
 > Yes, that's the reason why we stop aggregating to the preferred LLC
 > once the the utilization of the
 > LLC becomes too high relative to the other LLCs.
 >
 > If you know your threads characteristics before hand on which of them
 > share data together, you probably can use cgroup/cpuset
 > from user space to separate out the threads.
 >
 > There's not enough info from occupancy data for OS to group
 > the threads by data sharing. Perhaps an alternative if NUMA balancing
 > is on is to group tasks by their task numa group instead of by mm.
 >
 > That would incur the page scanning overhead etc and make
 > cache aware scheduling be dependent on NUMA balancing.
 >
 >
 >>
 >> So I'm wondering, why not move ‘mm->mm_sched_cpu’ to ‘task_struct’,
 >> so that each thread can individually track its preferred LLC?
 >> What are the losses in doing so?
 >
 > You would need a way to group related tasks together and put them
 > on the same LLC.  Either group them by mm or some other means.
 >

While Vern's use case is common in production environments, switching
to per-task_struct prefer_llc might not aggregate the threads to
dedicated LLCs. It is possible that each thread will stick to its
old LLC because the thread was forked there and the occupancy is
high on that old LLC. As a result, threads are randomly "pinned"
to different LLCs.

The question becomes: how can we figure out the threads that share
data? Can the kernel detect this, or get the hint from user space?

Yes, the numa_group in NUMA load balancing indicates
that several tasks manipulate the same page, which could be an
indicator. Besides, if task A frequently wakes up task B, does it
mean A and B have the potential to share data? Furthermore, if
task A wakes up B via a pipe, it might also indicate that A has
something to share with B. I just wonder if we can introduce a
structure to gather this information together.

thanks,
Chenyu