Re: [PATCH RFC 0/2] sched: Introduce idle SMT priority for asymmetric capacity systems

[Shrikanth Hegde <sshegde@linux.ibm.com>]

Hi Mete.

On 10/12/26 6:02 PM, Mete Durlu wrote:
> Hi,
> sorry for the late reply, I have been sick since Friday.
> I though about what has been written for a bit.
> 

No issues. Take care.

>>>> Well, when we see steal time, it usually means there is high contention and
>>>> we are using more vCPUs at this moment than possible.
>>>> So we mark them as non-preferred. Note that they are idle cpus. But non-preferred.
>>>>
>>>> That means preferred CPUs may have more than 1 task. If we spill over if there
>>>> is idle non-preferred CPUs, we will be back to square one. No?
>>>
>>> I agree that CPUs excluded by the steal governor should remain avoided even when
>>> all preferred CPUs are busy.
>>>
>>> I'm wondering if we should have two levels: cpu_preferred_mask acting as a hard
>>> affinity constraint and an entitlement-based soft affinity within that mask? The
>>
>> That not what entitlement usually means. At least not PowerVM. when one creates VM,
>> they assign VP=Virtual Cores and EC=Entitled Cores. and Hypervisor ensure at least EC
>> worth of cores are always assigned to that VM. There is no need of priority within that
>> entitlement.
> 
> Wouldn't it make sense to first fully load entitled cores before
> moving onto virtual cores on Power? On s390 we would definitely
> benefit from it on some cases.
> 

Only when there is a high physical CPU contention.

>>> soft affinity would prefer high-entitlement CPUs: first fully idle cores, then
>>> idle SMT siblings. Once those CPUs are busy, tasks could spill onto
>>
>> I think when one has high steal time and hence reduced preferred CPUs, there will
>> likely be no fully idle cores within preferred CPUs.
> 
> Shrikanth, wouldn't something like below still make sense?
> 
>> Note that current logic takes out last set of CPUs and find_new_ilb iterates from
>> the beginning. So if there is idle core within the preferred CPUs, it will be chosen.
>>
>>> lower-entitlement CPUs still inside cpu_preferred_mask, again preferring fully
>>> idle cores over idle SMT siblings. If all CPUs inside cpu_preferred_mask are
>>> busy, tasks would queue there. The soft affinity would never override the
>>> governor's restriction by spilling onto CPUs it excluded due to contention.
>>>
>> As we were discussing, as long as threads don't end up on non-preferred CPUs,
>> we might be okay. With the above, find_new_ilb will becomes very complex IMHO
>> and I am not sure if it is worth it.
>>
>> likely what we need is,
>> - Fully idle preferred Core.
>> - idle preferred CPU.
>> - any idle CPU.
> 
> Andrea does this resemble what you were describing;
> 
> 1-) (steal time == 0)
> ====================================================================
> cpu_preffered_mask = active_cpus
> cpu_soft_affinity = 0
> 
> All cpus are available for scheduler to pick and use.
> Idle cores are preferred in case of asymmetric capacities.
> 
> 2-) (steal time > 0) && (steal time <= initial_threshold*)
> ====================================================================
> cpu_preffered_mask = active_cpus
> cpu_soft_affinity = entitled cpus
> 
> In this case entitled cores get fully filled (SMT siblings first)
> before spilling load onto the rest of the cpus on preferred mask.
> *Initial threshold is a value I just devised. Ideally arch specific
> and dynamically configurable, setting it to just 0 should also
> be an option.
> 
> 3-) (steal time > initial_threshold)
> ====================================================================
> cpu_preffered_mask = entitled cpus + steal governor's modifications
> cpu_soft_affinity = entitled cpus
> 
> Carrying the logic from case 2 but shrinking the preferred mask
> according to the increased steal time value.
> 
> Shrikanth/Andrea,
> does what I describe above make sense and would it make a reasonable
> addition to the vCPU backoff mechanism?
> 

I am confused. Are you talking about
1. picking CPU to run idle load balancing?
Or
2. actual task placement decisions?

If it is 1, we should do only if there is any good improvement with real life workloads IMHO,
as current find_new_ilb isn't aware of preferred CPUs. Even in that case just picking
an idle preferred CPU over an idle non-preferred Core is better than above i think.

If it is 2, I am not sure. getting soft affinity is kind of tricky with load balance.
It puts maximum effort in spreading the tasks. Since you have steal time, there is already
enough tasks to create physical CPU contention.

PS: Steal governor provides a tunable named low_threshold, which means any steal value
below is practically 0. So there is no soft affinity there.
> *Considering guests with hardware based topology and underlying
> hypervisor scheduling vCPUs per core;
> I'd expect this approach to allow for guests having a smaller
> footprint for longer and steal time to start appearing later than
> usual as the entitled cores are fully utilized before touching other
> cores.
> 
> *This is the case for s390 for example. I am not sure if it is the same
> for Power or any other architecture.