Re: [PATCH RFC 0/2] sched: Introduce idle SMT priority for asymmetric capacity systems

[Mete Durlu <meted@linux.ibm.com>]

Hi,
sorry for the late reply, I have been sick since Friday.
I though about what has been written for a bit.

>>> Well, when we see steal time, it usually means there is high 
>>> contention and
>>> we are using more vCPUs at this moment than possible.
>>> So we mark them as non-preferred. Note that they are idle cpus. But 
>>> non-preferred.
>>>
>>> That means preferred CPUs may have more than 1 task. If we spill over 
>>> if there
>>> is idle non-preferred CPUs, we will be back to square one. No?
>>
>> I agree that CPUs excluded by the steal governor should remain avoided 
>> even when
>> all preferred CPUs are busy.
>>
>> I'm wondering if we should have two levels: cpu_preferred_mask acting 
>> as a hard
>> affinity constraint and an entitlement-based soft affinity within that 
>> mask? The
> 
> That not what entitlement usually means. At least not PowerVM. when one 
> creates VM,
> they assign VP=Virtual Cores and EC=Entitled Cores. and Hypervisor 
> ensure at least EC
> worth of cores are always assigned to that VM. There is no need of 
> priority within that
> entitlement.

Wouldn't it make sense to first fully load entitled cores before
moving onto virtual cores on Power? On s390 we would definitely
benefit from it on some cases.

>> soft affinity would prefer high-entitlement CPUs: first fully idle 
>> cores, then
>> idle SMT siblings. Once those CPUs are busy, tasks could spill onto
> 
> I think when one has high steal time and hence reduced preferred CPUs, 
> there will
> likely be no fully idle cores within preferred CPUs.

Shrikanth, wouldn't something like below still make sense?

> Note that current logic takes out last set of CPUs and find_new_ilb 
> iterates from
> the beginning. So if there is idle core within the preferred CPUs, it 
> will be chosen.
> 
>> lower-entitlement CPUs still inside cpu_preferred_mask, again 
>> preferring fully
>> idle cores over idle SMT siblings. If all CPUs inside 
>> cpu_preferred_mask are
>> busy, tasks would queue there. The soft affinity would never override the
>> governor's restriction by spilling onto CPUs it excluded due to 
>> contention.
>>
> As we were discussing, as long as threads don't end up on non-preferred 
> CPUs,
> we might be okay. With the above, find_new_ilb will becomes very complex 
> IMHO
> and I am not sure if it is worth it.
> 
> likely what we need is,
> - Fully idle preferred Core.
> - idle preferred CPU.
> - any idle CPU.

Andrea does this resemble what you were describing;

1-) (steal time == 0)
====================================================================
cpu_preffered_mask = active_cpus
cpu_soft_affinity = 0

All cpus are available for scheduler to pick and use.
Idle cores are preferred in case of asymmetric capacities.

2-) (steal time > 0) && (steal time <= initial_threshold*)
====================================================================
cpu_preffered_mask = active_cpus
cpu_soft_affinity = entitled cpus

In this case entitled cores get fully filled (SMT siblings first)
before spilling load onto the rest of the cpus on preferred mask.
*Initial threshold is a value I just devised. Ideally arch specific
and dynamically configurable, setting it to just 0 should also
be an option.

3-) (steal time > initial_threshold)
====================================================================
cpu_preffered_mask = entitled cpus + steal governor's modifications
cpu_soft_affinity = entitled cpus

Carrying the logic from case 2 but shrinking the preferred mask
according to the increased steal time value.

Shrikanth/Andrea,
does what I describe above make sense and would it make a reasonable
addition to the vCPU backoff mechanism?

*Considering guests with hardware based topology and underlying
hypervisor scheduling vCPUs per core;
I'd expect this approach to allow for guests having a smaller
footprint for longer and steal time to start appearing later than
usual as the entitled cores are fully utilized before touching other
cores.

*This is the case for s390 for example. I am not sure if it is the same
for Power or any other architecture.