Re: [PATCH] irqchip/gic-v3-its: Balance initial LPI affinity across CPUs

[John Garry <john.garry@huawei.com>]

On 20/01/2020 17:42, Marc Zyngier wrote:

Hi Marc,

>>>      static u64 its_irq_get_msi_base(struct its_device *its_dev)
>>> @@ -2773,28 +2829,34 @@ static int its_irq_domain_activate(struct
>>> irq_domain *domain,
>>>    {
>>>    	struct its_device *its_dev = irq_data_get_irq_chip_data(d);
>>>    	u32 event = its_get_event_id(d);
>>> -	const struct cpumask *cpu_mask = cpu_online_mask;
>>> -	int cpu;
>>> +	int ret = 0, cpu = nr_cpu_ids;
>>> +	const struct cpumask *reqmask;
>>> +	cpumask_var_t mask;
>>>    -	/* get the cpu_mask of local node */
>>> -	if (its_dev->its->numa_node >= 0)
>>> -		cpu_mask = cpumask_of_node(its_dev->its->numa_node);
>>> +	if (irqd_affinity_is_managed(d))
>>> +		reqmask = irq_data_get_affinity_mask(d);
>>> +	else
>>> +		reqmask = cpu_online_mask;
>>>    -	/* Bind the LPI to the first possible CPU */
>>> -	cpu = cpumask_first_and(cpu_mask, cpu_online_mask);
>>> -	if (cpu >= nr_cpu_ids) {
>>> -		if (its_dev->its->flags & ITS_FLAGS_WORKAROUND_CAVIUM_23144)
>>> -			return -EINVAL;
>>> +	if (!alloc_cpumask_var(&mask, GFP_KERNEL))
>>> +		return -ENOMEM;
>>>    -		cpu = cpumask_first(cpu_online_mask);
>>> +	its_compute_affinity(d, reqmask, mask);
>>> +	cpu = its_pick_target_cpu(mask);
>>> +	if (cpu >= nr_cpu_ids) {
>>> +		ret = -EINVAL;
>>> +		goto out;
>>>    	}
>>>    +	atomic_inc(per_cpu_ptr(&cpu_lpi_count, cpu));
>>
>> I wonder if we should only consider managed interrupts in this
>> accounting?
>>
>> So cpu0 is effectively going to be excluded from the balancing, as it
>> will have so many lpis targeted.
> 
> Maybe, but only if the provided managed affinity gives you the
> opportunity of placing the LPI somewhere else. 

Of course, if there's no other cpu in the mask then so be it.

If the managed
> affinity says CPU0 only, then that's where you end up.
> 

If my debug code is correct (with the above fix), cpu0 had 763 
interrupts targeted on my D06 initially :)

But it's not just cpu0. I find initial non-managed interrupt affinity 
masks are set generally on cpu cluster/numa node masks, so the first 
cpus in those masks are bit over-subscribed, so then we may be spreading 
the managed interrupts over less cpus in the mask.

This is a taste of lpi distribution on my 96 core system:
cpu0 763
cpu1 2
cpu3 1
cpu4 2
cpu5 2
cpu6 0
cpu7 0
cpu8 2
cpu9 1
cpu10 0
...
cpu16 2
...
cpu24 8
...
cpu48 10 (numa node boundary)
...


>> And, for the others, even if we balance all the LPIs, won't irqbalance
>> (if running, obviously) can come along and fiddle with these
>> non-managed interrupt affinities anyway?
> 
> Of course, irqbalance will move things around. But that should be to
> CPUs that do not have too many screaming interrupts.
> 
> Thanks,
> 
>           M.

Cheers,
John

>