Brice Goglin wrote:
> David Addison a écrit :
> 
>> Hi,
>>
>> here is a patch we use to integrate the Quadrics NICs into the Linux 
>> kernel.
>> The patch adds hooks to the Linux VM subsystem so that registered 
>> 'IOPROC'
>> devices can be informed of page table changes.
>> This allows the Quadrics NICs to perform user RDMAs safely, without 
>> requiring
>> page pinning. Looking through some of the recent IB and Ammasso 
>> discussions,
>> it may also prove useful to those NICs too.
> 
> 
> Hi,
> 
> I worked on a similar patch to help updating a registration cache on
> Myrinet. I came to the problem of deciding between registering ioproc
> to the entire address space (1) or only to some VMA (2).
> You're doing (1), I tried (2).
> 
> (2) avoids calling ioproc hooks for all pages that are never involved
> in any communication. This might be good if the amount of pages that
> are involved is not too high and if the coproc_ops cost is a little bit
> high.
> Do you have any numbers about this in real applications on QsNet ?
> 
We have always taken approach (1) as it seems to be the simplest method
and offers the model where the whole user process space can be made available
for RDMA operations.
Admittedly, the update calls for pages which are not going to be used for
RDMA operations is an overhead, but the device driver can elect not to
present update functions and instead rely on pre-registration of comms
buffers.
For invalidates the device driver will have knowledge of the registered
regions and can quickly ignore irrelevant unloads.

With HPC applications in general we find the memory image is pretty static
over the life of the job and hence most of the costs are taken as the
pages are created during job startup and initialisation.

> I see two drawback in (2).
> First, it requires to play with the list of ioproc_ops when VMA are
> merged or split. Actually, it's not that bad since the list often
> contains only 1 ioproc_ops.
> Secondly, you have to add the ioproc to all involved VMA at some point.
> It's easy when the API asks the application to register, you just add
> the ioproc_ops to the target VMA during registration. But, I guess it's
> not easy with Quadrics, right ?
> 
In the past we have allowed dynamic page faulting of all application exposed
memory via RDMA operations.
However, our newer libraries do now implement a registration cache so we
can pre-load translations to our NIC MMU (or pin, if kernel invalidate hooks
are not available).
However, I still prefer model (1) as it allows both implementations and
appears to be much simpler in terms of the linux kernel changes required.

> 
> I see in your patch that ioproc are not inherited during fork.
> How do you support fork in your driver/lib then ?
> What if a COW page is given to the son and the copy to the father
> while some IO are being processed ? Do you require the application to
> call a specific routine after forking ?
> Don't you think it might be good to add a hook in the fork code
> so that ioproc are inherited or duplicated pages are invalidated
> in the card ?
> 
Yes, on fork() our programming model is for the child to attach to the
device again. The QsNet model has a NIC MMU context for each process
so it makes sense for each process to attach and have independent
IOPROC and NIC memory management.

But you're right, there should be IOPROC hooks to ensure that the device
cannot write to COW pages after the fork. I've added a new callback for this;
ioproc_wrprotect_page() called in copy_one_pte(), and a new revised patch
is attached (Jesper: with whitespace corrections too ;-)

> Regards,
> Brice

Thanks for your comments,

Cheers
David.