Re: [PATCH v9 05/18] PCI/P2PDMA: Document directional ACS routing

[Leon Romanovsky <leon@kernel.org>]

On Tue, Oct 06, 2026 at 04:32:05PM -0500, Bjorn Helgaas wrote:
> On Thu, Oct 01, 2026 at 02:55:13PM +0300, Leon Romanovsky wrote:
> > From: Leon Romanovsky <leonro@nvidia.com>
> > 
> > P2PDMA documentation describes ACS controls as path-wide, although Request
> > and Completion controls apply to different transaction directions and only
> > affect peer-versus-upstream decisions at the path divergence.
> > 
> > Document the fixed client and provider roles, the divergence port checked
> > for each TLP direction, and the conservative handling of unreadable ACS
> > state. Clarify which controls disable_acs_redir changes.
> > 
> > Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
> > Tested-by: Tushar Dave <tdave@nvidia.com>
> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> > ---
> >  Documentation/admin-guide/kernel-parameters.txt | 15 +++++++++------
> >  Documentation/driver-api/pci/p2pdma.rst         | 13 +++++++++++++
> >  2 files changed, 22 insertions(+), 6 deletions(-)
> > 
> > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > index 68647ff4bdd2..bc83e07dd5fc 100644
> > --- a/Documentation/admin-guide/kernel-parameters.txt
> > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > @@ -5291,12 +5291,15 @@ Kernel parameters
> >  		disable_acs_redir=<pci_dev>[; ...]
> >  				Specify one or more PCI devices (in the format
> >  				specified above) separated by semicolons.
> > -				Each device specified will have the PCI ACS
> > -				redirect capabilities forced off which will
> > -				allow P2P traffic between devices through
> > -				bridges without forcing it upstream. Note:
> > -				this removes isolation between devices and
> > -				may put more devices in an IOMMU group.
> > +				Each device specified will have the PCI ACS P2P
> > +				Request Redirect, Completion Redirect, and Egress
> > +				Control features forced off. This may allow P2P
> > +				traffic through bridges that would otherwise be
> > +				redirected upstream. This may allow P2P traffic
> > +				through bridges that would otherwise be redirected
> > +				upstream and thus this removes isolation between
> > +				devices and may cause affected devices to share
> > +				an IOMMU group.
> >  		config_acs=
> >  				Format:
> >  				<ACS flags>@<pci_dev>[; ...]
> > diff --git a/Documentation/driver-api/pci/p2pdma.rst b/Documentation/driver-api/pci/p2pdma.rst
> > index 80f8fec9b0e9..42b18610bf7d 100644
> > --- a/Documentation/driver-api/pci/p2pdma.rst
> > +++ b/Documentation/driver-api/pci/p2pdma.rst
> > @@ -15,6 +15,19 @@ then based on the ACS settings the transaction can route entirely within
> >  the PCIe hierarchy and never reach the root port. The kernel will evaluate
> >  the PCIe topology and always permit P2P in these well-defined cases.
> >  
> > +The client remains the PCIe requester when it reads or writes provider memory.
> > +Where the paths diverge, the kernel therefore evaluates P2P Request Redirect
> > +and Egress Control on the client-side port, and P2P Completion Redirect on the
> > +provider-side port for completions from a read. An enabled Egress Control is
> > +conservatively treated as a Request redirect.
> 
> I think "where the paths diverge" means the point where a port decides
> whether to route a TLP up through its Upstream Port (or to the RC) or
> back down via a sibling Downstream Port?

Yes.

> 
> I'm not sure p2pdma.rst includes the context to interpret "divergence"
> yet.  I think the code comments in [4/18] might also need a little
> more context about divergence.
> 
> IIUC, this divergence point is basically a static point where the ACS
> controls being applied makes a routing difference.

Right. Previously, we did not account for ACS bits, so any "junction"
automatically caused P2P to be considered unsupported.

Thanks

> 
> > +Below the divergence, the route toward the other branch is already upstream,
> > +so those P2P redirect controls do not affect it. Redirect controls for the
> > +reverse transaction directions do not affect the mapping. P2P DMA is routed
> > +through the host bridge when either applicable port redirects. If an ACS
> > +Control register cannot be read, P2P DMA is rejected because the kernel cannot
> > +establish a usable route.
> > +
> >  This evaluation assumes clients issue strictly ordered Requests carrying an
> >  Untranslated address. Its result is not defined when clients use Relaxed
> >  Ordering or issue ATS-translated Requests because those TLP attributes can
> > 
> > -- 
> > 2.55.0
> >