An opportunity to fix IOMMU grouping here for PCI devices

[wrobelda]

Hi,

I was investigating the IOMMU group separation for my x710-DA2 multi-function NIC and it hit me that this project could actually fix an underlying issue with these Tiny devices, i.e. that all the NICs on these cards end up in the same IOMMU group:

01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
	Subsystem: Intel Corporation Ethernet 10G 2P X710 Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 2
(...)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
	Subsystem: Intel Corporation Ethernet Converged Network Adapter X710
	Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 2

This is down to the fact that the CPUs used by these desktops do not support Access Control Services (ACS), which would otherwise allow separating these into different groups. Sure, there's a known hack/workaround (pcie_acs_override), but it's risky and doesn't always work — and it indeed does not work with these particular Lenovo desktops (at least m920x and p350 that I owned).

That having said, the PCH's PCI bridge actually does support ACS, see:

root@pve2:~# lspci -vvv -s 00:1b.0
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0) (prog-if 00 [Normal decode])
	Subsystem: Lenovo Cannon Lake PCH PCI Express Root Port
(...)
	Capabilities: [140 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans-

Which means that if the x4 PCI slot was actually wired to the PCH instead of CPU-bound PCI lane, the NICs would be placed in separate IOMMU groups. Having them in separate groups has an obvious advantage, as it enables passing them through to separate VMs independently and in a memory-safe manner.

Which makes me wonder if there's a way you could design another version of this riser, this time having both m2 slots wired to PCI lanes and the PCI slot to PCH lane? To be honest, I think it actually makes more sense anyway to have both m2 drives wired the same way?