What is the state of DFL+OPAE implementation - Intel PAC N3000

[hwspeedy]

Hi,

I am trying to upgrade from Centos 7 (kernel 3.x) running the old altera spi edition for FPGA BMC communication, to a debian 12 testing (Kernel 6.5) with OPAE build from github master. The card is Intel PAC N3000V.

I have compiled the kernel to include DFL (Unfortunately not included as modules in debian 12 testing/trixie yet.)

This kernel version 6.5.10 should include the latest DFL - it seams to have been included in kernel 6.1?

All four XXSV710 network interfaces work.

I can see the card is registered and uses the DFL modules:

root@Linux:~# lsmod|grep dfl
uio_dfl 12288 0
dfl_n3000_nios 16384 0
dfl_fme_br 12288 0
uio 28672 1 uio_dfl
dfl_fme_region 16384 0
dfl_fme_mgr 16384 1
dfl_afu 40960 0
dfl_fme 57344 0
dfl_pci 16384 0
dfl 45056 6 dfl_pci,uio_dfl,dfl_fme,dfl_fme_br,dfl_n3000_nios,dfl_afu
fpga_region 20480 3 dfl_fme_region,dfl_fme,dfl
fpga_bridge 20480 4 dfl_fme_region,fpga_region,dfl_fme,dfl_fme_br
fpga_mgr 28672 4 dfl_fme_region,fpga_region,dfl_fme_mgr,dfl_fme

But i still do not have communication through fpgainfo:

root@EpycWorkLinux:~# fpgainfo fme
Intel FPGA Programmable Acceleration Card N3000
board_n3000.c:115:read_bmcfw_version() ERROR : Failed to get read object
board_n3000.c:186:read_max10fw_version() ERROR : Failed to get read object
Board Management Controller, MAX10 NIOS FW version:
Board Management Controller, MAX10 Build version:
//****** FME ******//
Interface : DFL
Object Id : 0xEC00000
PCIe s:b:d.f : 0000:44:00.0
Vendor Id : 0x8086
Device Id : 0x0B30
SubVendor Id : 0x8086
SubDevice Id : 0x0000
Socket Id : 0x01
Ports Num : 07
Bitstream Id : 0xFFFFFFFFFFFFFFFF
Bitstream Version : 15.15.15
Pr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb
Boot Page : factory

PS: I have stopped the FPGAD to ensure it does not power down the Intel PAC N3000V wrongly due to missing sensor information. Did not change anything.

PS: The card is running FPGA/BMC/NIOS stock images v1.3.16.

Am i doing something obviously wrong?

Or is this the expected current state of the development for DFL and OPAE?

In advance thanx for any info.

[hwspeedy]

Updating DFL kernel modules to github latest version
Still no BMC values (fpgainfo power,temp,mac,etc).

The bitstream id+version+boot page reports correctly.
mmlink also seams to work.

root@Linux:~/inteldevstack/rtl/n3000_1_3_v1.5.7/hello_afu/hw/pac/remote_debug$ fpgainfo fme
Intel FPGA Programmable Acceleration Card N3000
board_n3000.c:115:read_bmcfw_version() ERROR : Failed to get read object
board_n3000.c:186:read_max10fw_version() ERROR : Failed to get read object
Board Management Controller, MAX10 NIOS FW version:
Board Management Controller, MAX10 Build version:
//****** FME ******//
Interface : DFL
Object Id : 0xED00000
PCIe s:b:d.f : 0000:87:00.0
Vendor Id : 0x8086
Device Id : 0x0B30
SubVendor Id : 0x8086
SubDevice Id : 0x0000
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x23000110010310
Bitstream Version : 0.2.3
Pr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb
Boot Page : user

So is DFL+OPAE not yet working for the original card starting these projects? or am I doing something wrong?

Thanx for any info.

[pcolberg]

Support for the N3000 card has been discontinued, but you can try building the latest linux-dfl kernel from the fpga-ofs-dev-6.1-lts branch to check whether the upstream kernel support is incomplete or whether the cause is some other issue. Besides fpgainfo, what output do you get from sensors?

[hwspeedy]

@pcolberg

Who are you in regards to this project? - Please explain Support for the N3000 card has been discontinued, thanx.

Yes, The latest submitted version from this github - added to kernel 6.7rc1 - I have tried that with the kernel 6.5 as written above - OPAE is not up2date for DFL and DFL does not contain all the info from BMC controller.

fpgainfo from original OPAE tools
[root@wolli ~]# fpgainfo bmc
Board Management Controller, MAX10 NIOS FW version D.2.0.21
Board Management Controller, MAX10 Build version D.2.0.6
//****** BMC SENSORS ******//
Object Id : 0xF500000
PCIe s:b:d.f : 0000:06:00.0
Device Id : 0x0b30
Numa Node : 0
Ports Num : 01
Bitstream Id : 0x23000110010310
Bitstream Version : 0.2.3
Pr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb
( 1) Board Power : 71.68 Watts
( 2) 12V Backplane Current : 3.28 Amps
( 3) 12V Backplane Voltage : 11.94 Volts
( 4) 1.2V Voltage : 1.20 Volts
( 6) 1.8V Voltage : 1.83 Volts
( 8) 3.3V Voltage : 3.27 Volts
(10) FPGA Core Voltage : 0.90 Volts
(11) FPGA Core Current : 16.67 Amps
(12) FPGA Core Temperature : 66.00 Celsius
(13) Board Temperature : 60.50 Celsius
(14) QSFP A Voltage : 1.67 Volts
(15) QSFP A Temperature : 83.00 Celsius
(24) 12V AUX Current : 2.72 Amps
(25) 12V AUX Voltage : 11.96 Volts
(37) QSFP B Voltage : N/A
(38) QSFP B Temperature : N/A
(44) Retimer A Core Temperature : 68.00 Celsius
(45) Retimer A Serdes Temperature : 69.00 Celsius
(46) Retimer B Core Temperature : 0.00 Celsius
(47) Retimer B Serdes Temperature : 0.00 Celsius

DFL finest and newest
As you can see in the N3000 board code for DFL only a hwmon with temp is made for sensors. The sysfs shows wrong value for nios_firmware.

Haven't used this DFL much but seams it crashes if you remove the FPGA PCIe interface by OPAE programming tools. - Is that supposed to happen?

[hwspeedy]

Found a couple of missing modules
Suggest that the kernel configuration FPGA_DFL_NIOS_INTEL_PAC_N3000 depends on the UIO_DFL and the SPI controller that it needs for cooperation with fpgainfo in opae.

SPI modules loaded
root@EpycWorkLinux:/usr/src/linux-source-6.5# lsmod |grep spi
intel_m10_bmc_spi 12288 0
regmap_spi_avmm 12288 1 intel_m10_bmc_spi
intel_m10_bmc_core 16384 1 intel_m10_bmc_spi
spi_altera_platform 12288 0
spi_altera_core 12288 1 spi_altera_platform

Seams to be too many SPI controllers? - maybe wrong module naming? - Any suggestions?

fpgainfo phy looks better with correct versions and access to link status:
root@EpycWorkLinux:/usr/src/linux-source-6.5# fpgainfo phy
Intel FPGA Programmable Acceleration Card N3000
Board Management Controller, MAX10 NIOS FW version: D.2.0.21
Board Management Controller, MAX10 Build version: D.2.0.6
//* * * * * * PHY * * * * * //
Interface : DFL
Object Id : 0xED00000
PCIe s:b:d.f : 0000:87:00.0
Vendor Id : 0x8086
Device Id : 0x0B30
SubVendor Id : 0x8086
SubDevice Id : 0x0000
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x23000110010310
Bitstream Version : 0.2.3
Pr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb
// * * * * * PHY GROUP 0 * * * * * //
Direction : Host side
Speed : 25 Gbps
Number of PHYs : 4
// * * * * * PHY GROUP 1 * * * * * //
Direction : Line side
Speed : 40 Gbps
Number of PHYs : 4
// * * * * * Intel C827 Retimer * * * * * *//
Port0 25G : Down
Port1 25G : Down
Port2 25G : Down
Port3 25G : Down
Retimer A Version : 101c.1064
Retimer B Version : 0.0

fpgainfo power, temp, bmc are still no good:
oot@EpycWorkLinux:/usr/src/linux-source-6.5# fpgainfo bmc
Intel FPGA Programmable Acceleration Card N3000
Board Management Controller, MAX10 NIOS FW version: D.2.0.21
Board Management Controller, MAX10 Build version: D.2.0.6
//****** BMC SENSORS ******//
Interface : DFL
Object Id : 0xED00000
PCIe s:b:d.f : 0000:87:00.0
Vendor Id : 0x8086
Device Id : 0x0B30
SubVendor Id : 0x8086
SubDevice Id : 0x0000
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x23000110010310
Bitstream Version : 0.2.3
Pr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb
metrics_utils.c:587:enum_fpga_metrics() ERROR : Failed to Enum Power and Thermal metrics.
metrics.c:78:xfpga_fpgaGetNumMetrics() ERROR : Failed to Discover Metrics
Error getting number of metrics: not found
Error reading metrics from BMC: not found

PS: Still only a temperature sensor registered in sensors.

[hwspeedy]

Latest update on DFL+OPAE on debian 12 Testing/Trixie - Kernel 6.5

After finding all the missing modules, that is used by N3000 Nios module, the OPAE fgpainfo tool works and so does the added hwmon sensors.

See PR: Drivers/FPGA/DFL - Intel PAC N3000 missing modules select #22

fpgainfo bmc
root@EpycWorkLinux:~# fpgainfo bmc
Intel FPGA Programmable Acceleration Card N3000
Board Management Controller, MAX10 NIOS FW version: D.2.0.21
Board Management Controller, MAX10 Build version: D.2.0.6
//* * * * * * BMC SENSORS * * * * * *//
Interface : DFL
Object Id : 0xED00000
PCIe s:b:d.f : 0000:C4:00.0
Vendor Id : 0x8086
Device Id : 0x0B30
SubVendor Id : 0x8086
SubDevice Id : 0x0000
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x23000110010310
Bitstream Version : 0.2.3
Pr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb
( 1) 1.2V Voltage : 1.20 Volts
( 2) 12V AUX Current : 2.84 Amps
( 3) QSFP1 Temperature : N/A
( 4) QSFP1 Supply Voltage : N/A
( 5) Retimer B SerDes Temperature : 0.00 Celsius
( 6) Board Power : 74061.00 Watts
( 7) 12V AUX Voltage : 11.96 Volts
( 8) Board Temperature : 58.00 Celsius
( 9) Retimer A Temperature : 79.50 Celsius
(10) FPGA Core Voltage : 0.90 Volts
(11) 1.8V Voltage : 1.81 Volts
(12) FPGA Core Current : 17.95 Amps
(13) FPGA Die Temperature : 76.00 Celsius
(14) Retimer A SerDes Temperature : 81.00 Celsius
(15) 12V Backplane Voltage : 12.13 Volts
(16) 3.3V Voltage : 3.26 Volts
(17) 12V Backplane Current : 3.07 Amps
(18) QSFP0 Temperature : 83.00 Celsius
(19) QSFP0 Supply Voltage : 1.67 Volts
(20) Retimer B Temperature : 0.00 Celsius

Output of sensors
dfl_fme_thermal-isa-0000
Adapter: ISA adapter
temp1: +79.0°C

n3000bmc_hwmon-isa-0000
Adapter: ISA adapter
QSFP0 Supply Voltage: 1.67 V
QSFP1 Supply Voltage: N/A
FPGA Core Voltage: 900.00 mV
12V Backplane Voltage: 12.13 V
1.2V Voltage: 1.20 V
12V AUX Voltage: 11.97 V
1.8V Voltage: 1.81 V
3.3V Voltage: 3.26 V
Board Temperature: +58.0°C (high = +85.0°C, hyst = +75.0°C)
(crit = +100.0°C, hyst = +90.0°C)
FPGA Die Temperature: +76.0°C (high = +90.0°C, crit = +100.0°C)
QSFP0 Temperature: +83.0°C (high = +65.0°C, crit = +67.0°C)
ERROR: Can't get value of subfeature temp4_max: Can't read
ERROR: Can't get value of subfeature temp4_crit: Can't read
QSFP1 Temperature: N/A (high = +0.0°C, crit = +0.0°C)
Retimer A Temperature: +79.5°C
Retimer A SerDes Temperature: +80.5°C
Retimer B Temperature: +0.0°C
Retimer B SerDes Temperature: +0.0°C
Board Power: 72.03 W
FPGA Core Current: 17.83 A
12V Backplane Current: 3.15 A
12V AUX Current: 2.82 A

fpgainfo phy
root@EpycWorkLinux:~# fpgainfo phy
Intel FPGA Programmable Acceleration Card N3000
Board Management Controller, MAX10 NIOS FW version: D.2.0.21
Board Management Controller, MAX10 Build version: D.2.0.6
//* * * * * * PHY * * * * * //
Interface : DFL
Object Id : 0xED00000
PCIe s:b:d.f : 0000:C4:00.0
Vendor Id : 0x8086
Device Id : 0x0B30
SubVendor Id : 0x8086
SubDevice Id : 0x0000
Socket Id : 0x00
Ports Num : 01
Bitstream Id : 0x23000110010310
Bitstream Version : 0.2.3
Pr Interface Id : f3c99413-5081-4aad-bced-07eb84a6d0bb
// * * * * * PHY GROUP 0 * * * * * //
Direction : Host side
Speed : 25 Gbps
Number of PHYs : 4
// * * * * * PHY GROUP 1 * * * * * //
Direction : Line side
Speed : 40 Gbps
Number of PHYs : 4
// * * * * * Intel C827 Retimer * * * * * *//
Port0 25G : Up
Port1 25G : Down
Port2 25G : Down
Port3 25G : Down
Retimer A Version : 101c.1064
Retimer B Version : 0.0

Issues/WIP
Seams like the secure update, flash update is changed from initial OPAE version 1.3
Seams like the system crashes when OPAE reprogram PCIe part - I was kind of seeing this as a part of what the DFL framework should resolve. Needs more investigation.

The Altera Max10 BMC is integrated into the linux kernel but the FPGA/CPLD and NIOS source code is missing. That should be released to OFS - it's Open FPGA Stack - Right?

[nathanborak]

Closing this issue due to inactivity/age. If you would like to access the BMC source code, it is found at https://github.com/otcshare/ofs-bmc and requires access granted by contacting your Intel sales rep.