Incorrect cache-info output with nosmt linux kernel command line

[jbd]

Hello,

I'm using the main branch of today (3c8b153) and I've got incorrect output in cache-info on a machine booted with the "nosmt" kernel command line parameter:

WITH the "nosmt" parameter (note the weird "shared by XX processors" at the end of each line):

[NOSMT]$ ./cache-info 
Max cache size (upper bound): 16777216 bytes
L1 instruction cache: 96 x 32 KB, 8-way set associative (64 sets), 64 byte lines, shared by 97 processors
L1 data cache: 96 x 32 KB, 8-way set associative (64 sets), 64 byte lines, shared by 97 processors
L2 unified cache: 96 x 512 KB (inclusive), 8-way set associative (1024 sets), 64 byte lines, shared by 97 processors
L3 unified cache: 24 x 16 MB (exclusive), 16-way set associative (16384 sets), 64 byte lines, shared by 100 processors

WITHOUT the "nosmt" parameter:

[SMT]$ ./cache-info 
Max cache size (upper bound): 16777216 bytes
L1 instruction cache: 96 x 32 KB, 8-way set associative (64 sets), 64 byte lines, shared by 2 processors
L1 data cache: 96 x 32 KB, 8-way set associative (64 sets), 64 byte lines, shared by 2 processors
L2 unified cache: 96 x 512 KB (inclusive), 8-way set associative (1024 sets), 64 byte lines, shared by 2 processors
L3 unified cache: 24 x 16 MB (exclusive), 16-way set associative (16384 sets), 64 byte lines, shared by 8 processors

lscpu output WITH the "nosmt" parameter:

Architecture:         x86_64
CPU op-mode(s):       32-bit, 64-bit
Byte Order:           Little Endian
CPU(s):               192
On-line CPU(s) list:  0-95
Off-line CPU(s) list: 96-191
Thread(s) per core:   1
Core(s) per socket:   48
Socket(s):            2
NUMA node(s):         2
Vendor ID:            AuthenticAMD
BIOS Vendor ID:       Advanced Micro Devices, Inc.
CPU family:           23
Model:                49
Model name:           AMD EPYC 7552 48-Core Processor
BIOS Model name:      AMD EPYC 7552 48-Core Processor                
Stepping:             0
CPU MHz:              2851.220
CPU max MHz:          2200.0000
CPU min MHz:          1500.0000
BogoMIPS:             4400.33
Virtualization:       AMD-V
L1d cache:            32K
L1i cache:            32K
L2 cache:             512K
L3 cache:             16384K
NUMA node0 CPU(s):    0-47
NUMA node1 CPU(s):    48-95
Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es

lscpu output WITHOUT the "nosmt" parameter:

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              192
On-line CPU(s) list: 0-191
Thread(s) per core:  2
Core(s) per socket:  48
Socket(s):           2
NUMA node(s):        2
Vendor ID:           AuthenticAMD
BIOS Vendor ID:      Advanced Micro Devices, Inc.
CPU family:          23
Model:               49
Model name:          AMD EPYC 7552 48-Core Processor
BIOS Model name:     AMD EPYC 7552 48-Core Processor                
Stepping:            0
CPU MHz:             2200.000
CPU max MHz:         2200.0000
CPU min MHz:         1500.0000
BogoMIPS:            4400.15
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-47,96-143
NUMA node1 CPU(s):   48-95,144-191
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es

The difference is are on the On-line/Off-line CPU(s) lists.

It caught me by surprise with a SIGFPE when using the NNPACK project (Maratyszcza/NNPACK#218).

Thank you for your help.

[jbd]

Please find attached the output of cache-info with and without the nosmt option, when cpuinfo is built with debug log level.

cache-info.nosmt.txt
cache-info.smt.txt

[prashanthswami]

Looking at the x86 logic (I'm horribly unfamiliar with it, so bear with me here :)), it looks like we'd only accidentally generate N caches if we assigned them each different L1I IDs. Digging through the code paths here, I don't see anywhere we would accidentally mark the same ID as different IDs, so that leads me to believe that our l1i_id calculation fails somehow.

# How we calculate the apic_bits for each cache in `cpuinfo_x86_decode_cache_properties`.
const uint32_t cores = 1 + ((regs.eax >> 14) & UINT32_C(0x00000FFF));
const uint32_t apic_bits = bit_length(cores);
...
# How we later mask to acquire the L1I ID.
const uint32_t apic_id = linux_processors[i].apic_id;
const uint32_t l1i_id = apic_id & ~bit_mask(x86_processor.cache.l1i.apic_bits);

I'd hazard a guess that this means either:

1. The apic_id is wrong for all processors with 'nosmt' enabled?
2. The apic_bits is wrong for all caches with 'nosmt' enabled?

[Flamefire]

I just ran into the same issue: AMD EPYC 9334 with SMT :

$ ./cache-info 
Max cache size (upper bound): 33554432 bytes
L1 instruction cache: 64 x 32 KB, 8-way set associative (64 sets), 64 byte lines, shared by 65 processors
L1 data cache: 64 x 32 KB, 8-way set associative (64 sets), 64 byte lines, shared by 65 processors
L2 unified cache: 64 x 1 MB (inclusive), 8-way set associative (2048 sets), 64 byte lines, shared by 65 processors
L3 unified cache: 8 x 32 MB (exclusive), 16-way set associative (32768 sets), 64 byte lines, shared by 72 processors

So NNPACK reads a size of 32KB L1D shared by 65(???) threads

That off-by-one seems to stem from it counting by apic_id and the 64 physical cores shared the same apic_id (zero) with the first "virtual" core (processor index 64)

[Flamefire]

@jbd would you be able to test my PR for this: #291
@prashanthswami The cause I identified matches your 2nd guess:

The apic_id is wrong for all processors with 'nosmt' enabled?

There is already code to set a flag if and only if the APIC ID was determined for a processor. When SMT is disabled the hyperthreads (e.g. half the processors) don't have an APIC ID (defaults to zero) and hence don't have the flag. So the calculation of the L1I/D_id using a default APIC ID doesn't make much sense to me. This leads to the strange off-by-one error as the defaulted processors share the ID zero with the single "correct" one.