Fixing x2APIC on MSI boards for linux
Motivation⌗
x2APIC allowes for more efficient virtualization by avoiding MMIO exits and instead switching to MSRs for management of the APIC. It also allows for more than 255 cores, but that is not important yet for desktop CPUs. On the MSI motherboards I possess, (PRO X870-P WIFI and PRO B650M-A WIFI), x2APIC cannot be used by default, so virtualization overhead is heavier.
Unhiding the option⌗
On the MSI motherboards I possess, (PRO X870-P WIFI and PRO B650M-A WIFI), Local APIC Mode is a hidden AMD CBS setting.
To enable x2APIC, the UEFI firmware can be modified to unhide the full CBS menu (see UEFI-Editor for a guide). After unhiding the full CBS menu, the user can set the Local APIC Mode in UEFI.
Fixing linux⌗
Currently, linux defaults to x2APIC cluster mode, which for some reason I have not tried to figure out yet, behaves poorly on these MSI motherboards. A quick workaround I found was to set x2apic_phys in the kernel commandline. This forces using the APIC in physical mode instead of the broken cluster mode.
Linux behaviour⌗
Linux will default to cluster mode if the flags at offset 0x70 in the FACP table don’t require x2APIC physical mode. Below are the flags that are set for my MSI motherboard.
[070h 0112 004h] Flags (decoded below) : 0003C5A5
WBINVD instruction is operational (V1) : 1
WBINVD flushes all caches (V1) : 0
All CPUs support C1 (V1) : 1
C2 works on MP system (V1) : 0
Control Method Power Button (V1) : 0
Control Method Sleep Button (V1) : 1
RTC wake not in fixed reg space (V1) : 0
RTC can wake system from S4 (V1) : 1
32-bit PM Timer (V1) : 1
Docking Supported (V1) : 0
Reset Register Supported (V2) : 1
Sealed Case (V3) : 0
Headless - No Video (V3) : 0
Use native instr after SLP_TYPx (V3) : 0
PCIEXP_WAK Bits Supported (V4) : 1
Use Platform Timer (V4) : 1
RTC_STS valid on S4 wake (V4) : 1
Remote Power-on capable (V4) : 1
Use APIC Cluster Model (V4) : 0
Use APIC Physical Destination Mode (V4) : 0
Hardware Reduced (V5) : 0
Low Power S0 Idle (V5) : 0
The flag we care about is Use APIC Physical Destination Mode (V4) : 0. Since it is not set, Linux reads the FACP table and decides to use cluster mode.
Firmware patch workaround⌗
Setting the kernel cmdline option x2apic_phys does solve the issue, but it seems janky. Since MSI allows us to flash unsigned firmware using BIOS Flashback, we can patch the function that generates the table, and reflash a modified firmware.
I was able to find the module that generates the FACP table by searching the firmware binary for the byte-swapped flags from the dumped table:
0003C5A5 → A5C50300
Then the PE32 image was extracted and opened in IDA. An immediate value search was performed for 0003C5A5, giving one result:
Using this, and the knowledge that this function appears to be reused on many boards, we obtain the following signature for this code segment:
C7 ?? 70 A5 C5 03 00 -> C7..70A5C50300
Translating to the UEFIPatch format:
16D0A23E-C09C-407D-A14A-AD058FDD0CA1 10 P:C7..70A5C50300:C7..70A5C50B00
I tested this and found that the patch applies on a few different vendors of varying AM5 chipset types.
Performance⌗
Benchmarks suggest that for a 96 core system, there is about a 10% performance regression for IPI delivery when using physical mode for the Xen Hypervisor. For a desktop class CPU with 16 cores, this performance overhead is likely much smaller.
At this time I have not completed my own benchmarks to verify this claim. I have also not validated that this solution results in the correct behaviour for interrupt delivery, use at your own risk.
Remarks⌗
Given that this patch applies on other vendors, and assuming they have working cluster mode x2APIC, then this is a bandaid patch to a deeper issue of broken x2APIC cluster mode on affected MSI motherboards. I have not yet looked into why x2APIC cluster mode does not function on these MSI boards. Windows behaves fine in all Local APIC Mode settings.