Cpu profiles (1/3): CPUID adjustments #50

olivereanderson · 2025-12-09T16:54:01Z

This is the first of three PRs that have been split off from #25

This PR contains all the functionality needed to launch CHV with a CPU profile (currently limited to Intel Skylake and Sapphire rapids processors), minus the fact that MSR restrictions are still missing.

The next two PRs will contain CPUID definitions and the CPU profiles generation tool respectively.

Note: This contains a lot of code, because of the generated JSON files corresponding to the CPU profiles. The JSON files are currently pretty printed in order to make debugging easier at this relatively early stage.

The feature in more detail

Why do we even need this?

Recall that software is usually developed to run on a variety of processors with various features. In order for the software to dynamically discover which hardware features may be utilized one typically uses the CPUID instruction to query the CPU for information. In some cases one can also obtain CPU information via so called MSRs (model specific registers), but we will leave those out of this discussion for the time being.

Consider now the case that a guest is running some workload on host A which then gets migrated to host B where host B has a different processor than A. If this is a live migration (i.e. it is performed while the software is running), then the guest's workload can easily run into a time of check to time of use error.

Luckily hypervisors are able to manipulate what CPUID returns to guests when called which we can and will take advantage of in order to make live migration safer. Indeed if there is a subset of CPU features and properties that
are shared by all CPUs in a cluster, then if all hosts restrict themselves to that subset then live migration suddenly becomes a lot less problematic (although still not an entirely solved problem).

CPU profiles at a very high level

Using an existing CPU profile

In this PR we patch cloud hypervisor so that users may specify a CPU profile on the command line. This will in turn restrict what CPUID values the guest obtains and may thus affect functionality and performance, but may still be the best tool in the box when live migration to (subtly) different hardware is desired.

From a user perspective one simply includes the profile=<desired cpu profile> argument to the cpus parameter e.g.

--cpus boot=8,profile=sapphire-rapids

in order to utilize the Intel Sapphire rapids server CPU profile. If the host has hardware considered to be compatible with the chosen profile, then cloud hypervisor should work just like before, with the exception that guest's may see certain other values when inspecting CPUID.

The design in a bit of detail

The implementation is based on the understanding that we do not only need to work with bit sets, but we also need to manipulate some multi-bit values. Indeed, some CPUID output values indicate how many leaves there are (such as for instance leaf 0, sub-leaf 0, EAX), while others may tell you the number of sub-leaves, or the size of a state component for instance. Note that if the CPUID instruction is executed with an invalid leaf, some processors will return the data for the highest basic information leaf. In a live-migration setting this could easily lead to a time of check to time of use error!

Since we cannot simply deal with bit sets (at least to the best of our understanding) we unfortunately end up with a rather more complex solution than what we had initially hoped for.

CPU Profile policies

The idea is that we have a static list describing all known values one can obtain from CPUID (on an Intel CPU) and also such a list of the values defined by KVM (within the leaf range reserved for hypervisors). Note that this list is not included in this PR, but will be the part of our next PR.

From the aforementioned list of CPUID values we built a tool to generate CPU profiles when executed on the hardware that one wishes to be compatible with. The CPU profile generation tool (which will be revealed in a follow up PR) produces
data (currently JSON files) that we load at runtime and use to adjust the CPUID entries that guests will see when executing the CPUID instruction.

Immediate Follow up tasks

Introduce a least common denominator test that can run in CI.
Make CPU profiles take MSR restrictions into account. The first approximation would perhaps be to store whatever is returned when the compatibility_target calls [KVM_GET_MSR_FEATURE_INDEX_LIST]https://docs.kernel.org/virt/kvm/api.html#kvm-get-msr-index-list-kvm-get-msr-feature-index-list) and set a filter that by default denies anything not listed there.
Generate more thorough migration/cpuid compatibility checks than what (already exists upstream) by using the MigrationCompatibilityRequirements in the cpuid definitions introduced here.

Since enabling AMX tile state components affect the result returned by `Hypervisor::get_supported_cpuid` we want this enabled prior to checking CPUID compatibility between the source and destination VMs. Although this is not required today, it is necessary in order for the upcoming CPU profiles correctly, and it will also be necessary once the check_cpuid_compatibility checks are extended to take state components into account. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

Temporary workaround until we switch over to the WIP fix upstream Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

phip1611 · 2025-12-10T13:56:37Z

hypervisor/src/arch/x86/mod.rs

-            XSAVE_FAM_LENGTH
-                .set(fam_length)
-                .expect("This should only be set once");
+            let _ = XSAVE_FAM_LENGTH.set(fam_length);


commit message Temporary workaround until we switch over to the WIP fix upstream - is this intentional?

Yes.

Once cloud-hypervisor#7534 is merged then we can remove this code.

phip1611

Amazing, outstanding work! Great git commit history. I left a few remarks.

arch/src/x86_64/cpuid_definitions/mod.rs

arch/src/x86_64/cpu_profile.rs

phip1611 · 2025-12-10T14:03:14Z

arch/src/x86_64/cpu_profile.rs

+
+        if !amx {
+            // In this case we will need to wipe out the AMX tile state components (if they are included in the profile)
+            for adj in data.adjustments.iter_mut() {


would it make sense to have some sort of a generic wipe out bits from CPU id function here?

I would prefer not to make drastic changes like that at this stage. This is all rather complex and there is no clean way of doing things, at least not without refactoring the world.

One relatively simple alternative would be to have two profiles for every supported CPU (one with and one without AMX). Then this wouldn't be necessary at all, but if more complicated features like that get introduced in the future you end up with a combinatorial explosion of profiles that need to be generated.

at least not without refactoring the world.

👍🏻 alright, i'm fine with it

arch/src/x86_64/mod.rs

These data structures are required to define CPU profiles. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

We want CPU profiles to keep a record of the hypervisor type and cpu vendor that they are intended to work with. This is made more convenient if all of these types implement common traits (used for serialization). Signed-Off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

We introduce essential data structures together with basic functionality that is necessary to apply a CPU profile to a host. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

We integrate the CPU profile into the various configs that ultimately get set by the user. This quickly ends up involving multiple files, luckily Rust helps us find which ones via compilation errors. Signed-Off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

If a CPU profile is configured it should result in guests seeing a restricted subset of CPUID. This is what we finally achieve in this commit. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

We include CPU profiles corresponding to Intel Skylake and Sapphire rapids server that we generated using our WIP CPU profile generation tool. Signed-of-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

phip1611

LGTM!

phip1611 · 2025-12-15T09:43:25Z

arch/src/x86_64/cpu_profile.rs

 pub enum CpuProfile {
    #[default]
    Host,
+    #[cfg(feature = "kvm")]


is it the right thing to feature gate them for KVM?

Yes and no.

Adjusting CPUID to what is set in the profile is in principle hypervisor agnostic. However, the adjustments error if the hypervisor's call to Hypervisor::get_supported_cpuid doesn't return entries for leaf values the CPU profile expects.

If there is a big drift between which CPU features are supported by KVM and MSHV, then since we are currently only generating CPU profiles with KVM as an input to the generation tool, it would then be unlikely that the CPU profiles work with the MSHV hypervisor and it is perhaps then better not to advertise them to the user in that case.

I see two ways forward:

Generate CPU profiles with both hypervisors. We can definitely make this invisible to the user API and just select the profile corresponding to the hypervisor that is currently in use at runtime. If we choose this path then I think it is also OK that some profiles may only be available for one hypervisor.

We hope that the two hypervisors support essentially the same set of CPU features (at least among those that make sense in terms of live migration) and we just keep generating CPU profiles with whatever KVM supports.

Please reach out to Thomas and ask him about this. We do not have to take action in this PR but I think Thomas knows more background here and we can then persist this in some form of ticket

olivereanderson changed the base branch from main to gardenlinux December 9, 2025 16:54

olivereanderson force-pushed the cpu-profiles-cpuid branch from e352825 to 323094e Compare December 9, 2025 16:56

olivereanderson force-pushed the cpu-profiles-cpuid branch 2 times, most recently from 1e7385c to 85f573c Compare December 9, 2025 17:14

hypervisor: Permit enabling AMX tile state components more than once

0eb0dcf

Temporary workaround until we switch over to the WIP fix upstream Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

olivereanderson force-pushed the cpu-profiles-cpuid branch 2 times, most recently from a90c74b to 9ea0f49 Compare December 9, 2025 17:35

olivereanderson mentioned this pull request Dec 10, 2025

CPU profiles (2/3): Cpuid definitions #51

Closed

phip1611 assigned olivereanderson Dec 10, 2025

phip1611 self-requested a review December 10, 2025 13:49

olivereanderson changed the title ~~Cpu profiles: CPUID~~ Cpu profiles: CPUID (1/3) Dec 10, 2025

phip1611 reviewed Dec 10, 2025

View reviewed changes

olivereanderson changed the title ~~Cpu profiles: CPUID (1/3)~~ Cpu profiles (1/3): CPUID Dec 10, 2025

phip1611 requested changes Dec 10, 2025

View reviewed changes

olivereanderson force-pushed the cpu-profiles-cpuid branch from 9ea0f49 to 09b1c0e Compare December 11, 2025 08:23

olivereanderson changed the title ~~Cpu profiles (1/3): CPUID~~ Cpu profiles (1/3): CPUID adjustments Dec 11, 2025

olivereanderson force-pushed the cpu-profiles-cpuid branch from 09b1c0e to ac4d939 Compare December 11, 2025 10:09

olivereanderson added 2 commits December 11, 2025 11:17

arch: Initial data structures for describing CPUID parameters

a77f6c5

These data structures are required to define CPU profiles. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

olivereanderson force-pushed the cpu-profiles-cpuid branch from ac4d939 to ed96e07 Compare December 11, 2025 10:17

olivereanderson added 2 commits December 11, 2025 11:57

arch: CpuProfile data structures

f62dc8d

We introduce essential data structures together with basic functionality that is necessary to apply a CPU profile to a host. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com

olivereanderson force-pushed the cpu-profiles-cpuid branch from ed96e07 to 22a2319 Compare December 11, 2025 10:59

This was referenced Dec 11, 2025

CPU profiles (2/3): CPUID definitions #54

Closed

Cpu profiles (2/3): CPUID definitions #55

Draft

olivereanderson force-pushed the cpu-profiles-cpuid branch from 22a2319 to ca3273a Compare December 12, 2025 13:20

olivereanderson mentioned this pull request Dec 12, 2025

CPU Profiles (Part 1: CPUID) #25

Closed

6 tasks

olivereanderson force-pushed the cpu-profiles-cpuid branch from ca3273a to b1b2db6 Compare December 12, 2025 15:38

olivereanderson force-pushed the cpu-profiles-cpuid branch from b1b2db6 to 1b94318 Compare December 12, 2025 16:40

phip1611 approved these changes Dec 15, 2025

View reviewed changes

phip1611 reviewed Dec 15, 2025

View reviewed changes

phip1611 requested a review from amphi December 15, 2025 14:15

Cpu profiles (1/3): CPUID adjustments #50

Are you sure you want to change the base?

Cpu profiles (1/3): CPUID adjustments #50

Uh oh!

Conversation

olivereanderson commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The feature in more detail

Why do we even need this?

CPU profiles at a very high level

Using an existing CPU profile

The design in a bit of detail

CPU Profile policies

Immediate Follow up tasks

Uh oh!

phip1611 Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

olivereanderson Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phip1611 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phip1611 Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

olivereanderson Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

phip1611 Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phip1611 left a comment

Choose a reason for hiding this comment

Uh oh!

phip1611 Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

olivereanderson Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

phip1611 Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

olivereanderson commented Dec 9, 2025 •

edited

Loading

olivereanderson Dec 10, 2025 •

edited

Loading

phip1611 Dec 15, 2025 •

edited

Loading