unbound vCPUs make for contentious cyclic juggling in vm_run

[iximeow]

over in byhve, vm_run tries to make sure that vCPU-specific resources (cyclics, etc) are kept on the same CPU as whichever is current. vm_localize_resources makes this happen, and up at the top of that is if (vcpu->lastloccpu == curcpu). if we're not on the same CPU, we'll have to move resources (such as vlapic_localize_resources moving a cyclic), and in the case of vlapic_localize_resources that's serialized through cpu_lock.

on a largeish system (256 threads) with a largeish VM (192 vCPUs), we're limited at somewhere in the range of 500-700k vm_run()/second because of this(!)

while we don't want to be doing 500-700k interrupts/second, ideally, we can save a bunch of system time by binding vCPUs to physical CPUs and ensuring if (vcpu->lastloccpu == curcpu). at the very least, perhaps binding vCPU threads to a smaller CPU group so the odds are improved.

[iximeow]

a proper treatment of this would have propolis-server take a list describing the vCPUs and the cores they should be bound to, and to do that allocation in the control plane as part of placing the VM. since we'd like this in 18, an interim option would be to just have propolis-{server,standalone} bind vCPUs to some range if the VM would have more than half of available vCPUs. by definition there can only be one VM that large on a sled, so that avoids the problem of multiple vCPUs being pinned to the same CPUs until we're explicit about this from the control plane.

[iximeow]

and to do that allocation in the control plane as part of placing the VM

thinking about the future propolis-server API a bit more while writing #991, this is a bit subtle. the vCPU->CPU mapping the control plane will chose will not be consistent across sleds, so the "proper treatment" will have to include data like MigrationTarget::replace_components that we use to amend a source instance spec into something suitable for the destination propolis-server.