VM stopped while Migration with Maintenance Mode (KVM)

[svenvogel]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Management Server

CLOUDSTACK VERSION

Master but maybe 4.8 upwards

CONFIGURATION

OS / ENVIRONMENT

KVM with QEMU 2.12 and libwirtd 5.0.0, i think other other versions are also affected because not always there is the problem.

SUMMARY

STEPS TO REPRODUCE

1. Set a Host in Maintenance Mode
2. VM Migration is started
3. If a VM cant migrate it will hard stopped

after @andrijapanicsb started a discussion on mailing list "VMs crashing/stopped during live migration?" i opened this issue request to track down the problems.

@ustcweizhou gave some interesting hints to this problem.

i picked up specially your point three. i think its most interesting to track this down.

3. vm is stopped due to some cloudstack bugs. for example, when we put a
   host to maintenance, the vm will be stopped if (1) no other host is Up in
   same cluster, or (2) vm migration fails and no retry in cloudstack, or (3)
   multiple vms are migrated to same destination at the same time but there is
   no enough memory on the destination.

EXPECTED RESULTS

The expected result should be to stop the maintenance mode but dont switch off the virtual machines

ACTUAL RESULTS

virtual machines will be hard stopped

[svenvogel]

@ustcweizhou do you have some idea how we can these part 3?

[andrijapanicsb]

I believe the biggest problem I saw was VM crashing due to qemu bugs/issue (i.e. recently saw this one “qemu-kvm general protection error in libtcmalloc.so.4.4.5”) – can’t speak for other issues, since I don’t recall having them with ACS 4.5/4.8 with Ubuntu 14.04 (qemu 2.0 initially, later upgraded to 2.5) If not mistaken, there is indeed a behavior that if there is no enough capacities in the cluster (test with a single KVM host in the cluster), the VMs will be stopped – and that is a thing to change in my opinion. Maintenance should fail and host should be in ErrorInMaintenance mode. If you want to do maintenance of that host, put the storage into the maintenance mode and you are good to go. Even better, make an additional parameter like “forceMaintenance” which will indeed stop VMs in order to reach Maintenance mode. Othewise will simply be in ErrorInMaintenance. I really don’ recall having issues with 5 VMs/agent threads – and I’ve been doing a lot of maintenance with couple of hunders of VMs 2-3 years ago (ACS 4.8). This one should be reproduced first. Regards, Andrija Panic Andrija Panic Cloud Architect s: +44 20 3603 0540 | m: +381 69 272 3690 e: andrija.panic@shapeblue.com  |  w: www.shapeblue.com  |  t:  @shapeblue a: Amadeus House, Floral Street, London WC2E 9DPUK Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design & Build | CSForge - rapid IaaS deployment framework CloudStack Consulting | CloudStack Software Engineering CloudStack Infrastructure Support | CloudStack Bootcamp Training Courses

[weizhouapache]

@svenvogel @andrijapanicsb

I think @andrijapanicsb 's suggestion is good. Suppose we have a new parameter forceMaintenance in the command

(1) if there are vms running on the host. if there is no any other server in Up status and Enabled state in the cluster
(1.1) if forceMaintenance = true, stop all vms
(1.2) if forceMaintenance = false, throw exception, host is put to ErrorInMaintenance or Up status.

(2) if there are other servers in the cluster, all vms will be migrated away from the host
(2.1) if all vm migrations succeed, then host is put to Maintenance.
(2.2) if vm migration failed, then try to migrate vm to another host . If vm migration failed after 5 times (this can be configured)), then
(2.2.1) if forceMaintenance = true, schedule a task to stop/start the vm.
(2.2.2) if forceMaintenance = false, throw exception, host is put to ErrorInMaintenance.

any suggestion ?

[svenvogel]

@ustcweizhou @andrijapanicsb i think this sounds good.

one question.

(2.2.1) if forceMaintenance = true, schedule a task to stop/start the vm.

@ustcweizhou why we should stop and start them?

additional we encounter the following problem.

1. iso could not be mounted or storage pool on destination host, i dont know why this is a problem, manually mounting was no problem. maybe anyone has information about that?
2. if QEMU or Host Version version are different machine type will change the flags <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
   i think this should be on view of the infrastructure admin.

[andrijapanicsb]

@weizhouapache that sounds OK to me - I assume you handle capacity issues on a per VM basis.

@svenvogel 2.2.1 - if VM can't be live migrated - then stop it and start on another host to avoid permanent downtime - makes sense perfectly (if forceMaint=true)

[andrijapanicsb]

CC @anuragaw since you were aware of the issue and want to review any code produced...

[andrijapanicsb]

CC @rhtyd @PaulAngus @nvazquez

[weizhouapache]

@ustcweizhou @andrijapanicsb i think this sounds good.

one question.

(2.2.1) if forceMaintenance = true, schedule a task to stop/start the vm.

@ustcweizhou why we should stop and start them?

@svenvogel if vm cannot be migrated after 5 times, to continue with the maintenance (because forceMaintenance is true), we need to stop it and start it on another server. make sense ?

additional we encounter the following problem.

1. iso could not be mounted or storage pool on destination host, i dont know why this is a problem, manually mounting was no problem. maybe anyone has information about that?

yes, it is one root cause of failed migration.

2. if QEMU or Host Version version are different machine type will change the flags <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
   i think this should be on view of the infrastructure admin.

If hosts have different cpu model or qemu version, they should use custom cpu or cpu flags. It is not an issue in my point of view.

[andrijapanicsb]

Agree on that last point of QEMU or HOST versions - not an issue and has been like that forever (nor I think would be possible since that would change what the OS sees and might crash it).

[svenvogel]

@andrijapanicsb @ustcweizhou yes make sense. Agree. now i understand it.

[GabrielBrascher]

@andrijapanicsb “forceMaintenance” is something that could help indeed; in some specific corner cases it would be interesting to have such a workaround.

I believe the biggest problem I saw was VM crashing due to qemu bugs/issue (i.e. recently saw this one “qemu-kvm general protection error in libtcmalloc.so.4.4.5”) – can’t speak for other issues, since I don’t recall having them with ACS 4.5/4.8 with Ubuntu 14.04 (qemu 2.0 initially, later upgraded to 2.5) If not mistaken, there is indeed a behavior that if there is no enough capacities in the cluster (test with a single KVM host in the cluster), the VMs will be stopped – and that is a thing to change in my opinion. Maintenance should fail and host should be in ErrorInMaintenance mode. If you want to do maintenance of that host, put the storage into the maintenance mode and you are good to go. Even better, make an additional parameter like “forceMaintenance” which will indeed stop VMs in order to reach Maintenance mode. Othewise will simply be in ErrorInMaintenance.

[anuragaw]

My 2 cents - in case of single host adding forceMaintenance is really going to complicate code unnecessarily. Marking host to maintenance and migration jobs for VMs are separate background tasks. Putting a host into maintenance is not going to be a frequent operation inmho worth the code complexity.

Perhaps sticking to the general theory - If Migrations cannot be completed and host cannot be put to maintenance safely, we put the host to ErrorInMaintenance state. At the same time we log/throw exception for informing the admins.

Admins can put the VMs off or snapshot or whatever works best - then they can manually shutdown all VMs on a host as needed and retry maintenance transition.

Thoughts @andrijapanicsb , @svenvogel , @GabrielBrascher , @ustcweizhou?