server: Decouple service (compute) offering and disk offering (#5008)

Currently, our compute offerings and disk offerings are tightly coupled with respect to many aspects. For example, if a compute offering is created, a corresponding disk offering entry is also created with the same ID as the reference. Also creating compute offering takes few disk-related parameters which anyway goes to the corresponding disk offering only. I think this design was initially made to address compute offering for the root volume created from a template. Also changing the offering of a volume is tightly coupled with storage tags and has to be done in different APIs either migrateVolume or resizeVolume. Changing of disk offering should be seamless and should consider new storage tags, new size and place the volume in appropriate state as defined in disk offering.

more details are mentioned here https://cwiki.apache.org/confluence/display/CLOUDSTACK/Compute+offering+and+disk+offering+refactoring

* Schema changes and disk offering column change from "type" to "compute_only"

* Few more changes

* Decoupled service offering and disk offering

* Remove diskofferingid from vminstance VO

* Decouple service offering and disk offering states

* diskoffering getsize() is only for strict disk offerings

* Fix deployVM flow

* Added new API params to compute offering creation

* Add diskofferingstrictness to serviceoffering vo under quota

* Added overrideDiskOfferingId parameter in deploy VM API which will override disk offering for the root disk both in template and ISO case

Added diskSizeStrictness parameter in create Disk offering API which will decide whether to restrict resize or disk offering change of a volume

* Fix User vm response to show proper service offering and disk offerings

* Added disk size strictness in disk offering response

* Added disk offering strictness to the service offering response

* Remove comments

* Added UI changes for Disk offering strictness in add compute offering form and Disk size strictness in add disk offering form

* Added diskoffering details to the service offering response

* Added UI changes in deployvm wizard to accept override disk offering id

* Fix delete compute offering

* Fix VM deployment from custom service offering

* Move uselocalstorage column access from service offering to disk offering

* UI: Separated compute and disk releated parameters in add compute offering wizard, also added association to disk offering

* Fixed diskoffering automatic selection on add compute offering wizard

* UI: move compute only toggle button outside the box in add compute offering wizard

* Added volumeId parameter to listDiskOfferings API and the disksizestrictness flag of the current disk offering is honored while list disk offerings

* Added configuration parameter to decide whether to check volume tags on the destination storagepool during migration

* Added disk offering change checks during resize volume operation

* Added new API changeofferingforVolume API and corresponding changes

* Add UI form for changeOfferingForVolume API

* Fix UI conflicts

* Fix service offering usage as disk offering

* Fix unit test failures

* fix user_vm_view

* Addressed review comments

* Fixed service_offering_view

* Fix service offering edit flow

* Fix service offering constructor to address custom offering

* Fix domain_router_view to get proper service offering id

* Removed unused import

* Addressed review comments and fixed update service offering flow with storage tags

* Added marvin test cases for checking disk offering strictness

* review comments addressed

* Remove system_use column from disk offering join

* update volume_view to update system_use column from service offering and not disk offering

* Fix changeOfferingForVolume API for custom disk offering

* Fix global setting implementation

* Fix list volumes, after changing system_use column from disk offering to service offering in volume_view

* Changes for override root disk offering in deployvm wizard in case of custom offering

* Fix a unit test case

* Fixed recent unit test cases with new serviceofferingvo constructor

* Fix unit test in VolumeApiServiceImpl

* Added storage id for the list disk offering API and corresponding UI changes in migrateVolume and changeOfferingForVolume flow

* Rename global configuration parameter from storage.pool.tags.disk.offering.strictness to match.storage.pool.tags.with.disk.offering

* Fix smoke test failures

* Added tool tip for migrate volume UI form

* Address review comments and fix UI form of deploy VM in case of ISO.

* Fixed resize volume UI form for data disk

* UI changes to disable override root disk size when override root disk offering is enabled

* UI fix in deploy vm wizard

* Fix listdiskoffering after rebasing with main

* Fixed UI in migrate and changeofferingfor volume to handle empty disk offering list
Removed the volume's current disk offering from listDiskOffering response list

* Added custom Iops to resize volume form and removed the current disk offering during change offering for volume UI form

* Fix false response on updateDiskOffering API

* Added search field for changeofferingforvolume UI form

* Fix resize volume and migrate volume to update volume path if DRS is applied on volume in datastore cluster

* Removed DB changes from 4.16 upgrade file

* Resolving merge conflicts with main 4.17

* Added support for auto migration and auto resize of the root volume upon changing the service offering for VM.

* UI: Added automigrate checkbox in scale VM form

* Addes since attributes to new API params

* Added shrinkOK parameter to changeofferingforvolume API

* Added shrinkOk param to UI in changeOfferingforVolume form

* Added shrinkOk flag to scaleVM and changeServiceForVirtualMachines and UI form

* Removed old foreign key constraint on IDs of service offering and disk offering

* Allow resize and automigrate of root volume if required in all cases of service offering change

* Allow only resize to higher disk size from UI

* Fixing vue syntax error

* Make UI changes to provide root disk size box when the linked disk offering is of custom

* Converted from check box to toggle in scale VM, changeoffering, resize and migrate volume forms

* Fix resize volume operation to update the VM settings

* Fix migratevolume form to pick selected storage pool id in list diskofferings API

Author: harikrishna-patnala

api/src/main/java/com/cloud/event/EventTypes.java

@@ -258,6 +258,7 @@ public class EventTypes {
     public static final String EVENT_VOLUME_UPDATE = "VOLUME.UPDATE";
     public static final String EVENT_VOLUME_DESTROY = "VOLUME.DESTROY";
     public static final String EVENT_VOLUME_RECOVER = "VOLUME.RECOVER";
+    public static final String EVENT_VOLUME_CHANGE_DISK_OFFERING = "VOLUME.CHANGE.DISK.OFFERING";
 
     // Domains
     public static final String EVENT_DOMAIN_CREATE = "DOMAIN.CREATE";
@@ -768,6 +769,7 @@ public class EventTypes {
         entityEventDetails.put(EVENT_VOLUME_RESIZE, Volume.class);
         entityEventDetails.put(EVENT_VOLUME_DESTROY, Volume.class);
         entityEventDetails.put(EVENT_VOLUME_RECOVER, Volume.class);
+        entityEventDetails.put(EVENT_VOLUME_CHANGE_DISK_OFFERING, Volume.class);
 
         // Domains
         entityEventDetails.put(EVENT_DOMAIN_CREATE, Domain.class);

api/src/main/java/com/cloud/offering/DiskOffering.java

@@ -34,10 +34,6 @@ enum State {
         Inactive, Active,
     }
 
-    enum Type {
-        Disk, Service
-    };
-
     State getState();
 
     enum DiskCacheMode {
@@ -61,8 +57,6 @@ public String toString() {
 
     String getName();
 
-    boolean isSystemUse();
-
     String getDisplayText();
 
     ProvisioningType getProvisioningType();
@@ -152,5 +146,7 @@ public String toString() {
 
     void setCacheMode(DiskCacheMode cacheMode);
 
-    Type getType();
+    boolean isComputeOnly();
+
+    boolean getDiskSizeStrictness();
 }

api/src/main/java/com/cloud/offering/ServiceOffering.java

@@ -25,37 +25,34 @@
 /**
  * offered.
  */
-public interface ServiceOffering extends DiskOffering, InfrastructureEntity, InternalIdentity, Identity {
-    public static final String consoleProxyDefaultOffUniqueName = "Cloud.com-ConsoleProxy";
-    public static final String ssvmDefaultOffUniqueName = "Cloud.com-SecondaryStorage";
-    public static final String routerDefaultOffUniqueName = "Cloud.Com-SoftwareRouter";
-    public static final String elbVmDefaultOffUniqueName = "Cloud.Com-ElasticLBVm";
-    public static final String internalLbVmDefaultOffUniqueName = "Cloud.Com-InternalLBVm";
+public interface ServiceOffering extends InfrastructureEntity, InternalIdentity, Identity {
+    static final String consoleProxyDefaultOffUniqueName = "Cloud.com-ConsoleProxy";
+    static final String ssvmDefaultOffUniqueName = "Cloud.com-SecondaryStorage";
+    static final String routerDefaultOffUniqueName = "Cloud.Com-SoftwareRouter";
+    static final String elbVmDefaultOffUniqueName = "Cloud.Com-ElasticLBVm";
+    static final String internalLbVmDefaultOffUniqueName = "Cloud.Com-InternalLBVm";
     // leaving cloud.com references as these are identifyers and no real world addresses (check against DB)
 
-    public enum StorageType {
+    enum State {
+        Inactive, Active,
+    }
+
+    enum StorageType {
         local, shared
     }
 
-    @Override
     String getDisplayText();
 
-    @Override
     Date getCreated();
 
-    @Override
-    String getTags();
-
     /**
      * @return user readable description
      */
-    @Override
     String getName();
 
     /**
      * @return is this a system service offering
      */
-    @Override
     boolean isSystemUse();
 
     /**
@@ -98,12 +95,6 @@ public enum StorageType {
      */
     Integer getMulticastRateMbps();
 
-    /**
-     * @return whether or not the service offering requires local storage
-     */
-    @Override
-    boolean isUseLocalStorage();
-
     /**
      * @return tag that should be present on the host needed, optional parameter
      */
@@ -117,5 +108,35 @@ public enum StorageType {
 
     boolean isDynamic();
 
+    void setState(ServiceOffering.State state);
+
+    ServiceOffering.State getState();
+
+    void setName(String name);
+
+    String getUniqueName();
+
+    void setUniqueName(String uniqueName);
+
+    void setDisplayText(String displayText);
+
+    boolean isCustomized();
+
+    void setCustomized(boolean customized);
+
+    Date getRemoved();
+
+    void setRemoved(Date removed);
+
+    void setSortKey(int key);
+
+    int getSortKey();
+
+    Long getDiskOfferingId();
+
     boolean isDynamicScalingEnabled();
+
+    Boolean getDiskOfferingStrictness();
+
+    void setDiskOfferingStrictness(boolean diskOfferingStrictness);
 }

api/src/main/java/com/cloud/server/ManagementService.java

@@ -394,14 +394,17 @@ public interface ManagementService {
 
     /**
      * List storage pools for live migrating of a volume. The API returns list of all pools in the cluster to which the
-     * volume can be migrated. Current pool is not included in the list.
+     * volume can be migrated. Current pool is not included in the list. In case of vSphere datastore cluster storage pools,
+     * this method removes the child storage pools and adds the corresponding parent datastore cluster for API response listing
      *
      * @param Long volumeId
      * @return Pair<List<? extends StoragePool>, List<? extends StoragePool>> List of storage pools in cluster and list
      *         of pools with enough capacity.
      */
     Pair<List<? extends StoragePool>, List<? extends StoragePool>> listStoragePoolsForMigrationOfVolume(Long volumeId);
 
+    Pair<List<? extends StoragePool>, List<? extends StoragePool>> listStoragePoolsForMigrationOfVolumeInternal(Long volumeId, Long newDiskOfferingId, Long newSize, Long newMinIops, Long newMaxIops, boolean keepSourceStoragePool);
+
     String[] listEventTypes();
 
     Pair<List<? extends HypervisorCapabilities>, Integer> listHypervisorCapabilities(Long id, HypervisorType hypervisorType, String keyword, Long startIndex,

api/src/main/java/com/cloud/storage/VolumeApiService.java

@@ -22,6 +22,7 @@
 import java.util.Map;
 
 import org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd;
+import org.apache.cloudstack.api.command.user.volume.ChangeOfferingForVolumeCmd;
 import org.apache.cloudstack.api.command.user.volume.CreateVolumeCmd;
 import org.apache.cloudstack.api.command.user.volume.DetachVolumeCmd;
 import org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd;
@@ -152,4 +153,6 @@ Snapshot takeSnapshot(Long volumeId, Long policyId, Long snapshotId, Account acc
     Volume destroyVolume(long volumeId, Account caller, boolean expunge, boolean forceExpunge);
 
     Volume recoverVolume(long volumeId);
+
+    Volume changeDiskOfferingForVolume(ChangeOfferingForVolumeCmd cmd) throws ResourceAllocationException;
 }

api/src/main/java/com/cloud/vm/DiskProfile.java

@@ -42,6 +42,8 @@ public class DiskProfile {
     private Long iopsReadRate;
     private Long iopsWriteRate;
     private String cacheMode;
+    private Long minIops;
+    private Long maxIops;
 
     private HypervisorType hyperType;
 
@@ -227,4 +229,22 @@ public void setCacheMode(String cacheMode) {
     public String getCacheMode() {
         return cacheMode;
     }
+
+
+    public Long getMinIops() {
+        return minIops;
+    }
+
+    public void setMinIops(Long minIops) {
+        this.minIops = minIops;
+    }
+
+    public Long getMaxIops() {
+        return maxIops;
+    }
+
+    public void setMaxIops(Long maxIops) {
+        this.maxIops = maxIops;
+    }
+
 }

api/src/main/java/com/cloud/vm/UserVmService.java

@@ -218,7 +218,7 @@ UserVm createBasicSecurityGroupVirtualMachine(DataCenter zone, ServiceOffering s
         String userData, String sshKeyPair, Map<Long, IpAddresses> requestedIps, IpAddresses defaultIp, Boolean displayVm, String keyboard,
         List<Long> affinityGroupIdList, Map<String, String> customParameter, String customId, Map<String, Map<Integer, String>> dhcpOptionMap,
         Map<Long, DiskOffering> dataDiskTemplateToDiskOfferingMap,
-        Map<String, String> userVmOVFProperties, boolean dynamicScalingEnabled) throws InsufficientCapacityException,
+        Map<String, String> userVmOVFProperties, boolean dynamicScalingEnabled, Long overrideDiskOfferingId) throws InsufficientCapacityException,
         ConcurrentOperationException, ResourceUnavailableException, StorageUnavailableException, ResourceAllocationException;
 
     /**
@@ -300,7 +300,7 @@ UserVm createAdvancedSecurityGroupVirtualMachine(DataCenter zone, ServiceOfferin
         HTTPMethod httpmethod, String userData, String sshKeyPair, Map<Long, IpAddresses> requestedIps, IpAddresses defaultIps, Boolean displayVm, String keyboard,
         List<Long> affinityGroupIdList, Map<String, String> customParameters, String customId, Map<String, Map<Integer, String>> dhcpOptionMap,
         Map<Long, DiskOffering> dataDiskTemplateToDiskOfferingMap,
-        Map<String, String> userVmOVFProperties, boolean dynamicScalingEnabled) throws InsufficientCapacityException,
+        Map<String, String> userVmOVFProperties, boolean dynamicScalingEnabled, Long overrideDiskOfferingId) throws InsufficientCapacityException,
         ConcurrentOperationException, ResourceUnavailableException, StorageUnavailableException, ResourceAllocationException;
 
     /**
@@ -379,7 +379,7 @@ UserVm createAdvancedVirtualMachine(DataCenter zone, ServiceOffering serviceOffe
         String hostName, String displayName, Long diskOfferingId, Long diskSize, String group, HypervisorType hypervisor, HTTPMethod httpmethod, String userData,
         String sshKeyPair, Map<Long, IpAddresses> requestedIps, IpAddresses defaultIps, Boolean displayVm, String keyboard, List<Long> affinityGroupIdList,
         Map<String, String> customParameters, String customId, Map<String, Map<Integer, String>> dhcpOptionMap, Map<Long, DiskOffering> dataDiskTemplateToDiskOfferingMap,
-        Map<String, String> templateOvfPropertiesMap, boolean dynamicScalingEnabled, String type)
+        Map<String, String> templateOvfPropertiesMap, boolean dynamicScalingEnabled, String type, Long overrideDiskOfferingId)
 
         throws InsufficientCapacityException, ConcurrentOperationException, ResourceUnavailableException, StorageUnavailableException, ResourceAllocationException;
 

api/src/main/java/com/cloud/vm/VirtualMachine.java

@@ -323,8 +323,6 @@ public boolean isUsedBySystem() {
 
     long getServiceOfferingId();
 
-    Long getDiskOfferingId();
-
     Long getBackupOfferingId();
 
     String getBackupExternalId();

api/src/main/java/org/apache/cloudstack/api/ApiConstants.java

@@ -112,12 +112,15 @@ public class ApiConstants {
     public static final String DISK = "disk";
     public static final String DISK_OFFERING_ID = "diskofferingid";
     public static final String NEW_DISK_OFFERING_ID = "newdiskofferingid";
+    public static final String OVERRIDE_DISK_OFFERING_ID = "overridediskofferingid";
     public static final String DISK_KBS_READ = "diskkbsread";
     public static final String DISK_KBS_WRITE = "diskkbswrite";
     public static final String DISK_IO_READ = "diskioread";
     public static final String DISK_IO_WRITE = "diskiowrite";
     public static final String DISK_IO_PSTOTAL = "diskiopstotal";
     public static final String DISK_SIZE = "disksize";
+    public static final String DISK_SIZE_STRICTNESS = "disksizestrictness";
+    public static final String DISK_OFFERING_STRICTNESS = "diskofferingstrictness";
     public static final String DOWNLOAD_DETAILS = "downloaddetails";
     public static final String UTILIZATION = "utilization";
     public static final String DRIVER = "driver";
@@ -589,6 +592,7 @@ public class ApiConstants {
     public static final String LIVE_MIGRATE = "livemigrate";
     public static final String MIGRATE_ALLOWED = "migrateallowed";
     public static final String MIGRATE_TO = "migrateto";
+    public static final String AUTO_MIGRATE = "automigrate";
     public static final String GUID = "guid";
     public static final String VSWITCH_TYPE_GUEST_TRAFFIC = "guestvswitchtype";
     public static final String VSWITCH_TYPE_PUBLIC_TRAFFIC = "publicvswitchtype";

api/src/main/java/org/apache/cloudstack/api/command/admin/offering/CreateDiskOfferingCmd.java

@@ -159,6 +159,9 @@ public class CreateDiskOfferingCmd extends BaseCmd {
     @Parameter(name = ApiConstants.STORAGE_POLICY, type = CommandType.UUID, entityType = VsphereStoragePoliciesResponse.class,required = false, description = "Name of the storage policy defined at vCenter, this is applicable only for VMware", since = "4.15")
     private Long storagePolicy;
 
+    @Parameter(name = ApiConstants.DISK_SIZE_STRICTNESS, type = CommandType.BOOLEAN, description = "To allow or disallow the resize operation on the disks created from this disk offering, if the flag is true then resize is not allowed", since = "4.17")
+    private Boolean diskSizeStrictness;
+
     @Parameter(name = ApiConstants.DETAILS, type = CommandType.MAP, description = "details to specify disk offering parameters", since = "4.16")
     private Map details;
 
@@ -301,6 +304,11 @@ public Map<String, String> getDetails() {
     public Long getStoragePolicy() {
         return storagePolicy;
     }
+
+    public boolean getDiskSizeStrictness() {
+        return diskSizeStrictness != null ? diskSizeStrictness : false;
+    }
+
     /////////////////////////////////////////////////////
     /////////////// API Implementation///////////////////
     /////////////////////////////////////////////////////