Adding new Dell EMC ECS Object Storage Plugin for CloudStack #12124

mhkadhum · 2025-11-24T14:01:47Z

Description

This PR adds support for a Dell EMC ECS S3 object storage plugin for Apache CloudStack. ECS is a software-defined object storage platform that supports both object and file-system protocols, with a focus on scalable and reliable object storage. We have been using ECS in production for three years, and this work extends CloudStack’s existing MinIO plugin to provide similar functionality for ECS.

The implementation supports the full lifecycle of S3-compatible buckets on ECS, including user provisioning, bucket creation, policy management, versioning, creation-time encryption, and integration with CloudStack’s S3 Browser. All functionality has been tested in a lab environment on Ubuntu using a CloudStack development setup based on the official installation guidelines.

Key architectural differences from MinIO:

Management API Integration
ECS requires use of the ECS Management API (port 4443, or 443 when fronted by HAProxy). CloudStack authenticates with management-user credentials to perform bucket and user operations.

Namespace Requirements
A dedicated ECS namespace is required for CloudStack-managed buckets. Multiple namespaces allow different CloudStack environments to share the same ECS cluster.

S3 Endpoints (Public and Private)
ECS exposes S3 services on ports 9020/9021 (non-TLS/TLS). In our deployment, these are routed through HAProxy and exposed externally on port 443. The Public URL is displayed to CloudStack users, while the Private URL is used internally.

TLS Handling
The “Allow Insecure HTTPS” option controls whether CloudStack accepts untrusted certificates when communicating with the ECS Management API.

User Provisioning Workflow
When a CloudStack user creates their first bucket, CloudStack provisions a corresponding ECS object user using the CloudStack UUID with a cs- prefix. ECS generates access and secret keys once, which CloudStack securely stores and reuses for subsequent bucket operations.

Bucket features:

Encryption: Supported only during bucket creation; CloudStack hides encryption in the update view.
Bucket Policy: Supports Private and Public configurations.
Versioning: Fully supported through the S3 API rather than the Management API.
Object Lock: Not supported in this release; the UI hides the option and API calls return a clear error.

Bucket modification supports quota changes, versioning updates, and policy changes. Encryption is excluded because ECS does not allow changing it after creation.

CloudStack’s S3 Browser supports upload, download, delete, listing, and prefix filtering through the ECS S3 endpoint. ECS prevents deletion of non-empty buckets, and CloudStack surfaces these errors accordingly.

We welcome review and feedback. The development fork is available here:
https://github.com/mhkadhum/cloudstack

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

The plugin was tested in an Ubuntu-based CloudStack development environment created by cloning the CloudStack source and following the official installation instructions. Testing included:

Management API authentication and namespace operations
Object-user provisioning and credential handling
Bucket creation, deletion, versioning, quota updates, and policy changes
S3 Browser operations (upload, download, delete, listing, prefix navigation)
Error handling for invalid configurations and unsupported Object Lock requests
TLS and insecure-TLS scenarios
HAProxy-fronted deployments for both Management API and S3 endpoints

…Provider

boring-cyborg · 2025-11-24T14:01:50Z

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
Be sure to read the CloudStack Coding Conventions.
Apache CloudStack is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
Slack: https://apachecloudstack.slack.com/

jbampton

You have failing tests for pre-commit and license checks.

Have a quick read up on pre-commit it really is a series of basic checks we run:

https://github.com/apache/cloudstack/blob/main/PRE-COMMIT.md

Also we normally add the ASF license header to all files or add an exclude

codecov · 2025-11-24T14:29:39Z

Codecov Report

❌ Patch coverage is 0% with 1052 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.53%. Comparing base (8171d95) to head (6e3fb3e).
⚠️ Report is 30 commits behind head on main.

Files with missing lines	Patch %	Lines
...age/datastore/driver/EcsObjectStoreDriverImpl.java	0.00%	907 Missing ⚠️
...tastore/lifecycle/EcsObjectStoreLifeCycleImpl.java	0.00%	132 Missing ⚠️
...datastore/provider/EcsObjectStoreProviderImpl.java	0.00%	13 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12124      +/-   ##
============================================
- Coverage     17.56%   17.53%   -0.04%     
- Complexity    15538    15548      +10     
============================================
  Files          5912     5916       +4     
  Lines        529383   530514    +1131     
  Branches      64660    64933     +273     
============================================
+ Hits          92984    93002      +18     
- Misses       425941   427052    +1111     
- Partials      10458    10460       +2

Flag	Coverage Δ
uitests	`3.58% <ø> (-0.01%)`	⬇️
unittests	`18.59% <0.00%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sureshanaparti · 2025-11-25T13:11:26Z

@mhkadhum can you check the build errors?

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

.../main/resources/META-INF/cloudstack/storage-object-ecs/spring-storage-object-ecs-context.xml

mhkadhum · 2025-11-26T20:20:10Z

Thanks for the feedback!

ive added the missing license headers and ran pre-commit locally, and pushed an updated commit

DaanHoogland · 2025-11-27T13:34:14Z

tools/marvin/setup.py



-VERSION = "4.23.0.0-SNAPSHOT"
+VERSION = "4.23.0.0"


this should not be in the PR

Hi @DaanHoogland
This was updated after running the pre-commit, should i keep it as
VERSION = "4.23.0.0-SNAPSHOT"
?

yes, it should only be updated by building (not by pre-commit) somebody introduced that behaviour for testing I think and it forces us to be more vigilent at checking in, so I don’t mind. We need to keep snapshot though untill releasing .

Ok,
I've just added it back.

DaanHoogland · 2025-11-28T10:46:30Z

looks generally good @mhkadhum , thanks for the contribution. Will you be maintaining this? As it is a 3rd party component and it may require hardware or licenses not available to the project.

mhkadhum · 2025-11-28T11:20:14Z

looks generally good @mhkadhum , thanks for the contribution. Will you be maintaining this? As it is a 3rd party component and it may require hardware or licenses not available to the project.

Yes, I will be the maintainer for the plugin. I have made a new commit to fix the previous build. Could you kindly review it

DaanHoogland

I have not found any oddities in the code, but one remark (in two parts ;) ) Feel free to ignore but I think it will help in the future.

...main/java/org/apache/cloudstack/storage/datastore/lifecycle/EcsObjectStoreLifeCycleImpl.java

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

…rom lifecycle , introduce caching to ECS token

mhkadhum · 2025-12-03T11:49:21Z

Hi,
@DaanHoogland @abh1sar
Just made the Update talked about in my previous comment with @DaanHoogland
The screen displays the token creation request, which responds with the token and expiration details in the response headers. These headers are then utilized in the code to store the token for the specified duration.

X-SDS-AUTH-MAX-AGE

mhkadhum · 2025-12-11T05:48:36Z

Hello @abh1sar @DaanHoogland
I just wanted to quickly follow up on my last comment.
I understand u r probably busy but if u get a chance, could u take a look when possible ? Appreciate the help!

DaanHoogland · 2025-12-11T08:12:02Z

Hello @abh1sar @DaanHoogland I just wanted to quickly follow up on my last comment. I understand u r probably busy but if u get a chance, could u take a look when possible ? Appreciate the help!

sorry @mhkadhum , this #12124 (comment) you mean right? I didn’t read it as a question and though this is ready for testing. What issues are you having right now?

mhkadhum · 2025-12-11T08:39:45Z

Hello @abh1sar @DaanHoogland I just wanted to quickly follow up on my last comment. I understand u r probably busy but if u get a chance, could u take a look when possible ? Appreciate the help!

sorry @mhkadhum , this #12124 (comment) you mean right? I didn’t read it as a question and though this is ready for testing. What issues are you having right now?

No worries @DaanHoogland
Yes this one. There are no issues at the moment. I am simply checking the status of the PR. Additionally, I am planning to develop another plugin for a different third-party object storage.
Do you have an ECS environment where you plan to conduct the testing?

abh1sar · 2025-12-11T08:47:02Z

Hi @mhkadhum, I am planning to finish my review by early next week.

DaanHoogland · 2025-12-11T10:57:35Z

Do you have an ECS environment where you plan to conduct the testing?

No, but we will regard this as 3rd party extension and trust you with specific feature testing, only doing regression testing. (sponsored donation in the sense that you make an instance available for testing is welcome, but you maintaining it as even more ;)

abh1sar

Reviewed EcsObjectStoreDriverImpl
Overall functionality looks good but there are some comments on code maintainability and organisation. Please check.
Overall, I think use of reflections can be avoided in most cases.

abh1sar · 2025-12-17T12:43:02Z

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

+        final String name  = bucket.getName();
+
+        if (objectLock) {
+            throw new CloudRuntimeException("Dell ECS doesn't support this feature: object locking");


I think InvalidParameterValueException will be more suitable here

ive changed this to InvalidParameterValueException

abh1sar · 2025-12-17T13:20:23Z

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

+    private static final long EXPIRY_SKEW_SEC = 30;           // refresh early
+    private static final ConcurrentHashMap<TokenKey, TokenEntry> TOKEN_CACHE = new ConcurrentHashMap<>();
+    private static final ConcurrentHashMap<TokenKey, Object> TOKEN_LOCKS = new ConcurrentHashMap<>();
+


Could I suggest creating separate classes for Token Management and XML parsing for better maintainability and reusability?
The EcsObjectStoreDriverImpl is currently doing a lot of different things.

I agree.
Token management and XML parsing have been extracted into separate classes
(EcsMgmtTokenManager and EcsXmlParser) to improve maintainability and reuse.

abh1sar · 2025-12-17T13:24:07Z

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

+    private static final String MGMT_URL   = "mgmt_url";      // e.g. https://ecs-api.example.com
+    private static final String SA_USER    = "sa_user";       // service account user
+    private static final String SA_PASS    = "sa_password";   // service account password
+    private static final String NAMESPACE  = "namespace";     // e.g. cloudstack
+    private static final String INSECURE   = "insecure";      // "true" to ignore TLS cert/host
+    private static final String S3_HOST    = "s3_host";       // S3 endpoint host (or URL if UI provides it)


Please consider moving all the constants to a new EcsConstants class.
EcsObjectStoreDriverImpl and EcsObjectStoreLifeCyclImpl both are using these.

abh1sar · 2025-12-17T13:29:39Z

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

+    private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if header missing
+    private static final long EXPIRY_SKEW_SEC = 30;           // refresh early


is this something that should be configurable during runtime (using global settings)?

ECS returns the token expiry via a response header at creation time. The default value is only used as a fallback in cases where the header is missing or unavailable. Since the expiry is determined by ECS and not intended to be tuned at runtime, adding this to global configuration does not seem necessary at this point.

abh1sar · 2025-12-18T02:11:38Z

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+    private static final Logger logger = LogManager.getLogger(EcsObjectStoreDriverImpl.class);


BaseObjectStoreDriverImpl already has a LogManager. No need to define it here.

abh1sar · 2025-12-18T03:51:06Z

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

+    private BucketVO resolveBucketVO(final BucketTO bucket, final long storeId) {
+        long id = getLongFromGetter(bucket, "getId", -1L);
+        if (id <= 0) id = getLongFromGetter(bucket, "getBucketId", -1L);
+        if (id > 0) {


The bucket's id is defined as NOT NULL AUTO_INCREMENT in the schema. So, it can never be <= 0.

The code below line 1274 is not required. I don't think it will ever happen that bucketVO is not found using the id , but it is found using uuid or name.

I have minimized this method

private BucketVO resolveBucketVO(final BucketTO bucket) { if (bucket == null) return null; long id = getLongFromGetter(bucket, "getId", -1L); if (id > 0) { return bucketDao.findById(id); } return null; }

abh1sar · 2025-12-18T03:51:58Z

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

+    }
+
+    private BucketVO resolveBucketVO(final BucketTO bucket, final long storeId) {
+        long id = getLongFromGetter(bucket, "getId", -1L);


We can just do long id = bucket.getId() as a bucket's id can never be null.

abh1sar · 2025-12-18T03:55:21Z

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

+                }
+            }
+
+            String accessKey = vo != null ? safeString(vo, "getAccessKey") : null;


I don't think we need reflections for getters anywhere in this file.
The interfaces are clearly defined.
this can be written as
String accessKey = vo.getAccessKey()

abh1sar · 2025-12-18T04:00:47Z

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

+    @SuppressWarnings("unchecked")
+    private BucketVO findBucketVOByStoreAndName(final long storeId, final String name) {
+        try {
+            java.lang.reflect.Method m = bucketDao.getClass().getMethod("listByStoreId", long.class);
+            Object res = m.invoke(bucketDao, storeId);
+            if (res instanceof List<?>) {
+                for (Object o : (List<?>) res) {
+                    if (o instanceof BucketVO) {
+                        BucketVO vo = (BucketVO) o;
+                        try { if (name.equals(vo.getName())) return vo; } catch (Throwable ignore) { }
+                    }
+                }
+            }
+        } catch (NoSuchMethodException ignored) {
+        } catch (Throwable t) {
+            logger.debug("ECS: listByStoreId fallback failed: {}", t.getMessage());
+        }
+        return null;
+    }
+
+    @SuppressWarnings("unchecked")
+    private BucketVO findBucketVOAnyByName(final String name) {
+        try {
+            java.lang.reflect.Method m = bucketDao.getClass().getMethod("listAll");
+            Object res = m.invoke(bucketDao);
+            if (res instanceof List<?>) {
+                for (Object o : (List<?>) res) {
+                    if (o instanceof BucketVO) {
+                        BucketVO vo = (BucketVO) o;
+                        try { if (name.equals(vo.getName())) return vo; } catch (Throwable ignore) { }
+                    }
+                }
+            }
+        } catch (NoSuchMethodException ignored) {
+        } catch (Throwable t) {
+            logger.debug("ECS: listAll scan failed: {}", t.getMessage());
+        }
+        return null;
+    }


If bucketDao.findById(id) has returned null that means the bucket doesn't exist in cloudstack db for some reason.
I don't think it will be found by any traversing any list as well.
So, if bucketDao.findById(id) returns null, we can just move on.

abh1sar · 2025-12-18T04:05:23Z

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java

+        final int maxTries = 45;
+        for (int attempt = 1; attempt <= maxTries; attempt++) {
+            BucketVO vo = resolveBucketVO(bucket, storeId);
+            if (vo == null) {


Should we go ahead with this operation if BucketVO is null i.e, it is not in the acs db, should we still manage it?

mhkadhum · 2025-12-18T17:56:52Z

Reviewed EcsObjectStoreDriverImpl Overall functionality looks good but there are some comments on code maintainability and organisation. Please check. Overall, I think use of reflections can be avoided in most cases.

Hello @abh1sar
Thank you for ur feedback, am working on addressing all the issues u provided, and updated the new code

root and others added 3 commits November 18, 2025 15:46

Adding New Dell ECS Plugin to the API & editing Frontend for the ECS …

8f36ebf

…Provider

Update AddObjectStorage.vue ECS Field Names and desc.

f131d41

Merge branch 'apache:main' into main

625fb4e

boring-cyborg bot added component:build component:storage component:UI labels Nov 24, 2025

jbampton reviewed Nov 24, 2025

View reviewed changes

sureshanaparti added the component:object-storage Object Storage label Nov 25, 2025

DaanHoogland reviewed Nov 25, 2025

View reviewed changes

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java Outdated Show resolved Hide resolved

DaanHoogland reviewed Nov 25, 2025

View reviewed changes

.../main/resources/META-INF/cloudstack/storage-object-ecs/spring-storage-object-ecs-context.xml Show resolved Hide resolved

DaanHoogland reviewed Nov 25, 2025

View reviewed changes

.../main/resources/META-INF/cloudstack/storage-object-ecs/spring-storage-object-ecs-context.xml Outdated Show resolved Hide resolved

add missing license

124649f

boring-cyborg bot added component:marvin Python Warning... Python code Ahead! labels Nov 26, 2025

mhkadhum requested review from DaanHoogland and jbampton November 27, 2025 07:01

DaanHoogland reviewed Nov 27, 2025

View reviewed changes

mhkadhum requested a review from DaanHoogland November 27, 2025 18:03

reverting update by pre-commit on tools/marvin/setup.py

ce40daf

DaanHoogland added has-3rdparty-components 3rd-party:ecs labels Nov 28, 2025

adding ECS Module in the Plugins pom.xml

7dd1bf2

DaanHoogland reviewed Nov 28, 2025

View reviewed changes

...main/java/org/apache/cloudstack/storage/datastore/lifecycle/EcsObjectStoreLifeCycleImpl.java Outdated Show resolved Hide resolved

...S/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java Show resolved Hide resolved

DaanHoogland requested a review from abh1sar December 3, 2025 06:58

Update ECS Driver, split to small helpers , remove SuppressWarnings f…

6e3fb3e

…rom lifecycle , introduce caching to ECS token

abh1sar reviewed Dec 18, 2025

View reviewed changes

rohityadavcloud requested review from sureshanaparti and weizhouapache December 18, 2025 05:35

		private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if header missing
		private static final long EXPIRY_SKEW_SEC = 30; // refresh early



		VERSION = "4.23.0.0-SNAPSHOT"
		VERSION = "4.23.0.0"

Adding new Dell EMC ECS Object Storage Plugin for CloudStack #12124

Are you sure you want to change the base?

Adding new Dell EMC ECS Object Storage Plugin for CloudStack #12124

Conversation

mhkadhum commented Nov 24, 2025

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

boring-cyborg bot commented Nov 24, 2025

Uh oh!

jbampton left a comment

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sureshanaparti commented Nov 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mhkadhum commented Nov 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Nov 28, 2025

Uh oh!

mhkadhum commented Nov 28, 2025

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mhkadhum commented Dec 3, 2025

Uh oh!

mhkadhum commented Dec 11, 2025

Uh oh!

DaanHoogland commented Dec 11, 2025

Uh oh!

mhkadhum commented Dec 11, 2025

Uh oh!

abh1sar commented Dec 11, 2025

Uh oh!

DaanHoogland commented Dec 11, 2025

Uh oh!

abh1sar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

codecov bot commented Nov 24, 2025 •

edited

Loading

mhkadhum Dec 25, 2025 •

edited

Loading

mhkadhum commented Dec 18, 2025 •

edited

Loading