Skip to content

Fix/prometheus metadata sorting #12112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Sinscerly wants to merge 2 commits into
base: main
Choose a base branch
from
+103 −26
Open

Fix/prometheus metadata sorting #12112

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
77092bc
fix prometheus metadata and sorted. Adds in TYPE and HELP
Sinscerly Nov 20, 2025
8851dd9
fix random linting issue
Sinscerly Nov 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 103 additions & 26 deletions ...ations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -512,20 +512,48 @@ public void updateMetrics() {
public String getMetrics() {
final StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("# Cloudstack Prometheus Metrics\n");
for (final Item item : metricsItems) {

List<Item> sortedItems = metricsItems.stream()
.sorted((item1, item2) -> item1.name.compareTo(item2.name))
.collect(Collectors.toList());

String currentMetricName = null;

for (Item item : sortedItems) {
if (!item.name.equals(currentMetricName)) {
currentMetricName = item.name;
stringBuilder.append("# HELP ").append(currentMetricName).append(" ")
.append(item.getHelp()).append("\n");
Copy link
Contributor

@nvazquez nvazquez Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Sinscerly - I have a question around the metrics using tags in which you have modified the help text - if not mistaken the text won't be updated as the modification will happen after invoking item.toMetricsString() so at this line the previous help text will be displayed, is this correct?

Copy link
Author

@Sinscerly Sinscerly Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @nvazquez,

The HELP and TYPE only have to be added before each group of metrics. Only once for cloudstack_domain_resource_count, regardless of how many items with different tags share that metric name.

Does this clarify your question?

Copy link
Contributor

@nvazquez nvazquez Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sinscerly sorry for the delay. My concern was only for the metrics that override the help field on the item.toMetricsString() call.

For example:
On the metrics iteration, when creating the metrics string for ItemHostCpu the string will always start with the default HELP text (Host CPU usage in MHz) despite having tags or not - I see in case the metric contains tags it should display Host CPU usage in MHz grouped by host tags. Please correct me if I'm wrong

Additionally, I think this if block if (!item.name.equals(currentMetricName)) can be removed

Copy link
Author

@Sinscerly Sinscerly Dec 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @nvazquez,

Well the item ItemHostCpu returns two different metrics: cloudstack_host_cpu_usage_mhz_total and cloudstack_host_cpu_usage_mhz_total_by_tag. I just made sure the correct help line is outputted for that metric.

In the end all metrics are sorted and each unique help line is added. so this will give:

# HELP cloudstack_host_cpu_usage_mhz_total <help description>
... metrics for above type
... ''
# HELP cloudstack_host_cpu_usage_mhz_total_by_tag <help description>
... metrics for above type

So it seemed logical for me to also change that. Although I'm doubting it will work, as the toMetricsString is called after adding in the HELP and TYPE I see. Although this would require some bigger rework of adding metrics I think, split it up in separate items.

Removing the if (!item.name.equals(currentMetricName)) will cause harm as you only want it once for every unique metric name. As you want:

# HELP cloudstack_domain_resource_count Resource usage count per domain
# TYPE cloudstack_domain_resource_count gauge
cloudstack_domain_resource_count{domain="/", type="memory"} 0
cloudstack_domain_resource_count{domain="/", type="cpu"} 0
cloudstack_domain_resource_count{domain="/", type="gpu"} 0
cloudstack_domain_resource_count{domain="/", type="primary_storage"} 0

And not the HELP and TYPE for every single metric item:

# HELP cloudstack_domain_resource_count Resource usage count per domain
# TYPE cloudstack_domain_resource_count gauge
cloudstack_domain_resource_count{domain="/", type="memory"} 0
# HELP cloudstack_domain_resource_count Resource usage count per domain
# TYPE cloudstack_domain_resource_count gauge
cloudstack_domain_resource_count{domain="/", type="cpu"} 0
# HELP cloudstack_domain_resource_count Resource usage count per domain
# TYPE cloudstack_domain_resource_count gauge
cloudstack_domain_resource_count{domain="/", type="gpu"} 0
# HELP cloudstack_domain_resource_count Resource usage count per domain
# TYPE cloudstack_domain_resource_count gauge
cloudstack_domain_resource_count{domain="/", type="primary_storage"} 0

Copy link
Contributor

@nvazquez nvazquez Dec 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Sinscerly please ignore my second comment then :)

About the first one, do you think that can be addressed on a separate PR, leaving this PR as it is now?

stringBuilder.append("# TYPE ").append(currentMetricName).append(" ")
.append(item.getType()).append("\n");
}

stringBuilder.append(item.toMetricsString()).append("\n");
}

return stringBuilder.toString();
}

private abstract class Item {
String name;
String help;
String type;

public Item(final String nm) {
public Item(final String nm, final String hlp, final String tp) {
name = nm;
help = hlp;
type = tp;
}

public abstract String toMetricsString();

public String getHelp() {
return help;
}

public String getType() {
return type;
}
}

class ItemVM extends Item {
Expand All @@ -535,7 +563,9 @@ class ItemVM extends Item {
long total;

public ItemVM(final String zn, final String zu, final String st, long cnt) {
super("cloudstack_vms_total");
super("cloudstack_vms_total",
"Total number of virtual machines",
"gauge");
zoneName = zn;
zoneUuid = zu;
filter = st;
Expand All @@ -556,7 +586,9 @@ class ItemVMByTag extends Item {
String hosttags;

public ItemVMByTag(final String zn, final String zu, final String st, long cnt, final String tags) {
super("cloudstack_vms_total_by_tag");
super("cloudstack_vms_total_by_tag",
"Total number of virtual machines grouped by host tags",
"gauge");
zoneName = zn;
zoneUuid = zu;
filter = st;
Expand All @@ -577,7 +609,9 @@ class ItemVolume extends Item {
int total;

public ItemVolume(final String zn, final String zu, final String st, int cnt) {
super("cloudstack_volumes_total");
super("cloudstack_volumes_total",
"Total number of volumes",
"gauge");
zoneName = zn;
zoneUuid = zu;
filter = st;
Expand All @@ -598,7 +632,9 @@ class ItemHost extends Item {
String hosttags;

public ItemHost(final String zn, final String zu, final String st, int cnt, final String tags) {
super("cloudstack_hosts_total");
super("cloudstack_hosts_total",
"Total number of hosts",
"gauge");
zoneName = zn;
zoneUuid = zu;
state = st;
Expand All @@ -610,6 +646,7 @@ public ItemHost(final String zn, final String zu, final String st, int cnt, fina
public String toMetricsString() {
if (StringUtils.isNotEmpty(hosttags)) {
name = "cloudstack_hosts_total_by_tag";
help = "Total number of hosts grouped by tags";
return String.format("%s{zone=\"%s\",filter=\"%s\",tags=\"%s\"} %d", name, zoneName, state, hosttags, total);
}
return String.format("%s{zone=\"%s\",filter=\"%s\"} %d", name, zoneName, state, total);
Expand All @@ -628,7 +665,9 @@ class ItemVMCore extends Item {
String hosttags;

public ItemVMCore(final String zn, final String zu, final String hn, final String hu, final String hip, final String fl, final Long cr, final int dedicated, final String tags) {
super("cloudstack_host_vms_cores_total");
super("cloudstack_host_vms_cores_total",
"Total number of VM cores on hosts",
"gauge");
zoneName = zn;
zoneUuid = zu;
hostName = hn;
Expand All @@ -649,6 +688,7 @@ public String toMetricsString() {
return String.format("%s{zone=\"%s\",filter=\"%s\"} %d", name, zoneName, filter, core);
} else {
name = "cloudstack_host_vms_cores_total_by_tag";
help = "Total number of VM cores grouped by host tags";
return String.format("%s{zone=\"%s\",filter=\"%s\",tags=\"%s\"} %d", name, zoneName, filter, hosttags, core);
}
}
Expand All @@ -657,13 +697,14 @@ public String toMetricsString() {
}

class MissingHostInfo extends Item {

String zoneName;
String hostName;
MissingInfoFilter filter;

public MissingHostInfo(String zoneName, String hostname, MissingInfoFilter filter) {
super("cloudstack_host_missing_info");
super("cloudstack_host_missing_info",
"Hosts with missing capacity or statistics information",
"gauge");
this.zoneName = zoneName;
this.hostName = hostname;
this.filter = filter;
Expand All @@ -688,7 +729,9 @@ class ItemHostCpu extends Item {
String hosttags;

public ItemHostCpu(final String zn, final String zu, final String hn, final String hu, final String hip, final String of, final String fl, final double mh, final int dedicated, final String tags) {
super("cloudstack_host_cpu_usage_mhz_total");
super("cloudstack_host_cpu_usage_mhz_total",
"Host CPU usage in MHz",
"gauge");
zoneName = zn;
zoneUuid = zu;
hostName = hn;
Expand All @@ -708,6 +751,7 @@ public String toMetricsString() {
return String.format("%s{zone=\"%s\",filter=\"%s\"} %.2f", name, zoneName, filter, mhertz);
} else {
name = "cloudstack_host_cpu_usage_mhz_total_by_tag";
help = "Host CPU usage in MHz grouped by host tags";
return String.format("%s{zone=\"%s\",filter=\"%s\",tags=\"%s\"} %.2f", name, zoneName, filter, hosttags, mhertz);
}
}
Expand All @@ -728,7 +772,9 @@ class ItemHostMemory extends Item {
String hosttags;

public ItemHostMemory(final String zn, final String zu, final String hn, final String hu, final String hip, final String of, final String fl, final double membytes, final int dedicated, final String tags) {
super("cloudstack_host_memory_usage_mibs_total");
super("cloudstack_host_memory_usage_mibs_total",
"Host memory usage in MiB",
"gauge");
zoneName = zn;
zoneUuid = zu;
hostName = hn;
Expand All @@ -748,6 +794,7 @@ public String toMetricsString() {
return String.format("%s{zone=\"%s\",filter=\"%s\"} %.2f", name, zoneName, filter, miBytes);
} else {
name = "cloudstack_host_memory_usage_mibs_total_by_tag";
help = "Host memory usage in MiB grouped by host tags";
return String.format("%s{zone=\"%s\",filter=\"%s\",tags=\"%s\"} %.2f", name, zoneName, filter, hosttags, miBytes);
}
}
Expand All @@ -764,7 +811,9 @@ class ItemHostVM extends Item {
int total;

public ItemHostVM(final String zoneName, final String zoneUuid, final String hostName, final String hostUuid, final String hostIp, final int total) {
super("cloudstack_host_vms_total");
super("cloudstack_host_vms_total",
"Total number of VMs per host",
"gauge");
this.zoneName = zoneName;
this.zoneUuid = zoneUuid;
this.hostName = hostName;
Expand All @@ -790,7 +839,9 @@ class ItemPool extends Item {
double total;

public ItemPool(final String zn, final String zu, final String pn, final String pa, final String typ, final String of, final String fl, double cnt) {
super("cloudstack_storage_pool_gibs_total");
super("cloudstack_storage_pool_gibs_total",
"Storage pool capacity in GiB",
"gauge");
zoneName = zn;
zoneUuid = zu;
pname = pn;
Expand All @@ -817,7 +868,9 @@ class ItemPrivateIp extends Item {
int total;

public ItemPrivateIp(final String zn, final String zu, final String fl, int cnt) {
super("cloudstack_private_ips_total");
super("cloudstack_private_ips_total",
"Total number of private IP addresses",
"gauge");
zoneName = zn;
zoneUuid = zu;
filter = fl;
Expand All @@ -837,7 +890,9 @@ class ItemPublicIp extends Item {
int total;

public ItemPublicIp(final String zn, final String zu, final String fl, int cnt) {
super("cloudstack_public_ips_total");
super("cloudstack_public_ips_total",
"Total number of public IP addresses",
"gauge");
zoneName = zn;
zoneUuid = zu;
filter = fl;
Expand All @@ -857,7 +912,9 @@ class ItemSharedNetworkIp extends Item {
int total;

public ItemSharedNetworkIp(final String zn, final String zu, final String fl, int cnt) {
super("cloudstack_shared_network_ips_total");
super("cloudstack_shared_network_ips_total",
"Total number of shared network IP addresses",
"gauge");
zoneName = zn;
zoneUuid = zu;
filter = fl;
Expand All @@ -877,7 +934,9 @@ class ItemVlan extends Item {
int total;

public ItemVlan(final String zn, final String zu, final String fl, int cnt) {
super("cloudstack_vlans_total");
super("cloudstack_vlans_total",
"Total number of VLANs",
"gauge");
zoneName = zn;
zoneUuid = zu;
filter = fl;
Expand All @@ -894,7 +953,9 @@ class ItemDomainLimitCpu extends Item {
long cores;

public ItemDomainLimitCpu(final long c) {
super("cloudstack_domain_limit_cpu_cores_total");
super("cloudstack_domain_limit_cpu_cores_total",
"Total CPU core limit across all domains",
"gauge");
cores = c;
}

Expand All @@ -908,7 +969,9 @@ class ItemDomainLimitMemory extends Item {
long miBytes;

public ItemDomainLimitMemory(final long mb) {
super("cloudstack_domain_limit_memory_mibs_total");
super("cloudstack_domain_limit_memory_mibs_total",
"Total memory limit in MiB across all domains",
"gauge");
miBytes = mb;
}

Expand All @@ -927,7 +990,9 @@ class ItemHostIsDedicated extends Item {
int isDedicated;

public ItemHostIsDedicated(final String zoneName, final String zoneUuid, final String hostName, final String hostUuid, final String hostIp, final int isDedicated) {
super("cloudstack_host_is_dedicated");
super("cloudstack_host_is_dedicated",
"Whether a host is dedicated (1) or not (0)",
"gauge");
this.zoneName = zoneName;
this.zoneUuid = zoneUuid;
this.hostName = hostName;
Expand All @@ -949,7 +1014,9 @@ class ItemActiveDomains extends Item {
int total;

public ItemActiveDomains(final String zn, final String zu, final int cnt) {
super("cloudstack_active_domains_total");
super("cloudstack_active_domains_total",
"Total number of active domains",
"gauge");
zoneName = zn;
zoneUuid = zu;
total = cnt;
Expand All @@ -970,7 +1037,9 @@ class ItemHostDedicatedToAccount extends Item {

public ItemHostDedicatedToAccount(final String zoneName, final String hostName,
final String accountName, final String domainName, int isDedicated) {
super("cloudstack_host_dedicated_to_account");
super("cloudstack_host_dedicated_to_account",
"Host dedication to specific account",
"gauge");
this.zoneName = zoneName;
this.hostName = hostName;
this.accountName = accountName;
Expand All @@ -991,7 +1060,9 @@ class ItemPerDomainResourceLimit extends Item {
String resourceType;

public ItemPerDomainResourceLimit(final long c, final String domainName, final String resourceType) {
super("cloudstack_domain_resource_limit");
super("cloudstack_domain_resource_limit",
"Resource limits per domain",
"gauge");
this.cores = c;
this.domainName = domainName;
this.resourceType = resourceType;
Expand All @@ -1009,7 +1080,9 @@ class ItemPerDomainResourceCount extends Item {
String resourceType;

public ItemPerDomainResourceCount(final long mb, final String domainName, final String resourceType) {
super("cloudstack_domain_resource_count");
super("cloudstack_domain_resource_count",
"Resource usage count per domain",
"gauge");
this.miBytes = mb;
this.domainName = domainName;
this.resourceType = resourceType;
Expand All @@ -1027,7 +1100,9 @@ class ItemActiveAccounts extends Item {
int total;

public ItemActiveAccounts(final String zn, final String zu, final int cnt) {
super("cloudstack_active_accounts_total");
super("cloudstack_active_accounts_total",
"Total number of active accounts",
"gauge");
zoneName = zn;
zoneUuid = zu;
total = cnt;
Expand All @@ -1047,7 +1122,9 @@ class ItemVMsBySize extends Item {
int total;

public ItemVMsBySize(final String zn, final String zu, final int c, final int m, int cnt) {
super("cloudstack_vms_total_by_size");
super("cloudstack_vms_total_by_size",
"Total number of VMs grouped by CPU and memory size",
"gauge");
zoneName = zn;
zoneUuid = zu;
cpu = c;
Expand Down
Loading