Skip to content

chore: ensure metrics are correctly exposed #701

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maltesander merged 12 commits into from
Oct 30, 2025
Merged

chore: ensure metrics are correctly exposed #701

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
5352011
chore: ensure metrics are correctly exposed
xeniape Oct 15, 2025
8495dda
add changelog entry
xeniape Oct 15, 2025
3cce090
Update rust/operator-binary/src/crd/mod.rs
maltesander Oct 27, 2025
e992b5b
Update rust/operator-binary/src/hbase_controller.rs
maltesander Oct 27, 2025
969e1ae
Apply suggestions from code review
maltesander Oct 27, 2025
b6c96b8
Merge branch 'main' into chore/ensure-metrics-correctly-exposed
maltesander Oct 27, 2025
aabbecf
add metrics service to tls cert
maltesander Oct 27, 2025
8dfd7d6
clean up port retrieval methods
maltesander Oct 27, 2025
04ae9f1
pre commit
maltesander Oct 27, 2025
5d2b158
Merge remote-tracking branch 'origin/main' into chore/ensure-metrics-…
maltesander Oct 27, 2025
b562ea9
Apply suggestions from code review
maltesander Oct 29, 2025
3385544
use boolean for https instead of hbase cluster
maltesander Oct 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,19 @@
- `EOS_CHECK_MODE` (`--eos-check-mode`) to set the EoS check mode. Currently, only "offline" is supported.
- `EOS_INTERVAL` (`--eos-interval`) to set the interval in which the operator checks if it is EoS.
- `EOS_DISABLED` (`--eos-disabled`) to disable the EoS checker completely.
- Add `metrics` Services ([#701]).

### Changed

- Bump stackable-operator to `0.100.1` ([#705]).
- Changed env-vars to be consistent with config-utils in the entrypoint script ([#700]).
- BREAKING: The `prometheus.io/scrape` label moved from the `headless` Service to the `metrics` Service, which
uses `metrics` as the port name instead of the previous `ui-http`/`ui-https` port name ([#701]).

[#691]: https://github.com/stackabletech/hbase-operator/pull/691
[#697]: https://github.com/stackabletech/hbase-operator/pull/697
[#700]: https://github.com/stackabletech/hbase-operator/pull/700
[#701]: https://github.com/stackabletech/hbase-operator/pull/701
[#705]: https://github.com/stackabletech/hbase-operator/pull/705

## [25.7.0] - 2025-07-23
Expand Down
19 changes: 17 additions & 2 deletions docs/modules/hbase/pages/usage-guide/monitoring.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,20 @@ See xref:operators:monitoring.adoc[] for more details.

Starting with HBase 2.6 the URL for Prometheus metrics has changed.
This is because HBase offers now a built-in endpoint for this purpose.
This endpoint is available from the UI service.
For example, in the case of the master service, the URL is `http://<master-service>:16010/prometheus`.
This endpoint is available from the `metrics` Services.
For example, in the case of the master Service, the URL is `http://<hbasecluster-name>-master-<rolegroup-name>-metrics:16010/prometheus`.

== Authentication when using TLS

HBase exposes metrics through the same port as their web UI. Hence, when configuring HBase with TLS the metrics are also secured by TLS,
and the clients scraping the metrics endpoint need to authenticate against it. This could for example be accomplished by utilizing mTLS
between Kubernetes Pods with the xref:home:secret-operator:index.adoc[Secret Operator].

When using Prometheus `ServiceMonitor` for scraping, the `address` label needs relabeling to use the `headless` Service instead of the
`metrics` Service. This is because by default Prometheus targets the Pod IPs as endpoints, but since the Pod IPs are not
part of the certificate, the authentication will fail. Instead, the FQDN of the Pods, which can be added to the certificate, is used, but
this FQDN is only available through the `headless` Service.

A more detailed explanation can be found in the xref:home:nifi:usage_guide/monitoring.adoc[NiFi Operator Monitoring Docs] with a similar situation
and an example of a Prometheus `ServiceMonitor` configured for TLS in the
https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus-service-monitors.yaml[Monitoring Stack{external-link-icon}^].
122 changes: 72 additions & 50 deletions rust/operator-binary/src/crd/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,20 +65,24 @@ pub const SSL_CLIENT_XML: &str = "ssl-client.xml";
pub const HBASE_CLUSTER_DISTRIBUTED: &str = "hbase.cluster.distributed";
pub const HBASE_ROOTDIR: &str = "hbase.rootdir";

pub const HBASE_UI_PORT_NAME_HTTP: &str = "ui-http";
pub const HBASE_UI_PORT_NAME_HTTPS: &str = "ui-https";
pub const HBASE_REST_PORT_NAME_HTTP: &str = "rest-http";
pub const HBASE_REST_PORT_NAME_HTTPS: &str = "rest-https";
const HBASE_UI_PORT_NAME_HTTP: &str = "ui-http";
const HBASE_UI_PORT_NAME_HTTPS: &str = "ui-https";
const HBASE_REST_PORT_NAME_HTTP: &str = "rest-http";
const HBASE_REST_PORT_NAME_HTTPS: &str = "rest-https";
const HBASE_METRICS_PORT_NAME: &str = "metrics";

pub const HBASE_MASTER_PORT: u16 = 16000;
// HBase always uses 16010, regardless of http or https. On 2024-01-17 we decided in Arch-meeting that we want to stick
// the port numbers to what the product is doing, so we get the least surprise for users - even when this means we have
// inconsistency between Stackable products.
pub const HBASE_MASTER_UI_PORT: u16 = 16010;
pub const HBASE_MASTER_METRICS_PORT: u16 = 16010;
pub const HBASE_REGIONSERVER_PORT: u16 = 16020;
pub const HBASE_REGIONSERVER_UI_PORT: u16 = 16030;
pub const HBASE_REGIONSERVER_METRICS_PORT: u16 = 16030;
pub const HBASE_REST_PORT: u16 = 8080;
pub const HBASE_REST_UI_PORT: u16 = 8085;
pub const HBASE_REST_METRICS_PORT: u16 = 8085;
pub const LISTENER_VOLUME_NAME: &str = "listener";
pub const LISTENER_VOLUME_DIR: &str = "/stackable/listener";

Expand Down Expand Up @@ -513,52 +517,6 @@ impl v1alpha1::HbaseCluster {
.as_ref()
.map(|a| a.tls_secret_class.clone())
}

/// Returns required port name and port number tuples depending on the role.
/// Hbase versions 2.6.* will have two ports for each role. The metrics are available over the
/// UI port.
pub fn ports(&self, role: &HbaseRole) -> Vec<(String, u16)> {
match role {
HbaseRole::Master => vec![
("master".to_string(), HBASE_MASTER_PORT),
(self.ui_port_name(), HBASE_MASTER_UI_PORT),
],
HbaseRole::RegionServer => vec![
("regionserver".to_string(), HBASE_REGIONSERVER_PORT),
(self.ui_port_name(), HBASE_REGIONSERVER_UI_PORT),
],
HbaseRole::RestServer => vec![
(
if self.has_https_enabled() {
HBASE_REST_PORT_NAME_HTTPS
} else {
HBASE_REST_PORT_NAME_HTTP
}
.to_string(),
HBASE_REST_PORT,
),
(self.ui_port_name(), HBASE_REST_UI_PORT),
],
}
}

pub fn service_port(&self, role: &HbaseRole) -> u16 {
match role {
HbaseRole::Master => HBASE_MASTER_PORT,
HbaseRole::RegionServer => HBASE_REGIONSERVER_PORT,
HbaseRole::RestServer => HBASE_REST_PORT,
}
}

/// Name of the port used by the Web UI, which depends on HTTPS usage
pub fn ui_port_name(&self) -> String {
if self.has_https_enabled() {
HBASE_UI_PORT_NAME_HTTPS
} else {
HBASE_UI_PORT_NAME_HTTP
}
.to_string()
}
}

pub fn merged_env(rolegroup_config: Option<&BTreeMap<String, String>>) -> Vec<EnvVar> {
Expand Down Expand Up @@ -759,6 +717,70 @@ impl HbaseRole {
};
Ok(pvc)
}

/// Returns required port name and port number tuples depending on the role.
///
/// Hbase versions 2.6.* will have two ports for each role. The metrics are available on the
/// UI port.
pub fn ports(&self, hbase: &v1alpha1::HbaseCluster) -> Vec<(String, u16)> {
vec![
(self.data_port_name(hbase), self.data_port()),
(
Self::ui_port_name(hbase.has_https_enabled()).to_string(),
self.ui_port(),
),
]
}

pub fn data_port(&self) -> u16 {
match self {
HbaseRole::Master => HBASE_MASTER_PORT,
HbaseRole::RegionServer => HBASE_REGIONSERVER_PORT,
HbaseRole::RestServer => HBASE_REST_PORT,
}
}

pub fn data_port_name(&self, hbase: &v1alpha1::HbaseCluster) -> String {
match self {
HbaseRole::Master | HbaseRole::RegionServer => self.to_string(),
HbaseRole::RestServer => {
if hbase.has_https_enabled() {
HBASE_REST_PORT_NAME_HTTPS.to_owned()
} else {
HBASE_REST_PORT_NAME_HTTP.to_owned()
}
}
}
}

pub fn ui_port(&self) -> u16 {
match self {
HbaseRole::Master => HBASE_MASTER_UI_PORT,
HbaseRole::RegionServer => HBASE_REGIONSERVER_UI_PORT,
HbaseRole::RestServer => HBASE_REST_UI_PORT,
}
}

/// Name of the port used by the Web UI, which depends on HTTPS usage
pub fn ui_port_name(has_https_enabled: bool) -> &'static str {
if has_https_enabled {
HBASE_UI_PORT_NAME_HTTPS
} else {
HBASE_UI_PORT_NAME_HTTP
}
}

pub fn metrics_port(&self) -> u16 {
match self {
HbaseRole::Master => HBASE_MASTER_METRICS_PORT,
HbaseRole::RegionServer => HBASE_REGIONSERVER_METRICS_PORT,
HbaseRole::RestServer => HBASE_REST_METRICS_PORT,
}
}

pub fn metrics_port_name() -> &'static str {
HBASE_METRICS_PORT_NAME
}
}

fn default_resources(role: &HbaseRole) -> ResourcesFragment<HbaseStorageConfig, NoRuntimeLimits> {
Expand Down
Loading