Skip to content

Commit c92ae07

Browse files
cristiangrecomatthewnolfwildumclayton-cornell
authored
Add database_observability.mysql component (#2053)
* [draft] Add dbo11y component WIP * basic impl of Run and Update * export targets from component * scaffolding for collectors * default args * refactor ConnectionInfo * docs * rename * use instanceKey * db engine in component name * import logic for schema_table collector * Add logic for QuerySample collector * Add test for QuerySample collector * add test for SchemaTable collector * add test for ConnectionInfo collector * Fix logging msgs Co-authored-by: MattNolf <86960799+matthewnolf@users.noreply.github.com> * add index.md * review feedback: add goleak verification to tests * add example to docs * Call Stop when exiting Run loop * Add more logging before exiting loop * rename Run into Start * add todo about possible refactoring * fix tests after rename * Update internal/component/database_observability/mysql/collector/connection_info.go Co-authored-by: William Dumont <william.dumont@grafana.com> * fixes * rename component * rename to collect_interval * no need to save tmp new args * Update docs/sources/reference/components/database_observability/database_observability.mysql.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * extend docs structure * Update docs/sources/reference/components/database_observability/database_observability.mysql.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * fix table * review docs again * review shutdown --------- Co-authored-by: MattNolf <86960799+matthewnolf@users.noreply.github.com> Co-authored-by: William Dumont <william.dumont@grafana.com> Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>
1 parent e19c783 commit c92ae07

File tree

13 files changed

+1081
-1
lines changed

13 files changed

+1081
-1
lines changed

‎docs/sources/reference/compatibility/_index.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,10 @@ The following components, grouped by namespace, _export_ Targets.
4343
- [beyla.ebpf](../components/beyla/beyla.ebpf)
4444
{{< /collapse >}}
4545

46+
{{< collapse title="database_observability" >}}
47+
- [database_observability.mysql](../components/database_observability/database_observability.mysql)
48+
{{< /collapse >}}
49+
4650
{{< collapse title="discovery" >}}
4751
- [discovery.azure](../components/discovery/discovery.azure)
4852
- [discovery.consul](../components/discovery/discovery.consul)
@@ -236,6 +240,10 @@ The following components, grouped by namespace, _consume_ Loki `LogsReceiver`.
236240

237241
<!-- START GENERATED SECTION: CONSUMERS OF Loki `LogsReceiver` -->
238242

243+
{{< collapse title="database_observability" >}}
244+
- [database_observability.mysql](../components/database_observability/database_observability.mysql)
245+
{{< /collapse >}}
246+
239247
{{< collapse title="faro" >}}
240248
- [faro.receiver](../components/faro/faro.receiver)
241249
{{< /collapse >}}

‎docs/sources/reference/components/database_observability/_index.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
canonical: https://grafana.com/docs/alloy/latest/reference/components/database_observability/
3+
description: Learn about the database_observability components in Grafana Alloy
4+
title: database_observability
5+
weight: 100
6+
---
7+
8+
# database_observability
9+
10+
This section contains reference documentation for the `database_observability` components.
11+
12+
{{< section >}}

‎docs/sources/reference/components/database_observability/database_observability.mysql.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
---
2+
canonical: https://grafana.com/docs/alloy/latest/reference/components/database_observability.mysql/
3+
description: Learn about database_observability.mysql
4+
title: database_observability.mysql
5+
---
6+
7+
<span class="badge docs-labels__stage docs-labels__item">Experimental</span>
8+
9+
# database_observability.mysql
10+
11+
{{< docs/shared lookup="stability/experimental.md" source="alloy" version="<ALLOY_VERSION>" >}}
12+
13+
## Usage
14+
15+
```alloy
16+
database_observability.mysql "LABEL" {
17+
data_source_name = DATA_SOURCE_NAME
18+
forward_to = [LOKI_RECEIVERS]
19+
}
20+
```
21+
22+
## Arguments
23+
24+
The following arguments are supported:
25+
26+
| Name | Type | Description | Default | Required |
27+
| -------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------- | ------- | -------- |
28+
| `data_source_name` | `secret` | [Data Source Name](https://github.com/go-sql-driver/mysql#dsn-data-source-name) for the MySQL server to connect to. | | yes |
29+
| `forward_to` | `list(LogsReceiver)` | Where to forward log entries after processing. | | yes |
30+
| `collect_interval` | `duration` | How frequently to collect query samples from database | `"10s"` | no |
31+
32+
## Blocks
33+
34+
The `database_observability.mysql` component does not support any blocks, and is configured fully through arguments.
35+
36+
## Example
37+
38+
```alloy
39+
database_observability.mysql "orders_db" {
40+
data_source_name = "user:pass@mysql:3306/"
41+
forward_to = [loki.write.logs_service.receiver]
42+
}
43+
44+
prometheus.scrape "orders_db" {
45+
targets = database_observability.mysql.orders_db.targets
46+
honor_labels = true // required to keep job and instance labels
47+
forward_to = [prometheus.remote_write.metrics_service.receiver]
48+
}
49+
50+
prometheus.remote_write "metrics_service" {
51+
endpoint {
52+
url = sys.env("GCLOUD_HOSTED_METRICS_URL")
53+
basic_auth {
54+
username = sys.env("GCLOUD_HOSTED_METRICS_ID")
55+
password = sys.env("GCLOUD_RW_API_KEY")
56+
}
57+
}
58+
}
59+
60+
loki.write "logs_service" {
61+
endpoint {
62+
url = sys.env("GCLOUD_HOSTED_LOGS_URL")
63+
basic_auth {
64+
username = sys.env("GCLOUD_HOSTED_LOGS_ID")
65+
password = sys.env("GCLOUD_RW_API_KEY")
66+
}
67+
}
68+
}
69+
```
70+
71+
<!-- START GENERATED COMPATIBLE COMPONENTS -->
72+
73+
## Compatible components
74+
75+
`database_observability.mysql` can accept arguments from the following components:
76+
77+
- Components that export [Loki `LogsReceiver`](../../../compatibility/#loki-logsreceiver-exporters)
78+
79+
`database_observability.mysql` has exports that can be consumed by the following components:
80+
81+
- Components that consume [Targets](../../../compatibility/#targets-consumers)
82+
83+
{{< admonition type="note" >}}
84+
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly.
85+
Refer to the linked documentation for more details.
86+
{{< /admonition >}}
87+
88+
<!-- END GENERATED COMPATIBLE COMPONENTS -->

‎go.mod‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ require (
88
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.13.0
99
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0
1010
github.com/Azure/go-autorest/autorest v0.11.29
11+
github.com/DATA-DOG/go-sqlmock v1.5.2
1112
github.com/IBM/sarama v1.43.3
1213
github.com/KimMachineGun/automemlimit v0.6.0
1314
github.com/Lusitaniae/apache_exporter v0.11.1-0.20220518131644-f9522724dab4
@@ -768,7 +769,7 @@ require (
768769
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
769770
github.com/xhit/go-str2duration/v2 v2.1.0 // indirect
770771
github.com/xo/dburl v0.20.0 // indirect
771-
github.com/xwb1989/sqlparser v0.0.0-20180606152119-120387863bf2 // indirect
772+
github.com/xwb1989/sqlparser v0.0.0-20180606152119-120387863bf2
772773
github.com/yl2chen/cidranger v1.0.2 // indirect
773774
github.com/youmark/pkcs8 v0.0.0-20240424034433-3c2c7870ae76 // indirect
774775
github.com/yusufpapurcu/wmi v1.2.4 // indirect

‎go.sum‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1645,6 +1645,7 @@ github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvW
16451645
github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
16461646
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
16471647
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
1648+
github.com/kisielk/sqlstruct v0.0.0-20201105191214-5f3e10d3ab46/go.mod h1:yyMNCyc/Ib3bDTKd379tNMpB/7/H5TjM2Y9QJ5THLbE=
16481649
github.com/klauspost/asmfmt v1.3.2 h1:4Ri7ox3EwapiOjCki+hw14RyKk201CN4rzyCJRFLpK4=
16491650
github.com/klauspost/asmfmt v1.3.2/go.mod h1:AG8TuvYojzulgDAMCnYn50l/5QV3Bs/tp6j0HLHbNSE=
16501651
github.com/klauspost/compress v1.11.0/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=

‎internal/component/all/all.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package all
33

44
import (
55
_ "github.com/grafana/alloy/internal/component/beyla/ebpf" // Import beyla.ebpf
6+
_ "github.com/grafana/alloy/internal/component/database_observability/mysql" // Import database_observability.mysql
67
_ "github.com/grafana/alloy/internal/component/discovery/aws" // Import discovery.aws.ec2 and discovery.aws.lightsail
78
_ "github.com/grafana/alloy/internal/component/discovery/azure" // Import discovery.azure
89
_ "github.com/grafana/alloy/internal/component/discovery/consul" // Import discovery.consul

‎internal/component/database_observability/mysql/collector/connection_info.go‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
package collector
2+
3+
import (
4+
"context"
5+
"net"
6+
"regexp"
7+
"strings"
8+
9+
"github.com/go-sql-driver/mysql"
10+
"github.com/prometheus/client_golang/prometheus"
11+
)
12+
13+
var rdsRegex = regexp.MustCompile(`(?P<identifier>[^\.]+)\.([^\.]+)\.(?P<region>[^\.]+)\.rds\.amazonaws\.com`)
14+
15+
type ConnectionInfoArguments struct {
16+
DSN string
17+
Registry *prometheus.Registry
18+
}
19+
20+
type ConnectionInfo struct {
21+
DSN string
22+
Registry *prometheus.Registry
23+
InfoMetric *prometheus.GaugeVec
24+
}
25+
26+
func NewConnectionInfo(args ConnectionInfoArguments) (*ConnectionInfo, error) {
27+
infoMetric := prometheus.NewGaugeVec(prometheus.GaugeOpts{
28+
Name: "connection_info",
29+
Help: "Information about the connection",
30+
}, []string{"provider_name", "region", "db_instance_identifier"})
31+
32+
args.Registry.MustRegister(infoMetric)
33+
34+
return &ConnectionInfo{
35+
DSN: args.DSN,
36+
Registry: args.Registry,
37+
InfoMetric: infoMetric,
38+
}, nil
39+
}
40+
41+
func (c *ConnectionInfo) Start(ctx context.Context) error {
42+
cfg, err := mysql.ParseDSN(c.DSN)
43+
if err != nil {
44+
return err
45+
}
46+
47+
var (
48+
providerName = "unknown"
49+
providerRegion = "unknown"
50+
dbInstanceIdentifier = "unknown"
51+
)
52+
53+
host, _, err := net.SplitHostPort(cfg.Addr)
54+
if err == nil && host != "" {
55+
if strings.HasSuffix(host, "rds.amazonaws.com") {
56+
providerName = "aws"
57+
matches := rdsRegex.FindStringSubmatch(host)
58+
if len(matches) > 3 {
59+
dbInstanceIdentifier = matches[1]
60+
providerRegion = matches[3]
61+
}
62+
}
63+
}
64+
65+
c.InfoMetric.WithLabelValues(providerName, providerRegion, dbInstanceIdentifier).Set(1)
66+
return nil
67+
}
68+
69+
func (c *ConnectionInfo) Stop() {
70+
c.Registry.Unregister(c.InfoMetric)
71+
}

‎internal/component/database_observability/mysql/collector/connection_info_test.go‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
package collector
2+
3+
import (
4+
"context"
5+
"fmt"
6+
"strings"
7+
"testing"
8+
9+
"github.com/prometheus/client_golang/prometheus"
10+
"github.com/prometheus/client_golang/prometheus/testutil"
11+
"github.com/stretchr/testify/require"
12+
"go.uber.org/goleak"
13+
)
14+
15+
func TestConnectionInfo(t *testing.T) {
16+
defer goleak.VerifyNone(t)
17+
18+
const baseExpectedMetrics = `
19+
# HELP connection_info Information about the connection
20+
# TYPE connection_info gauge
21+
connection_info{db_instance_identifier="%s",provider_name="%s",region="%s"} 1
22+
`
23+
24+
testCases := []struct {
25+
name string
26+
dsn string
27+
expectedMetrics string
28+
}{
29+
{
30+
name: "generic dsn",
31+
dsn: "user:pass@tcp(localhost:3306)/db",
32+
expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "unknown", "unknown", "unknown"),
33+
},
34+
{
35+
name: "AWS/RDS dsn",
36+
dsn: "user:pass@tcp(products-db.abc123xyz.us-east-1.rds.amazonaws.com:3306)/db",
37+
expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "products-db", "aws", "us-east-1"),
38+
},
39+
}
40+
41+
for _, tc := range testCases {
42+
reg := prometheus.NewRegistry()
43+
44+
collector, err := NewConnectionInfo(ConnectionInfoArguments{
45+
DSN: tc.dsn,
46+
Registry: reg,
47+
})
48+
require.NoError(t, err)
49+
require.NotNil(t, collector)
50+
51+
err = collector.Start(context.Background())
52+
require.NoError(t, err)
53+
54+
err = testutil.GatherAndCompare(reg, strings.NewReader(tc.expectedMetrics))
55+
require.NoError(t, err)
56+
}
57+
}

0 commit comments

Comments
 (0)