|
| 1 | +--- |
| 2 | +navigation_title: Elastic {{product.apm}} connector |
| 3 | +description: The Elastic {{product.apm}} connector is an OpenTelemetry Collector component that generates pre-aggregated APM metrics from trace data. |
| 4 | +applies_to: |
| 5 | + stack: |
| 6 | + serverless: |
| 7 | + observability: |
| 8 | + product: |
| 9 | + edot_collector: |
| 10 | +products: |
| 11 | + - id: elastic-agent |
| 12 | + - id: observability |
| 13 | + - id: edot-collector |
| 14 | +--- |
| 15 | + |
| 16 | +# Elastic {{product.apm}} connector |
| 17 | + |
| 18 | +The Elastic {{product.apm}} connector generates pre-aggregated metrics from OpenTelemetry trace data. These metrics enable key {{product.apm}} features like service maps, transaction histograms, and service-level indicators with fast query performance. Instead of calculating metrics on the fly from potentially millions of transactions, the {{product.apm}} UIs can query pre-computed metric documents for quick data visualization. |
| 19 | + |
| 20 | +The connector works together with the [Elastic {{product.apm}} processor](elasticapmprocessor.md), which enriches OpenTelemetry traces with Elastic-specific attributes to ensure optimal compatibility with Elastic {{product.apm}} UIs. |
| 21 | + |
| 22 | +## Default usage in EDOT |
| 23 | + |
| 24 | +The `elasticapm` connector is included by default in EDOT Collector deployments that ingest trace data directly into {{es}}. It's not needed when using the [{{motlp}}](opentelemetry://reference/motlp.md), as the metric aggregation happens server-side. |
| 25 | + |
| 26 | +### Standalone deployments |
| 27 | + |
| 28 | +In standalone deployments, the Elastic {{product.apm}} connector is used in both agent and gateway modes: |
| 29 | + |
| 30 | +* **Agent mode**: The connector is part of the default [application and traces collection pipeline](../config/default-config-standalone.md#application-and-traces-collection-pipeline). It receives trace data from the pipeline, generates metrics, and forwards them to {{es}}. |
| 31 | + |
| 32 | +* **Gateway mode**: The connector is part of the [Gateway mode pipeline](../config/default-config-standalone.md#gateway-mode), where it generates metrics from traces received from other collectors running in agent mode before ingesting them into {{es}}. |
| 33 | + |
| 34 | + |
| 35 | +### Kubernetes deployment |
| 36 | + |
| 37 | +In Kubernetes, the Elastic {{product.apm}} connector runs in the [Gateway collectors pipeline](../config/default-config-k8s.md#gateway-collectors-pipeline) when using direct ingestion to {{es}}. The Gateway receives traces from DaemonSet collectors, generates {{product.apm}} metrics, and writes both metrics and traces to {{es}}. |
| 38 | + |
| 39 | +For more details about the Kubernetes configuration, refer to [Default configuration (Kubernetes)](../config/default-config-k8s.md). |
| 40 | + |
| 41 | +## Example configuration |
| 42 | + |
| 43 | +The Elastic {{product.apm}} connector typically requires minimal configuration. Usually, an empty configuration block is sufficient: |
| 44 | + |
| 45 | +```yaml |
| 46 | +connectors: |
| 47 | + elasticapm: {} |
| 48 | +``` |
| 49 | +
|
| 50 | +When combined with the `elasticapm` processor in a complete pipeline: |
| 51 | + |
| 52 | +```yaml |
| 53 | +receivers: |
| 54 | + otlp: |
| 55 | + protocols: |
| 56 | + grpc: |
| 57 | + endpoint: 0.0.0.0:4317 |
| 58 | + http: |
| 59 | + endpoint: 0.0.0.0:4318 |
| 60 | +
|
| 61 | +connectors: |
| 62 | + elasticapm: {} |
| 63 | +
|
| 64 | +processors: |
| 65 | + batch: |
| 66 | + send_batch_size: 1000 |
| 67 | + timeout: 1s |
| 68 | + elasticapm: {} |
| 69 | +
|
| 70 | +exporters: |
| 71 | + elasticsearch/otel: |
| 72 | + endpoints: |
| 73 | + - ${ELASTIC_ENDPOINT} |
| 74 | + api_key: ${ELASTIC_API_KEY} |
| 75 | + mapping: |
| 76 | + mode: otel |
| 77 | +
|
| 78 | +service: |
| 79 | + pipelines: |
| 80 | + traces: |
| 81 | + receivers: [otlp] |
| 82 | + processors: [batch, elasticapm] |
| 83 | + exporters: [elasticapm, elasticsearch/otel] |
| 84 | + |
| 85 | + metrics/aggregated-otel-metrics: |
| 86 | + receivers: [elasticapm] |
| 87 | + processors: [] |
| 88 | + exporters: [elasticsearch/otel] |
| 89 | +``` |
| 90 | + |
| 91 | +In this configuration, the `elasticapm` connector appears as both an exporter in the traces pipeline to generate {{product.apm}} metrics and as a receiver in the metrics pipeline to forward those metrics to {{es}}. |
| 92 | + |
| 93 | +## Generated metrics |
| 94 | + |
| 95 | +The Elastic {{product.apm}} connector generates the following types of aggregated metrics from trace data: |
| 96 | + |
| 97 | +| Metric Type | Description | |
| 98 | +|------------|-------------| |
| 99 | +| Transaction metrics | Aggregated statistics about service transactions, including throughput, latency distributions, and success rates. These metrics power service overview pages and transaction group views. | |
| 100 | +| Service destination metrics | Metrics that track dependencies between services, showing how services communicate with databases, message queues, and other external systems. These metrics are used to build service maps. | |
| 101 | +| Span metrics | Detailed metrics about individual span operations, including database queries, HTTP calls, and other operations. These provide granular performance insights for specific operations. | |
| 102 | +| Service summary metrics | High-level service health metrics including error rates, throughput, and overall latency. These metrics enable quick health checks and alerting. | |
| 103 | + |
| 104 | +All metrics are generated with appropriate aggregation periods and follow Elastic's metric schema for seamless integration with {{product.apm}} UIs. |
| 105 | + |
| 106 | +## Aggregation intervals |
| 107 | + |
| 108 | +The connector aggregates metrics over multiple time intervals to provide different granularities for analysis. By default, metrics are aggregated at three intervals: 1 minute, 10 minutes, and 60 minutes. |
| 109 | + |
| 110 | +## Best practices |
| 111 | + |
| 112 | +Follow these recommendations when using the Elastic {{product.apm}} connector: |
| 113 | + |
| 114 | +* **Always pair with the elasticapm processor**: The connector and processor work together to provide the full Elastic {{product.apm}} experience. The processor enriches traces while the connector generates {{product.apm}} metrics. Include both in your pipeline configuration for complete functionality. |
| 115 | + |
| 116 | +* **Place the connector as an exporter in the traces pipeline**: Configure the Elastic {{product.apm}} connector as an exporter in your traces pipeline, alongside your final data destination. This ensures the connector receives processed trace data and can generate accurate metrics. |
| 117 | + |
| 118 | +* **Create a separate metrics pipeline for connector output**: Set up a dedicated metrics pipeline with the connector as the receiver. This isolates metric handling and makes it easier to apply metric-specific processing if needed. |
| 119 | + |
| 120 | +* **Use only for direct {{es}} ingestion**: If you're using the {{motlp}}, you don't need the Elastic {{product.apm}} connector, because the endpoint handles metric aggregation automatically. Using both can cause conflicts or duplicate metrics. |
| 121 | + |
| 122 | +* **Keep the connector updated**: The Elastic {{product.apm}} connector evolves with new Elastic {{product.apm}} features. Keep your EDOT Collector version current to benefit from the latest enhancements and compatibility improvements. |
| 123 | + |
| 124 | +* **Monitor connector performance**: The connector aggregates metrics in memory before flushing. For high-throughput environments, monitor memory usage and adjust the aggregation interval or deployment resources as needed. |
| 125 | + |
| 126 | +## Limitations |
| 127 | + |
| 128 | +Be aware of these constraints and behaviors when using the Elastic {{product.apm}} connector: |
| 129 | + |
| 130 | +* **Required for Elastic {{product.apm}} metrics**: Without the Elastic {{product.apm}} connector, you'll only have raw trace data in {{es}}. Service maps, transaction histograms, and other metric-driven {{product.apm}} features might require the pre-aggregated metrics that this connector generates. |
| 131 | + |
| 132 | +* **Not available in contrib OTel Collector**: The Elastic {{product.apm}} connector is an Elastic-specific component not included in the standard OpenTelemetry Collector or Collector Contrib distributions. To use it, you must either use EDOT Collector or [build a custom collector](../custom-collector.md) that includes Elastic's components. |
| 133 | + |
| 134 | +* **Memory usage scales with cardinality**: The connector maintains in-memory aggregations for unique combinations of service names, transaction names, and other dimensions. High-cardinality data (many unique values) increases memory requirements. Monitor memory usage in high-cardinality environments. |
| 135 | + |
| 136 | +* **Minimal configuration options**: Unlike some connectors, the Elastic {{product.apm}} connector operates with mostly fixed behavior and offers few configuration parameters. While this simplifies setup, it also means you have limited ability to customize the aggregation logic. |
| 137 | + |
| 138 | + |
| 139 | +## Resources |
| 140 | + |
| 141 | +* [Contrib component: elasticapmconnector](https://github.com/elastic/opentelemetry-collector-components/tree/main/connector/elasticapmconnector) |
| 142 | +* [Elastic {{product.apm}} processor](elasticapmprocessor.md) |
| 143 | +* [Default configuration (Standalone)](../config/default-config-standalone.md#application-and-traces-collection-pipeline) |
| 144 | +* [Default configuration (Kubernetes)](../config/default-config-k8s.md) |
| 145 | +* [Build a custom collector with Elastic components](../custom-collector.md) |
| 146 | + |
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?v=4&size=40){kind=avatar}
0 commit comments