diff --git a/solutions/observability/applications/otel-rum.md b/solutions/observability/applications/otel-rum.md new file mode 100644 index 0000000000..3cca6803a5 --- /dev/null +++ b/solutions/observability/applications/otel-rum.md @@ -0,0 +1,940 @@ +--- +navigation_title: OpenTelemetry for Real User Monitoring (RUM) +description: Instrument web applications with OpenTelemetry for Real User Monitoring using Elastic Observability. +applies_to: + product: preview + stack: + serverless: + observability: +products: + - id: cloud-serverless + - id: observability +--- + +# OpenTelemetry for Real User Monitoring (RUM) + +:::{important} +Using OpenTelemetry for Real User Monitoring (RUM) with {{product.observability}} is currently in technical preview and should not be used in production environments. This feature may be changed or removed in a future release and has [known limitations](#known-limitations). Features in preview are not subject to support SLAs like GA features. +::: + +You can instrument your web application with OpenTelemetry browser instrumentation for use with {{product.observability}}. The following sections detail the required components and their proper configuration to acquire traces, logs, and metrics from the application to visualize them within {{kib}}. + +## Before you begin [before-you-begin] + +You need an OTLP endpoint to ingest data from the OpenTelemetry RUM instrumentation. If you're setting up a new deployment, [create](/solutions/observability/get-started.md) an {{ecloud}} hosted deployment or {{serverless-short}} project, which includes the [{{motlp}}](opentelemetry://reference/motlp.md). If you own a self-hosted stack or your deployment does not have the {{motlp}}, configure an [EDOT Collector in Gateway mode](https://www.elastic.co/docs/reference/edot-collector/modes#edot-collector-as-gateway). + +After you have prepared the OTLP endpoint, set up a reverse proxy to forward the telemetry from your web application origin to the Collector. You need a reverse proxy for the following reasons: + +- EDOT Collector requires an `Authorization` header with an ApiKey to accept OTLP exports. Setting up the required key in a web application makes it publicly available, which is not advisable. A reverse proxy can help you manage this key securely. +- If you have set up your own EDOT Collector, it's likely to have a different origin than your web application. In this scenario you have to set up [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS) for the web application in the EDOT Collector configuration file. This procedure can be cumbersome if you have to manage many applications. +- You can apply rate limiting or any other mechanisms to control traffic before it reaches the EDOT Collector. + +The following snippet shows the configuration for an NGINX reverse proxy to forward all telemetry to the EDOT Collector located at `collector.example.com` from the origin `webapp.example.com`: + +:::{dropdown} NGINX reverse proxy configuration +```nginx +server { + # Configuration for HTTP/HTTPS goes here + location / { + # Take care of preflight requests + if ($request_method = 'OPTIONS') { + add_header 'Access-Control-Max-Age' 1728000; # 20 days in seconds + add_header 'Access-Control-Allow-Origin' 'webapp.example.com' always; # Set the allowed origins for preflight requests + add_header 'Access-Control-Allow-Headers' 'Accept,Accept-Language,Authorization,Content-Language,Content-Type' always; + add_header 'Access-Control-Allow-Credentials' 'true' always; + add_header 'Content-Type' 'text/plain charset=UTF-8'; + add_header 'Content-Length' 0; + return 204; + } + + add_header 'Access-Control-Allow-Origin' 'webapp.example.com' always; # Set the allowed origins for requests + add_header 'Access-Control-Allow-Credentials' 'true' always; + add_header 'Access-Control-Allow-Headers' 'Accept,Accept-Language,Authorization,Content-Language,Content-Type' always; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + # Set the auth header for the Collector here. It's recommended to follow the security best practices + # for adding secrets into services according to your infrastructure and company policy. A couple of references: + # for Docker: https://docs.docker.com/engine/swarm/secrets/#intermediate-example-use-secrets-with-a-nginx-service + # for K8s: https://kubernetes.io/docs/concepts/configuration/secret/ + proxy_set_header Authorization 'ApiKey ...your Elastic API key...'; + proxy_pass https://collector.example.com:4318; + } +} +``` +::: + +:::{warning} +Avoid using the OTel RUM agent alongside any other {{apm-agent}}, including Elastic {{product.apm}} agents. Running multiple agents might lead to conflicting instrumentation, duplicate telemetry, or other unexpected behavior. +::: + +## Define the basic settings [otel-rum-basic-settings] + +The minimal configuration you need to instrument your web application with OpenTelemetry includes: + +- **OTEL_EXPORTER_OTLP_ENDPOINT**: The full URL of the proxy you have configured in the [OTLP endpoint](#before-you-begin) section. +- **OTEL_RESOURCE_ATTRIBUTES**: A JavaScript object that will be used to define the resource. The most important attributes to define are: + - `service.name` (string): Name of the application you're instrumenting. + - `service.version` (string, optional): A string representing the version or build of your app. + - `deployment.environment.name` (string, optional): Name of the environment where the app runs (if applicable); for example, "prod", "dev", or "staging". +- **OTEL_LOG_LEVEL**: Use this configuration to set the log level of the OpenTelemetry components you're going to use. Supported values are: `error`, `warn`, `info`, `debug`, `verbose`. + +## Set up OpenTelemetry for the browser [otel-rum-set-up-open-telemetry-for-the-browser] + +To instrument your web application with OpenTelemetry in the browser, you need to add a script that configures the essential components, including the context manager, signal providers, processors, and exporters. After adding the script, you can register the instrumentations so they can observe your app and send telemetry data to your endpoint. + +The following starter script is in plain JavaScript. If you use TypeScript, you can adapt this script by changing the file extension to `.ts` and adding the necessary type definitions. OpenTelemetry packages are written in TypeScript, so they include the appropriate type definitions for TypeScript. + +:::{note} +Each signal configuration is independent of the others and has to be configured independently. The OpenTelemetry API defaults to no-op providers for traces, metrics, and logs. +::: + +::::::{stepper} + +:::::{step} Set the configuration + +Start by setting the options to be used by all the signals and the instrumentation code, as well as initializing the internal logger. + +For this step, you need the following dependencies: + +- `@opentelemetry/api`: All the packages are included. Each signal configuration uses it to register the providers for each signal. +- `@opentelemetry/core`: Contains core types and some utilities for the rest of the packages. It parses strings to the correct type. + +To install the dependencies, run the following command: + +```bash +npm install @opentelemetry/api @opentelemetry/core +``` + +After the dependencies are installed, configure the following options: + +:::{dropdown} Configuration example + +```javascript +import { diag, DiagConsoleLogger } from '@opentelemetry/api'; +import { diagLogLevelFromString } from '@opentelemetry/core'; + +// Set the configuration options +const OTEL_LOG_LEVEL = 'info'; // Possible values: error, warn, info, debug, verbose +const OTEL_EXPORTER_OTLP_ENDPOINT = 'https://proxy.example.com'; // Point to your reverse proxy configured in the section above +const OTEL_RESOURCE_ATTRIBUTES = { + 'service.name': 'my-web-app', + 'service.version': '1.2.3', + 'deployment.environment.name': 'qa', + // You can add other attributes +}; + +// Set the log level for the OTEL components +// You can raise the level to "debug" if you want more details +diag.setLogger( + new DiagConsoleLogger(), + { logLevel: diagLogLevelFromString(OTEL_LOG_LEVEL) }, +); +diag.info('OTEL bootstrap', config); +``` + +::: + +::::: + +:::::{step} Define the resource + +A [resource](https://opentelemetry.io/docs/concepts/resources/) is an entity that generates telemetry, with its characteristics captured in resource attributes. An example is a web application operating within a browser that produces telemetry data. + +A standardized set of attributes is specified in [Browser resource semantic conventions](https://opentelemetry.io/docs/specs/semconv/resource/browser/), which can be included alongside those outlined in the configuration section. OpenTelemetry offers resource detectors like `browserDetector` to help set these attributes like brands, mobile, and platform. + +To define the resource, you need the following dependencies: + +- `@opentelemetry/resources`: This package helps you to define and work with resources because a Resource is not a plain object and has some properties (like immutability) and constraints. +- `@opentelemetry/browser-detector`: Detectors help you to define a resource by querying the runtime and environment and resolving some attributes. In this case, the browser detector resolves the language, brands, and mobile attributes of the browser namespace. + +To install the dependencies, run the following command: + +```bash +npm install @opentelemetry/resources @opentelemetry/browser-detector +``` + +After the dependencies are installed, define the resource for your instrumentation with the following code: + +:::{dropdown} Resource definition +```javascript +import { resourceFromAttributes, detectResources } from '@opentelemetry/resources'; +import { browserDetector } from '@opentelemetry/opentelemetry-browser-detector'; + +const detectedResources = detectResources({ detectors: [browserDetector] }); +let resource = resourceFromAttributes(OTEL_RESOURCE_ATTRIBUTES); +resource = resource.merge(detectedResources); +``` +::: +::::: + +:::::{step} (Optional) Configure tracing + +Configure a [TracerProvider](https://opentelemetry.io/docs/concepts/signals/traces/#tracer-provider) to enable instrumentations to transmit traces and allow for the creation of custom spans. A TracerProvider requires several components: + +- **Resource**: The resource to be associated with the spans created by the tracers. You defined this in the second step. +- **Span Processor**: A component that manages the spans generated by the tracers and forwards them to a [SpanExporter](https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-exporter). The exporter should be configured to direct data to an endpoint designated for traces. +- **Span Exporter**: Manages the transmission of spans to the Collector. +- **Context Manager**: A mechanism for managing the active Span context across asynchronous operations and threads. It ensures that when a new Span is created, it correctly identifies its parent Span. + +For this step, you need the following dependencies: + +- `@opentelemetry/sdk-trace-base`: This package contains all the core components to set up tracing regardless of the runtime they're running in (Node.js or browser). +- `@opentelemetry/sdk-trace-web`: This package contains a tracer provider that runs in web browsers. +- `@opentelemetry/exporter-trace-otlp-http`: This package contains the exporter for the HTTP/JSON protocol. +- `@opentelemetry/context-zone`: This package contains a context manager which uses on [zone.js](https://www.npmjs.com/package/zone.js) library. + +```bash +npm install @opentelemetry/sdk-trace-base\ + @opentelemetry/sdk-trace-web\ + @opentelemetry/exporter-trace-otlp-http\ + @opentelemetry/context-zone +``` + +After the dependencies are installed, configure and register a TracerProvider with the following code: + +:::{dropdown} Tracer provider configuration +```javascript +import { trace } from '@opentelemetry/api'; +import { ZoneContextManager } from '@opentelemetry/context-zone'; +import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'; +import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; +import { WebTracerProvider } from '@opentelemetry/sdk-trace-web'; + +// Set the traces endpoint based on the config provided +const tracesEndpoint = `${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces`; + +// Set the tracer provider for instrumentations and calls to the API to start and end spans +const tracerProvider = new WebTracerProvider({ + resource, // All spans will be associated with this resource + spanProcessors: [ + new BatchSpanProcessor(new OTLPTraceExporter({ + url: tracesEndpoint, + })), + ], +}); +tracerProvider.register({ contextManager: new ZoneContextManager() }); +``` +::: + +Now you can use the OpenTelemetry API to get a tracer and start creating your own spans. Instrumentations are also using OpenTelemetry API to get tracers and send telemetry data. Registering intrumentations after having the tracer provider set up ensures they have the right tracers when requested to the API. + +::::: + +:::::{step} (Optional) Configure metrics + +Similar to traces, configure a [MeterProvider](https://opentelemetry.io/docs/concepts/signals/metrics/#meter-provider) for metrics. A MeterProvider requires several components: + +- **Resource**: The resource to be associated with the metrics created by the meters. You defined this in the second step. +- **Metric Reader**: Used to determine how often metrics are collected and what destination they should be exported to. In this case, use a `PeriodicExportingMetricReader` configured to collect and export metrics at a fixed interval. +- **Metric Exporter**: Responsible for serializing and sending the collected and aggregated metric data to a backend observability platform. Use the OTLP/HTTP exporter. + +For this step, you need the following dependencies: + +- `@opentelemetry/sdk-metrics`: This package contains all the required components to set up metrics. +- `@opentelemetry/exporter-metrics-otlp-http`: This package contains the exporter for the HTTP/JSON protocol. + +To install the dependencies, run the following command: + +```bash +npm install @opentelemetry/sdk-metrics @opentelemetry/exporter-metrics-otlp-http +``` + +After the dependencies are installed, configure and register a MeterProvider with the following code: + +:::{dropdown} Meter provider configuration +```javascript +import { metrics } from '@opentelemetry/api'; +import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; +import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; + +// Set the metrics endpoint based on the config provided +const metricsEndpoint = `${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/metrics`; + +// Create metric reader to process metrics and export using OTLP +const metricReader = new PeriodicExportingMetricReader({ + exporter: new OTLPMetricExporter({ url: metricsEndpoint }), +}); + +// Create meter provider to send metrics +const meterProvider = new MeterProvider({ + resource: resource, // All metrics will be associated with this resource + readers: [metricReader], +}); +metrics.setGlobalMeterProvider(meterProvider); +``` +::: +::::: + +:::::{step} (Optional) Configure logs + +Like traces and metrics, configure a [LoggerProvider](https://opentelemetry.io/docs/specs/otel/logs/sdk/#loggerprovider) if you want to record relevant events that are happening in your application using the API or instrumentations. A LoggerProvider requires several components: + +- **Resource**: The resource to be associated with the log records created by the loggers. You defined this in the second step. +- **LogRecord Processor**: A component that manages the log records generated by the loggers and forwards them to a [LogExporter](https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordexporter). The exporter should be configured to direct data to an endpoint designated for logs. +- **Log Exporter**: Manages the transmission of log records to the Collector. + +For this step, you need the following dependencies: + +- `@opentelemetry/api-logs`: This package contains the logs API. This API is not yet included in the generic API package because logs are still experimental. +- `@opentelemetry/sdk-logs`: This package contains all the required components to set up logs. +- `@opentelemetry/exporter-logs-otlp-http`: This package contains the exporter for the HTTP/JSON protocol. + +To install the dependencies, run the following command: + +```bash +npm install @opentelemetry/api-logs @opentelemetry/sdk-logs @opentelemetry/exporter-logs-otlp-http +``` + +After the dependencies are installed, configure and register a LoggerProvider with the following code: + +:::{dropdown} Logger provider configuration +```javascript +import { logs, SeverityNumber } from '@opentelemetry/api-logs'; +import { BatchLogRecordProcessor, LoggerProvider } from '@opentelemetry/sdk-logs'; +import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http'; + +// Set the logs endpoint based on the config provided +const logsEndpoint = `${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/logs`; + +// Configure logging to send to the Collector +const logExporter = new OTLPLogExporter({ url: logsEndpoint }); + +const loggerProvider = new LoggerProvider({ + resource: resource, + processors: [new BatchLogRecordProcessor(logExporter)] +}); +logs.setGlobalLoggerProvider(loggerProvider); +``` +::: +::::: + +:::::{step} Register the instrumentations + +The final step for setting up Real User Monitoring (RUM) through OpenTelemetry is registering the instrumentations. Instrumentations are modules that automatically capture telemetry data, like network requests or DOM interactions, by using the OpenTelemetry API. + +With the OpenTelemetry SDK, resource attributes, and exporters already configured, all telemetry data generated by these registered instrumentations is automatically processed and exported. + +Install the following dependencies: + +- `@opentelemetry/instrumentation`: This package contains the core components of instrumentations along with some utilities. +- `@opentelemetry/auto-instrumentations-web`: This package contains the more common instrumentations for web apps. Which are: + - `@opentelemetry/instrumentation-document-load`: This instrumentation package measures the time it took the document to load and also the load timings of its resources. More info at [instrumentation-document-load](https://www.npmjs.com/package/@opentelemetry/instrumentation-document-load). + - `@opentelemetry/instrumentation-fetch`: This instrumentation keeps track of your web application requests made through the Fetch API. More info at [instrumentation-fetch](https://www.npmjs.com/package/@opentelemetry/instrumentation-fetch). + - `@opentelemetry/instrumentation-xml-http-request`: This instrumentation keeps track of your web application requests made through the XMLHttpRequest API. More info at [instrumentation-xml-http-request](https://www.npmjs.com/package/@opentelemetry/instrumentation-xml-http-request). + - `@opentelemetry/instrumentation-user-interaction`: This instrumentation measures user interactions in your web application. More info at [instrumentation-user-interaction](https://www.npmjs.com/package/@opentelemetry/instrumentation-user-interaction). +- `@opentelemetry/instrumentation-long-task`: This instrumentation is not part of the auto instrumentations package. It gathers information about long tasks being executed in your browser, helping to spot issues like unresponsive UI in your web application. More info at [instrumentation-long-task](https://www.npmjs.com/package/@opentelemetry/instrumentation-long-task). + +To install the dependencies, run the following command: + +```bash +npm install @opentelemetry/instrumentation\ + @opentelemetry/auto-instrumentations-web\ + @opentelemetry/instrumentation-long-task +``` + +After the dependencies are installed, configure and register instrumentations with the following code: + +:::{dropdown} Instrumentations registration +```javascript +import { registerInstrumentations } from '@opentelemetry/instrumentation'; +import { getWebAutoInstrumentations } from '@opentelemetry/auto-instrumentations-web'; +import { LongTaskInstrumentation } from '@opentelemetry/instrumentation-long-task'; + + +// Register instrumentations +registerInstrumentations({ + instrumentations: [ + getWebAutoInstrumentations(), + new LongTaskInstrumentation(), + ], +}); +``` +::: +::::: + +:::::{step} Complete the setup script + +All these pieces together give you a complete setup of all the signals for your web site or application. For convenience, it's better to have it in a separate file which can be named, for example, `telemetry.js`. This file should export a function that accepts the configuration allowing you to reuse the setup across different UIs. + +To install all the dependencies needed for the complete setup, run the following command: + +:::{dropdown} Install setup script dependencies +```bash +npm install @opentelemetry/api\ + @opentelemetry/core\ + @opentelemetry/resources\ + @opentelemetry/browser-detector\ + @opentelemetry/sdk-trace-base\ + @opentelemetry/sdk-trace-web\ + @opentelemetry/context-zone\ + @opentelemetry/exporter-trace-otlp-http\ + @opentelemetry/sdk-metrics\ + @opentelemetry/exporter-metrics-otlp-http\ + @opentelemetry/api-logs\ + @opentelemetry/sdk-logs\ + @opentelemetry/exporter-logs-otlp-http\ + @opentelemetry/instrumentation\ + @opentelemetry/auto-instrumentations-web\ + @opentelemetry/instrumentation-long-task +``` +::: + +After the dependencies are installed, you can wrap the setup in a function with the following code: + +:::{dropdown} Complete setup script example +```javascript +// file: telemetry.js +import { diag, DiagConsoleLogger, trace, metrics } from '@opentelemetry/api'; +import { diagLogLevelFromString } from '@opentelemetry/core'; +import { resourceFromAttributes, detectResources } from '@opentelemetry/resources'; +import { browserDetector } from '@opentelemetry/opentelemetry-browser-detector'; +import { ZoneContextManager } from '@opentelemetry/context-zone'; +import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'; +import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; +import { WebTracerProvider } from '@opentelemetry/sdk-trace-web'; +import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; +import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; +import { logs, SeverityNumber } from '@opentelemetry/api-logs'; +import { BatchLogRecordProcessor, LoggerProvider } from '@opentelemetry/sdk-logs'; +import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http'; +import { registerInstrumentations } from '@opentelemetry/instrumentation'; +import { getWebAutoInstrumentations } from '@opentelemetry/auto-instrumentations-web'; +import { LongTaskInstrumentation } from '@opentelemetry/instrumentation-long-task'; + +const initDone = Symbol('OTEL initialized'); + +// Expected properties of the config object: +// - logLevel +// - endpoint +// - resourceAttributes +export function initOpenTelemetry(config) { + // To avoid multiple calls + if (window[initDone]) { + return; + } + window[initDone] = true; + diag.setLogger( + new DiagConsoleLogger(), + { logLevel: diagLogLevelFromString(config.logLevel) }, + ); + diag.info('OTEL bootstrap', config); + + // Resource definition + const detectedResources = detectResources({ detectors: [browserDetector] }); + const resource = resourceFromAttributes(config.resourceAttributes) + .merge(detectedResources); + + // Trace signal setup + const tracesEndpoint = `${config.endpoint}/v1/traces`; + const tracerProvider = new WebTracerProvider({ + resource, + spanProcessors: [ + new BatchSpanProcessor(new OTLPTraceExporter({ + url: tracesEndpoint, + })), + ], + }); + tracerProvider.register({ contextManager: new ZoneContextManager() }) + + // Metrics signal setup + const metricsEndpoint = `${config.endpoint}/v1/metrics`; + const metricReader = new PeriodicExportingMetricReader({ + exporter: new OTLPMetricExporter({ url: metricsEndpoint }), + }); + const meterProvider = new MeterProvider({ + resource: resource, + readers: [metricReader], + }); + metrics.setGlobalMeterProvider(meterProvider); + + // Logs signal setup + const logsEndpoint = `${config.endpoint}/v1/logs`; + const logExporter = new OTLPLogExporter({ url: logsEndpoint }); + + const loggerProvider = new LoggerProvider({ + resource: resource, + processors: [new BatchLogRecordProcessor(logExporter)] + }); + logs.setGlobalLoggerProvider(loggerProvider); + + // Register instrumentations + registerInstrumentations({ + instrumentations: [ + getWebAutoInstrumentations(), + new LongTaskInstrumentation(), + ], + }); +} +``` +::: +::::: +:::::: + +## Integrate with your application + +With the setup script in a single file, you can now apply it to your web application. You can choose from two main approaches: + +1. **Import the code**: Use your build tooling to manage the dependencies and integrate the code into the application bundle. This is the simplest option and is recommended, although it increases the size of your application bundle. + +2. **Bundle in a file**: Use a bundler to generate a separate JavaScript file that you include in the `
` section of your HTML page. This approach keeps the telemetry code separate from your application bundle. + +### Import the code + +This approach is recommended, especially if you're using a web framework. The build tooling manages the dependencies and integrates the code into the application bundle. However, this approach increases the size of your application bundle. + +For example, if you're using Webpack, you can import the code like this: + +:::{dropdown} Example: Import telemetry.js in your app + +```javascript +import { initOpenTelemetry } from 'telemetry.js'; + +initOpenTelemetry({ + logLevel: 'info', + endpoint: 'https://proxy.example.com', + resourceAttributes: { + 'service.name': 'my-web-app', + 'service.version': '1', + } +}); +``` +::: + +If you're using a framework, there are some suitable places for it, depending on which one you're using: + +| Framework | Method | +|-----------|-------------| +| **React** | Create a component which initializes the instrumentation when mounted. The component should be added as child of the `