feat(opentelemetry sink): add automatic native log and trace to OTLP conversion

[szibis]

Summary

Add automatic conversion from Vector's native flat log and trace formats to OTLP (OpenTelemetry Protocol) format in the opentelemetry sink's otlp codec.

Problem: Users currently need 50+ lines of complex VRL to manually build the nested OTLP structure (resourceLogs → scopeLogs → logRecords with KeyValue arrays), and trace events fail entirely when sent to the OTLP sink without passthrough mode.

Solution: The OTLP encoder now automatically detects native log and trace events and converts them to valid OTLP protobuf. Pre-formatted OTLP events continue to use passthrough encoding (backward compatible).

Scope

Signal	Native → OTLP	Passthrough (use_otlp_decoding: true)
Logs	✅ Auto-converts	✅ Passthrough
Traces	✅ Auto-converts	✅ Passthrough
Metrics	✅ Auto-converts (see #24897)	✅ Passthrough

Performance Impact (Logs)

Scenario	OLD (VRL + encode)	NEW (auto-convert)	Improvement
Single event	374 µs	340 µs	9% faster
Batch 100	2,886 µs	620 µs	4.7x faster
Batch throughput	58 MiB/s	269 MiB/s	4.7x higher

Architecture Comparison

%%{init: {'theme': 'base', 'themeVariables': { 'lineColor': '#000000', 'primaryTextColor': '#000000'}}}%%
flowchart TB
    subgraph OLD["OLD: 50 lines VRL"]
        direction LR
        O1[Source] ==> O2[VRL Transform]
        O2 ==> O3[OTLP Encoder]
        O3 ==> O4[Collector]
    end

    subgraph NEW["NEW: Zero VRL"]
        direction LR
        N1[Source] ==> N2[OTLP Encoder]
        N2 ==> N3[Collector]
    end

    OLD ==> NEW

    style O1 fill:#ffffff,stroke:#000000,stroke-width:2px,color:#000000
    style O2 fill:#cccccc,stroke:#000000,stroke-width:3px,color:#000000
    style O3 fill:#ffffff,stroke:#000000,stroke-width:2px,color:#000000
    style O4 fill:#ffffff,stroke:#000000,stroke-width:2px,color:#000000
    style N1 fill:#ffffff,stroke:#000000,stroke-width:2px,color:#000000
    style N2 fill:#999999,stroke:#000000,stroke-width:3px,color:#000000
    style N3 fill:#ffffff,stroke:#000000,stroke-width:2px,color:#000000

    linkStyle default stroke:#000000,stroke-width:2px

Loading

Before vs After Comparison

Capability	Before PR	After PR
Native logs → OTLP sink
use_otlp_decoding: false + codec: otlp	❌ encoding error	✅ auto-converts
Non-OTLP source (file/syslog) + codec: otlp	❌ encoding error	✅ auto-converts
use_otlp_decoding: true + codec: otlp	✅ passthrough	✅ passthrough
Manual VRL rebuild needed	❌ 50+ lines required	✅ not needed
Native traces → OTLP sink
Trace events + codec: otlp	❌ encoding error	✅ auto-converts
use_otlp_decoding: true + codec: otlp	✅ passthrough	✅ passthrough
Metrics → OTLP sink
use_otlp_decoding: true + codec: otlp	✅ passthrough	✅ passthrough
Native metrics + codec: otlp	❌ not supported	✅ auto-converts (see #24897)
Auto-conversion field mapping (logs)
.message → body.stringValue	❌ manual VRL	✅ automatic
.severity_text → severityText	❌ manual VRL	✅ automatic
.severity_number → severityNumber	❌ manual VRL	✅ automatic
.attributes.* → logRecords[].attributes[]	❌ manual VRL	✅ automatic
.resources.* → resource.attributes[]	❌ manual VRL	✅ automatic
.trace_id → traceId	❌ manual VRL	✅ automatic
.span_id → spanId	❌ manual VRL	✅ automatic
.timestamp → timeUnixNano	❌ manual VRL	✅ automatic
Severity inferred from text	❌	✅
Hex validation on trace/span IDs	❌	✅
Multiple timestamp formats	❌	✅
Auto-conversion field mapping (traces)
.trace_id → traceId (16 bytes)	❌ encoding error	✅ automatic
.span_id → spanId (8 bytes)	❌ encoding error	✅ automatic
.parent_span_id → parentSpanId	❌ encoding error	✅ automatic
.name → name	❌ encoding error	✅ automatic
.kind → kind	❌ encoding error	✅ automatic
.start_time_unix_nano / .end_time_unix_nano	❌ encoding error	✅ automatic
.attributes.* → attributes[]	❌ encoding error	✅ automatic
.resources.* → resource.attributes[]	❌ encoding error	✅ automatic
.events → events[] (span events)	❌ encoding error	✅ automatic
.links → links[] (span links)	❌ encoding error	✅ automatic
.status → status (message, code)	❌ encoding error	✅ automatic
Data integrity (native → OTLP)
String body	❌ fails	✅
Structured body (kvlist/array)	❌ fails	⚠️ stringified
String attributes	❌ fails	✅
Complex attributes (array/nested)	❌ fails	⚠️ best-effort
ResourceLogs batching	❌ fails	✅ 1:1 per event
ScopeLogs grouping	❌ fails	✅ 1:1 per event
Wire efficiency
Batching structure	❌ destroyed — 1 record per plog.Logs	✅ preserved — all records stay together
Wire payload size	❌ N × (resource + scope + 1 record)	✅ 1 × (resource + scope + N records)
Error handling
Missing fields	❌ crash/drop	✅ graceful defaults
Invalid trace_id hex	❌ crash/drop	✅ warning, field omitted
Unknown timestamp format	❌ crash/drop	✅ warning, uses 0
Performance
Single event throughput	N/A (failed)	✅ 9% faster than VRL
Batch throughput (100 events)	N/A (failed)	✅ 4.7× faster than VRL
Throughput (MiB/s)	58 (VRL workaround)	269

Vector configuration

Before (Complex VRL Required)

50+ lines of VRL transformation

sources:
  otel_source:
    type: opentelemetry
    grpc:
      address: 0.0.0.0:4317

transforms:
  build_otlp_structure:
    type: remap
    inputs: ["otel_source.logs"]
    source: |
      resource_attrs = []
      if exists(.resources) {
        for_each(object!(.resources)) -> |k, v| {
          resource_attrs = push(resource_attrs, {
            "key": k,
            "value": { "stringValue": to_string(v) ?? "" }
          })
        }
      }

      log_attrs = []
      if exists(.attributes) {
        for_each(object!(.attributes)) -> |k, v| {
          attr_value = if is_boolean(v) {
            { "boolValue": v }
          } else if is_integer(v) {
            { "intValue": to_string!(v) }
          } else if is_float(v) {
            { "doubleValue": v }
          } else {
            { "stringValue": to_string(v) ?? "" }
          }
          log_attrs = push(log_attrs, { "key": k, "value": attr_value })
        }
      }

      .resource_logs = [{
        "resource": { "attributes": resource_attrs },
        "scopeLogs": [{
          "scope": {
            "name": .scope.name ?? "",
            "version": .scope.version ?? ""
          },
          "logRecords": [{
            "timeUnixNano": to_string(to_unix_timestamp(.timestamp, unit: "nanoseconds")),
            "severityText": .severity_text ?? "INFO",
            "severityNumber": .severity_number ?? 9,
            "body": { "stringValue": .message ?? "" },
            "attributes": log_attrs,
            "traceId": .trace_id ?? "",
            "spanId": .span_id ?? ""
          }]
        }]
      }]

sinks:
  otel_out:
    type: opentelemetry
    inputs: ["build_otlp_structure"]
    endpoint: http://collector:4317
    encoding:
      codec: otlp

After (Zero VRL)

sources:
  otel_source:
    type: opentelemetry
    grpc:
      address: 0.0.0.0:4317
    # use_otlp_decoding: false (default)

sinks:
  otel_out:
    type: opentelemetry
    inputs: ["otel_source.logs"]
    endpoint: http://collector:4317
    encoding:
      codec: otlp  # Automatic conversion!

Traces (Zero VRL)

sources:
  otel_source:
    type: opentelemetry
    grpc:
      address: 0.0.0.0:4317

sinks:
  otel_out:
    type: opentelemetry
    inputs: ["otel_source.traces"]
    endpoint: http://collector:4317
    encoding:
      codec: otlp  # Native traces auto-converted to OTLP protobuf

Native Metrics (Auto-Convert via #24897)

sources:
  host_metrics:
    type: host_metrics

sinks:
  otel_out:
    type: opentelemetry
    inputs: ["host_metrics"]
    endpoint: http://collector:4317
    encoding:
      codec: otlp  # Native metrics auto-converted to OTLP protobuf (see #24897)

With Optional Enrichment

sources:
  otel_source:
    type: opentelemetry
    grpc:
      address: 0.0.0.0:4317

transforms:
  enrich:
    type: remap
    inputs: ["otel_source.logs"]
    source: |
      # Simple flat field access - no nested structure needed
      .attributes.processed_by = "vector"
      .resources."deployment.region" = "us-west-2"

sinks:
  otel_out:
    type: opentelemetry
    inputs: ["enrich"]
    endpoint: http://collector:4317
    encoding:
      codec: otlp

Supported Native Log Format

{
  "message": "User login successful",
  "timestamp": "2024-01-15T10:30:00Z",
  "severity_text": "INFO",
  "severity_number": 9,
  "trace_id": "0123456789abcdef0123456789abcdef",
  "span_id": "fedcba9876543210",
  "attributes": {
    "user_id": "user-12345",
    "duration_ms": 42.5
  },
  "resources": {
    "service.name": "auth-service"
  },
  "scope": {
    "name": "auth-module",
    "version": "1.0.0"
  }
}

Supported Native Trace Format

{
  "trace_id": "0123456789abcdef0123456789abcdef",
  "span_id": "fedcba9876543210",
  "parent_span_id": "abcdef0123456789",
  "name": "HTTP GET /api/users",
  "kind": 2,
  "start_time_unix_nano": 1705312200000000000,
  "end_time_unix_nano": 1705312200042000000,
  "attributes": {
    "http.method": "GET",
    "http.status_code": 200
  },
  "resources": {
    "service.name": "api-gateway"
  },
  "status": {
    "code": 1,
    "message": "OK"
  },
  "events": [
    {
      "name": "request.start",
      "time_unix_nano": 1705312200000000000,
      "attributes": { "component": "handler" }
    }
  ],
  "links": []
}

Log Field Mapping

Native Field	OTLP Field	Notes
.message / .body / .msg	body.stringValue	Auto-detected
.timestamp	timeUnixNano	chrono, epoch, RFC3339
.severity_text	severityText
.severity_number	severityNumber	Auto-inferred from text if missing
.trace_id	traceId	Hex string → 16 bytes
.span_id	spanId	Hex string → 8 bytes
.attributes.*	attributes[]	Object → KeyValue array
.resources.*	resource.attributes[]	Object → KeyValue array
.scope.name	scope.name	Instrumentation scope
.scope.version	scope.version	Instrumentation scope

Trace Field Mapping

Native Field	OTLP Field	Notes
.trace_id	traceId	Hex string → 16 bytes
.span_id	spanId	Hex string → 8 bytes
.parent_span_id	parentSpanId	Hex string → 8 bytes
.name	name	Span operation name
.kind	kind	SpanKind enum (0-5)
.start_time_unix_nano	startTimeUnixNano	Nanosecond timestamp
.end_time_unix_nano	endTimeUnixNano	Nanosecond timestamp
.trace_state	traceState	W3C trace state string
.attributes.*	attributes[]	Object → KeyValue array
.resources.*	resource.attributes[]	Object → KeyValue array
.events[]	events[]	Span events (name, time, attributes)
.links[]	links[]	Span links (trace_id, span_id, attributes)
.status.code	status.code	StatusCode enum
.status.message	status.message	Status description
.dropped_attributes_count	droppedAttributesCount
.dropped_events_count	droppedEventsCount
.dropped_links_count	droppedLinksCount

How did you test this PR?

Unit Tests

cargo test -p opentelemetry-proto  # 72+ tests (log + trace conversion)
cargo test -p codecs --test otlp   # 28 tests

Tests cover:

• Basic encoding, error handling, type conversion, safe timestamp overflow handling
• Logs: Timestamp formats, severity inference, message fallbacks, hex validation
• Traces: Basic fields, timestamps, parent_span_id, trace_state, attributes, resources, status, events, links, dropped counts, invalid/wrong-length IDs, mixed valid/invalid fields
• Roundtrip encode/decode for both signals
• Validation edge cases (negative timestamps, severity clamping, ID length)

E2E Tests

cargo vdev test e2e opentelemetry-native

Docker Compose with telemetrygen validates:

• Native logs convert to valid OTLP
• Field preservation through pipeline
• Custom attributes via VRL transforms
• Correct event counting metrics

Benchmarks

cargo bench --bench codecs --features codecs-benches -- otlp_encoding

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Pre-formatted OTLP events (with resourceLogs/resourceSpans/resourceMetrics fields) continue using existing passthrough path. Native metric events return an explicit error with a clear message (same behavior as before for unsupported types).

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Issue/PR	Relationship
#22789	Directly related - Our codec-level approach provides the same outcome with simpler implementation
#24515	Solves - User requesting OTLP transform helper; this eliminates the need
#23971	Solves root cause - OTLP encoding failing; our encoder handles all formats
#22550	Complementary - Adds metrics support; our patterns can extend to metrics
#24897	Companion PR - Adds native metric → OTLP conversion (Counter, Gauge, Histogram, Summary, Distribution, Set)

• Related: feat!(opentelemetry sink): Add support for encoders #22789
• Closes: Support OTLP transform for opentelemetry sink #24515
• Related: OTLP encoding not working with OpenTelemetry sink #23971
• Related: feat(sink): add opentelemetry metrics support #22550
• Related: feat(opentelemetry sink): add native metric to OTLP conversion #24897
• Related: fix(opentelemetry lib): decode missing scope, schema_url, and resource fields #24905

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[szibis]

I have read the CLA Document and I hereby sign the CLA

[maycmlee]

Hi, thanks for opening this PR! I'm the docs reviewer. Just some small doc suggestions and also a question about using version numbers.

[szibis]

@pront this is PR #24897 that solves metrics based on this PR as a reference which should close the whole problem after both PR's merged.

[pront]

Hi @szibis, I will prioritize reviewing this PR.

[szibis]

@pront I think this should goes first - Companion decode-side fix: #24905 adds decode for missing scope, schema_url, and resource.dropped_attributes_count fields across logs, traces, and metrics. This ensures the fields that this PR's encode path handles are actually populated by the decode side.

[szibis]

Local Integration & Edge Case Testing

Set up a local test pipeline — Vector OTLP source on ports 4317/4318, piped through an OTLP sink to a second Vector instance on 4319/4320 for full roundtrip testing. Used grpcurl with proto files from lib/opentelemetry-proto/src/proto/opentelemetry-proto/ to send payloads. Sent 5 different log payloads and 2 trace payloads via grpcurl (string body, structured body, various severity levels, spans with events/links) through both passthrough and roundtrip configs. This PR covers native-to-OTLP conversion for both logs and traces, so both signal types went through the full encode path. Decoded output in the console sink looked correct, no bugs found. Both conversion paths are solid.

Fixed pre-existing clippy approx_constant lint in logs.rs and common.rs (test values like 3.14 triggering it).

Added 10 edge case tests to logs.rs, verified with cargo test -p opentelemetry-proto --lib logs::edge_case_tests. Covers array and nested map bodies, non-string body types, empty body, severity zero, observed timestamp without regular timestamp, unicode/emoji everywhere, multiple records per scope, and u64::MAX overflow. The trace conversion tests for this PR are already covered in spans.rs via the native_trace_conversion_tests module that was part of this branch.

Performance

Pushed 10k log events and 10k trace events in batches of 100 through the roundtrip pipeline — logs at ~4.7k events/s, traces at ~5k events/s, zero errors. These numbers mostly reflect grpcurl client overhead — a compiled gRPC client would be significantly faster. The important takeaway is stability: no crashes, no data corruption, no memory issues under sustained load.

[szibis]

#24905 (OTLP decode refactor)
↓
#24621 (OTLP encode logs/traces)
↓
#24897 (OTLP encode metrics)

Each PR rebases on top of the previous one after merge.

@pront after each merge I will do the cleanup+rebase to make this flow easiest as possible for Vector Team.