Understanding datadog_agent source internal metrics and logs

[techministrator]

Question

Hi vector team and everyone,

May I check and learn more about the internal metrics and logs related to datadog_agent source (ref).

A small context is that I've just recently rolled out the Vector to receive and handle a large amount of metrics traffic from the internal datadog agents (configuring using the observability_pipelines_worker option).

I've also configured quite an extensive dashboards to monitor vector and noticed periodic metric data of component_discarded_events_total with intentional tag/label as false (ref). Usually accompanying by:

• an internal ERROR log of Events dropped at datadog_agent source
• some internal WARN logs of HTTP error and Retrying after error (with some suppressions) at datadog_metrics sink
• memory buffer and usage increased expectedly
• sometimes Adaptive Concurrency RTT gets worse/spiked as well

So I'm trying to understand what was the real reason of the Events dropped issue in this case. In documentation, it just mentions there's due to errors but is it errors from Vector Sink or is it from the Source Datadog Agents in this case? Metrics Events dropped means all the data is permanently lost because I don't enable acknowledgement feature? Any recommendations to mitigate or avoid this issue?

I put some of the most important configurations below to avoid too much configs. Generally, I don't see any errors or discarded events at the transforms stage thus I remove it from this context to avoid noise.

Thank you for spending the time, I know this is a lot of context.

Vector Config

There are many context for the below configs though I'm open for recommendations:

• An average metric event size in our system is 4-5KB thus I have to decrease the max batch events to improve RTT from Datadog API Endpoint. If keeping the default size, it would take >30s and surely dropped along the way.
• Memory events buffer also is being used the maximum usually go up to 70-80% with this 400,000 events config
• Concurrency settings I tried to make it scale out as fast and as many as possible since this is Datadog Platform. They don't limit metrics submissions in their endpoint (ref). But this doesn't prevent 500 error msg returned sometimes and decrease concurrency (though back up quite fast).
• Also due to the above reasons, I decrease max backoff durations.

data_dir: "/var/lib/vector"

sources:
  datadog_agents:
    type: datadog_agent
    address: 0.0.0.0:9000
    disable_logs: true
    disable_traces: true
    store_api_key: true

sinks:
  datadog_platform:
    inputs:
      - datadog_agents
    type: datadog_metrics
    default_api_key: "${DD_API_KEY:?missing DD_API_KEY as default}"
    buffer:
      - type: memory
        max_events: 400000
        when_full: block
    batch:
      max_events: 1000
      timeout_secs: 1
    request:
      retry_attempts: 10
      retry_initial_backoff_secs: 1
      retry_max_duration_secs: 10
      retry_jitter_mode: Full
      timeout_secs: 60
      concurrency: adaptive
      adaptive_concurrency: 
        initial_concurrency: 600
        max_concurrency_limit: 1500
        decrease_ratio: 1.0
        rtt_deviation_scale: 3.0
        ewma_alpha: 0.4

Vector Logs

{
	"id": <obscured>
	"content": {
		"timestamp": "2025-10-01T10:52:09.060Z",
		"tags": [
			"source:undefined",
			"datadog.submission_auth:api_key",
			"datadog.api_key_uuid:504665cb-9c53-496d-b3c6-552e2eb73468"
		],
		"host": "<obscured>",
		"service": "<obscured>",
		"message": "Events dropped",
		"attributes": {
			"reason": "Source send cancelled.",
			"metadata": {
				"kind": "event",
				"module_path": "vector_common::internal_event::component_events_dropped",
				"target": "vector_common::internal_event::component_events_dropped"
			},
			"intentional": false,
			"count": 3044,
			"internal_log_rate_limit": true,
			"pid": 998,
			"source_type": "internal_logs",
			"env": "staging",
			"hostname": "<obscured>",
			"appname": "<obscured>",
			"service": "<obscured>",
			"vector": {
				"component_type": "datadog_agent",
				"component_id": "datadog_agents",
				"component_kind": "source"
			},
			"status": "ERROR",
			"timestamp": 1759315929060
		}
	}
}

[pront]

Hi @techministrator, apologies for the late response. This is a very interesting thread.

Regarding the event drops, the Source send cancelled errors are occurring within Vector, not at the upstream Datadog agents. Timeline:

1. Datadog agent successfully sends metrics to Vector's datadog_agent source
2. Vector attempts to forward them to the datadog_metrics sink
3. The sink experiences HTTP errors/retries (as shown in your WARN logs)
4. With buffer.when_full = "block", Vector's buffer fills up and blocks new incoming requests
5. I suspect the cancellation happens due to client-side timeouts while Vector is blocked waiting for buffer space

Is data Lost? Yes. The Datadog agents receive HTTP 200 OK responses from Vector and consider their data successfully delivered. Vector then drops the events internally due to backpressure. The agents have no way to know this happened and won't retry. This is working as designed, but it is not ideal. It is known limitation. See related discussion: #7186

Acknowledgements for the datadog_agent source might help here.

Did you consider reducing the batch size to potentially alleviate some ingestion issues? There is a maximum payload limit <= 5MB, very roughly I would aim for < 500 metrics per batch.