Skip to content

Add metric pipelines type to Observability Pipelines API #2853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
api-clients-generation-pipeline wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add metric pipelines type to Observability Pipelines API #2853

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
676 changes: 582 additions & 94 deletions .generator/schemas/v2/openapi.yaml
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions features/v2/observability_pipelines.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Feature: Observability Pipelines
Scenario: Create a new pipeline returns "Conflict" response
Given operation "CreatePipeline" enabled
And new "CreatePipeline" request
And body with value {"data": {"attributes": {"config": {"destinations": [{"id": "datadog-logs-destination", "inputs": ["filter-processor"], "type": "datadog_logs"}], "processors": [{"enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": ["datadog-agent-source"], "processors": [{"enabled": true, "id": "filter-processor", "include": "status:error", "type": "filter"}, {"enabled": true, "field": "message", "id": "json-processor", "include": "*", "type": "parse_json"}]}], "sources": [{"id": "datadog-agent-source", "type": "datadog_agent"}]}, "name": "Main Observability Pipeline"}, "type": "pipelines"}}
And body with value {"data": {"attributes": {"config": {"destinations": [{"id": "datadog-logs-destination", "inputs": ["filter-processor"], "type": "datadog_logs"}], "pipeline_type": "logs", "processors": [{"enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": ["datadog-agent-source"], "processors": [{"enabled": true, "id": "filter-processor", "include": "status:error", "type": "filter"}, {"enabled": true, "field": "message", "id": "json-processor", "include": "*", "type": "parse_json"}]}], "sources": [{"id": "datadog-agent-source", "type": "datadog_agent"}]}, "name": "Main Observability Pipeline"}, "type": "pipelines"}}
When the request is sent
Then the response status is 409 Conflict

Expand Down Expand Up @@ -115,7 +115,7 @@ Feature: Observability Pipelines
Given operation "UpdatePipeline" enabled
And new "UpdatePipeline" request
And request contains "pipeline_id" parameter from "REPLACE.ME"
And body with value {"data": {"attributes": {"config": {"destinations": [{"id": "datadog-logs-destination", "inputs": ["filter-processor"], "type": "datadog_logs"}], "processors": [{"enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": ["datadog-agent-source"], "processors": [{"enabled": true, "id": "filter-processor", "include": "status:error", "type": "filter"}, {"enabled": true, "field": "message", "id": "json-processor", "include": "*", "type": "parse_json"}]}], "sources": [{"id": "datadog-agent-source", "type": "datadog_agent"}]}, "name": "Main Observability Pipeline"}, "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "type": "pipelines"}}
And body with value {"data": {"attributes": {"config": {"destinations": [{"id": "datadog-logs-destination", "inputs": ["filter-processor"], "type": "datadog_logs"}], "pipeline_type": "logs", "processors": [{"enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": ["datadog-agent-source"], "processors": [{"enabled": true, "id": "filter-processor", "include": "status:error", "type": "filter"}, {"enabled": true, "field": "message", "id": "json-processor", "include": "*", "type": "parse_json"}]}], "sources": [{"id": "datadog-agent-source", "type": "datadog_agent"}]}, "name": "Main Observability Pipeline"}, "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "type": "pipelines"}}
When the request is sent
Then the response status is 409 Conflict

Expand Down
14 changes: 14 additions & 0 deletions lib/datadog_api_client/inflector.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3155,6 +3155,7 @@ def overrides
"v2.observability_pipeline_aws_auth" => "ObservabilityPipelineAwsAuth",
"v2.observability_pipeline_config" => "ObservabilityPipelineConfig",
"v2.observability_pipeline_config_destination_item" => "ObservabilityPipelineConfigDestinationItem",
"v2.observability_pipeline_config_pipeline_type" => "ObservabilityPipelineConfigPipelineType",
"v2.observability_pipeline_config_processor_group" => "ObservabilityPipelineConfigProcessorGroup",
"v2.observability_pipeline_config_processor_item" => "ObservabilityPipelineConfigProcessorItem",
"v2.observability_pipeline_config_source_item" => "ObservabilityPipelineConfigSourceItem",
Expand All @@ -3172,6 +3173,8 @@ def overrides
"v2.observability_pipeline_datadog_agent_source_type" => "ObservabilityPipelineDatadogAgentSourceType",
"v2.observability_pipeline_datadog_logs_destination" => "ObservabilityPipelineDatadogLogsDestination",
"v2.observability_pipeline_datadog_logs_destination_type" => "ObservabilityPipelineDatadogLogsDestinationType",
"v2.observability_pipeline_datadog_metrics_destination" => "ObservabilityPipelineDatadogMetricsDestination",
"v2.observability_pipeline_datadog_metrics_destination_type" => "ObservabilityPipelineDatadogMetricsDestinationType",
"v2.observability_pipeline_datadog_tags_processor" => "ObservabilityPipelineDatadogTagsProcessor",
"v2.observability_pipeline_datadog_tags_processor_action" => "ObservabilityPipelineDatadogTagsProcessorAction",
"v2.observability_pipeline_datadog_tags_processor_mode" => "ObservabilityPipelineDatadogTagsProcessorMode",
Expand Down Expand Up @@ -3221,6 +3224,12 @@ def overrides
"v2.observability_pipeline_google_pub_sub_destination_type" => "ObservabilityPipelineGooglePubSubDestinationType",
"v2.observability_pipeline_google_pub_sub_source" => "ObservabilityPipelineGooglePubSubSource",
"v2.observability_pipeline_google_pub_sub_source_type" => "ObservabilityPipelineGooglePubSubSourceType",
"v2.observability_pipeline_http_client_destination" => "ObservabilityPipelineHttpClientDestination",
"v2.observability_pipeline_http_client_destination_auth_strategy" => "ObservabilityPipelineHttpClientDestinationAuthStrategy",
"v2.observability_pipeline_http_client_destination_compression" => "ObservabilityPipelineHttpClientDestinationCompression",
"v2.observability_pipeline_http_client_destination_compression_algorithm" => "ObservabilityPipelineHttpClientDestinationCompressionAlgorithm",
"v2.observability_pipeline_http_client_destination_encoding" => "ObservabilityPipelineHttpClientDestinationEncoding",
"v2.observability_pipeline_http_client_destination_type" => "ObservabilityPipelineHttpClientDestinationType",
"v2.observability_pipeline_http_client_source" => "ObservabilityPipelineHttpClientSource",
"v2.observability_pipeline_http_client_source_auth_strategy" => "ObservabilityPipelineHttpClientSourceAuthStrategy",
"v2.observability_pipeline_http_client_source_type" => "ObservabilityPipelineHttpClientSourceType",
Expand All @@ -3234,6 +3243,11 @@ def overrides
"v2.observability_pipeline_logstash_source" => "ObservabilityPipelineLogstashSource",
"v2.observability_pipeline_logstash_source_type" => "ObservabilityPipelineLogstashSourceType",
"v2.observability_pipeline_metadata_entry" => "ObservabilityPipelineMetadataEntry",
"v2.observability_pipeline_metric_tags_processor" => "ObservabilityPipelineMetricTagsProcessor",
"v2.observability_pipeline_metric_tags_processor_rule" => "ObservabilityPipelineMetricTagsProcessorRule",
"v2.observability_pipeline_metric_tags_processor_rule_action" => "ObservabilityPipelineMetricTagsProcessorRuleAction",
"v2.observability_pipeline_metric_tags_processor_rule_mode" => "ObservabilityPipelineMetricTagsProcessorRuleMode",
"v2.observability_pipeline_metric_tags_processor_type" => "ObservabilityPipelineMetricTagsProcessorType",
"v2.observability_pipeline_metric_value" => "ObservabilityPipelineMetricValue",
"v2.observability_pipeline_new_relic_destination" => "ObservabilityPipelineNewRelicDestination",
"v2.observability_pipeline_new_relic_destination_region" => "ObservabilityPipelineNewRelicDestinationRegion",
Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/azure_storage_destination.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `azure_storage` destination forwards logs to an Azure Blob Storage container.
#
# **Supported pipeline types:** logs
class AzureStorageDestination
include BaseGenericModel

Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/microsoft_sentinel_destination.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `microsoft_sentinel` destination forwards logs to Microsoft Sentinel.
#
# **Supported pipeline types:** logs
class MicrosoftSentinelDestination
include BaseGenericModel

Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/observability_pipeline_add_env_vars_processor.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `add_env_vars` processor adds environment variable values to log events.
#
# **Supported pipeline types:** logs
class ObservabilityPipelineAddEnvVarsProcessor
include BaseGenericModel

Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/observability_pipeline_add_fields_processor.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `add_fields` processor adds static key-value fields to logs.
#
# **Supported pipeline types:** logs
class ObservabilityPipelineAddFieldsProcessor
include BaseGenericModel

Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/observability_pipeline_amazon_data_firehose_source.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `amazon_data_firehose` source ingests logs from AWS Data Firehose.
#
# **Supported pipeline types:** logs
class ObservabilityPipelineAmazonDataFirehoseSource
include BaseGenericModel

Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/observability_pipeline_amazon_open_search_destination.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `amazon_opensearch` destination writes logs to Amazon OpenSearch.
#
# **Supported pipeline types:** logs
class ObservabilityPipelineAmazonOpenSearchDestination
include BaseGenericModel

Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/observability_pipeline_amazon_s3_destination.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `amazon_s3` destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
#
# **Supported pipeline types:** logs
class ObservabilityPipelineAmazonS3Destination
include BaseGenericModel

Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/observability_pipeline_amazon_s3_source.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
module DatadogAPIClient::V2
# The `amazon_s3` source ingests logs from an Amazon S3 bucket.
# It supports AWS authentication and TLS encryption.
#
# **Supported pipeline types:** logs
class ObservabilityPipelineAmazonS3Source
include BaseGenericModel

Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/observability_pipeline_amazon_security_lake_destination.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `amazon_security_lake` destination sends your logs to Amazon Security Lake.
#
# **Supported pipeline types:** logs
class ObservabilityPipelineAmazonSecurityLakeDestination
include BaseGenericModel

Expand Down
12 changes: 11 additions & 1 deletion lib/datadog_api_client/v2/models/observability_pipeline_config.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ class ObservabilityPipelineConfig
# A list of destination components where processed logs are sent.
attr_reader :destinations

# The type of data being ingested. Defaults to `logs` if not specified.
attr_accessor :pipeline_type

# A list of processor groups that transform or enrich log data.
attr_accessor :processors

Expand All @@ -37,6 +40,7 @@ class ObservabilityPipelineConfig
def self.attribute_map
{
:'destinations' => :'destinations',
:'pipeline_type' => :'pipeline_type',
:'processors' => :'processors',
:'sources' => :'sources'
}
Expand All @@ -47,6 +51,7 @@ def self.attribute_map
def self.openapi_types
{
:'destinations' => :'Array<ObservabilityPipelineConfigDestinationItem>',
:'pipeline_type' => :'ObservabilityPipelineConfigPipelineType',
:'processors' => :'Array<ObservabilityPipelineConfigProcessorGroup>',
:'sources' => :'Array<ObservabilityPipelineConfigSourceItem>'
}
Expand Down Expand Up @@ -76,6 +81,10 @@ def initialize(attributes = {})
end
end

if attributes.key?(:'pipeline_type')
self.pipeline_type = attributes[:'pipeline_type']
end

if attributes.key?(:'processors')
if (value = attributes[:'processors']).is_a?(Array)
self.processors = value
Expand Down Expand Up @@ -145,6 +154,7 @@ def ==(o)
return true if self.equal?(o)
self.class == o.class &&
destinations == o.destinations &&
pipeline_type == o.pipeline_type &&
processors == o.processors &&
sources == o.sources &&
additional_properties == o.additional_properties
Expand All @@ -154,7 +164,7 @@ def ==(o)
# @return [Integer] Hash code
# @!visibility private
def hash
[destinations, processors, sources, additional_properties].hash
[destinations, pipeline_type, processors, sources, additional_properties].hash
end
end
end
28 changes: 15 additions & 13 deletions lib/datadog_api_client/v2/models/observability_pipeline_config_destination_item.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,25 +26,27 @@ class << self
# List of class defined in oneOf (OpenAPI v3)
def openapi_one_of
[
:'ObservabilityPipelineDatadogLogsDestination',
:'ObservabilityPipelineHttpClientDestination',
:'ObservabilityPipelineAmazonOpenSearchDestination',
:'ObservabilityPipelineAmazonS3Destination',
:'ObservabilityPipelineGoogleCloudStorageDestination',
:'ObservabilityPipelineSplunkHecDestination',
:'ObservabilityPipelineSumoLogicDestination',
:'ObservabilityPipelineElasticsearchDestination',
:'ObservabilityPipelineRsyslogDestination',
:'ObservabilityPipelineSyslogNgDestination',
:'ObservabilityPipelineAmazonSecurityLakeDestination',
:'AzureStorageDestination',
:'MicrosoftSentinelDestination',
:'ObservabilityPipelineCrowdStrikeNextGenSiemDestination',
:'ObservabilityPipelineDatadogLogsDestination',
:'ObservabilityPipelineElasticsearchDestination',
:'ObservabilityPipelineGoogleChronicleDestination',
:'ObservabilityPipelineGoogleCloudStorageDestination',
:'ObservabilityPipelineGooglePubSubDestination',
:'MicrosoftSentinelDestination',
:'ObservabilityPipelineNewRelicDestination',
:'ObservabilityPipelineSentinelOneDestination',
:'ObservabilityPipelineOpenSearchDestination',
:'ObservabilityPipelineAmazonOpenSearchDestination',
:'ObservabilityPipelineRsyslogDestination',
:'ObservabilityPipelineSentinelOneDestination',
:'ObservabilityPipelineSocketDestination',
:'ObservabilityPipelineAmazonSecurityLakeDestination',
:'ObservabilityPipelineCrowdStrikeNextGenSiemDestination',
:'ObservabilityPipelineGooglePubSubDestination'
:'ObservabilityPipelineSplunkHecDestination',
:'ObservabilityPipelineSumoLogicDestination',
:'ObservabilityPipelineSyslogNgDestination',
:'ObservabilityPipelineDatadogMetricsDestination'
]
end
# Builds the object
Expand Down
27 changes: 27 additions & 0 deletions lib/datadog_api_client/v2/models/observability_pipeline_config_pipeline_type.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
=begin
#Datadog API V2 Collection

#Collection of all Datadog Public endpoints.

The version of the OpenAPI document: 1.0
Contact: support@datadoghq.com
Generated by: https://github.com/DataDog/datadog-api-client-ruby/tree/master/.generator

Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License.
This product includes software developed at Datadog (https://www.datadoghq.com/).
Copyright 2020-Present Datadog, Inc.

=end

require 'date'
require 'time'

module DatadogAPIClient::V2
# The type of data being ingested. Defaults to `logs` if not specified.
class ObservabilityPipelineConfigPipelineType
include BaseEnumModel

LOGS = "logs".freeze
METRICS = "metrics".freeze
end
end
21 changes: 11 additions & 10 deletions lib/datadog_api_client/v2/models/observability_pipeline_config_processor_item.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,23 +27,24 @@ class << self
def openapi_one_of
[
:'ObservabilityPipelineFilterProcessor',
:'ObservabilityPipelineAddEnvVarsProcessor',
:'ObservabilityPipelineAddFieldsProcessor',
:'ObservabilityPipelineCustomProcessor',
:'ObservabilityPipelineDatadogTagsProcessor',
:'ObservabilityPipelineDedupeProcessor',
:'ObservabilityPipelineEnrichmentTableProcessor',
:'ObservabilityPipelineGenerateMetricsProcessor',
:'ObservabilityPipelineOcsfMapperProcessor',
:'ObservabilityPipelineParseGrokProcessor',
:'ObservabilityPipelineParseJSONProcessor',
:'ObservabilityPipelineQuotaProcessor',
:'ObservabilityPipelineAddFieldsProcessor',
:'ObservabilityPipelineReduceProcessor',
:'ObservabilityPipelineRemoveFieldsProcessor',
:'ObservabilityPipelineRenameFieldsProcessor',
:'ObservabilityPipelineGenerateMetricsProcessor',
:'ObservabilityPipelineSampleProcessor',
:'ObservabilityPipelineParseGrokProcessor',
:'ObservabilityPipelineSensitiveDataScannerProcessor',
:'ObservabilityPipelineOcsfMapperProcessor',
:'ObservabilityPipelineAddEnvVarsProcessor',
:'ObservabilityPipelineDedupeProcessor',
:'ObservabilityPipelineEnrichmentTableProcessor',
:'ObservabilityPipelineReduceProcessor',
:'ObservabilityPipelineThrottleProcessor',
:'ObservabilityPipelineCustomProcessor',
:'ObservabilityPipelineDatadogTagsProcessor'
:'ObservabilityPipelineMetricTagsProcessor'
]
end
# Builds the object
Expand Down
20 changes: 10 additions & 10 deletions lib/datadog_api_client/v2/models/observability_pipeline_config_source_item.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,22 +26,22 @@ class << self
# List of class defined in oneOf (OpenAPI v3)
def openapi_one_of
[
:'ObservabilityPipelineKafkaSource',
:'ObservabilityPipelineDatadogAgentSource',
:'ObservabilityPipelineSplunkTcpSource',
:'ObservabilityPipelineSplunkHecSource',
:'ObservabilityPipelineAmazonDataFirehoseSource',
:'ObservabilityPipelineAmazonS3Source',
:'ObservabilityPipelineFluentdSource',
:'ObservabilityPipelineFluentBitSource',
:'ObservabilityPipelineHttpServerSource',
:'ObservabilityPipelineSumoLogicSource',
:'ObservabilityPipelineRsyslogSource',
:'ObservabilityPipelineSyslogNgSource',
:'ObservabilityPipelineAmazonDataFirehoseSource',
:'ObservabilityPipelineFluentdSource',
:'ObservabilityPipelineGooglePubSubSource',
:'ObservabilityPipelineHttpClientSource',
:'ObservabilityPipelineHttpServerSource',
:'ObservabilityPipelineKafkaSource',
:'ObservabilityPipelineLogstashSource',
:'ObservabilityPipelineSocketSource'
:'ObservabilityPipelineRsyslogSource',
:'ObservabilityPipelineSocketSource',
:'ObservabilityPipelineSplunkHecSource',
:'ObservabilityPipelineSplunkTcpSource',
:'ObservabilityPipelineSumoLogicSource',
:'ObservabilityPipelineSyslogNgSource'
]
end
# Builds the object
Expand Down
2 changes: 2 additions & 0 deletions ...dog_api_client/v2/models/observability_pipeline_crowd_strike_next_gen_siem_destination.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `crowdstrike_next_gen_siem` destination forwards logs to CrowdStrike Next Gen SIEM.
#
# **Supported pipeline types:** logs
class ObservabilityPipelineCrowdStrikeNextGenSiemDestination
include BaseGenericModel

Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/observability_pipeline_custom_processor.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities.
#
# **Supported pipeline types:** logs
class ObservabilityPipelineCustomProcessor
include BaseGenericModel

Expand Down
4 changes: 3 additions & 1 deletion lib/datadog_api_client/v2/models/observability_pipeline_datadog_agent_source.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@
require 'time'

module DatadogAPIClient::V2
# The `datadog_agent` source collects logs from the Datadog Agent.
# The `datadog_agent` source collects logs/metrics from the Datadog Agent.
#
# **Supported pipeline types:** logs, metrics
class ObservabilityPipelineDatadogAgentSource
include BaseGenericModel

Expand Down
2 changes: 2 additions & 0 deletions lib/datadog_api_client/v2/models/observability_pipeline_datadog_logs_destination.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

module DatadogAPIClient::V2
# The `datadog_logs` destination forwards logs to Datadog Log Management.
#
# **Supported pipeline types:** logs
class ObservabilityPipelineDatadogLogsDestination
include BaseGenericModel

Expand Down
Loading
Loading