Init ignore_firelens_config option

[PettitWesley]

Background

Issue #, if available:

Use case and design discussion here: aws/containers-roadmap#964 (comment)

Description of changes:

See use_cases section update for a description of the feature.

Testing

Task Def

{
    "family": "init-ignore-firelens",
    "taskRoleArn": "arn:aws:iam::XXXXXXXXXX:role/ecs_task_role",
    "executionRoleArn": "arn:aws:iam::XXXXXXXXXX:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "memory": "1024",
    "cpu": "512",
    "containerDefinitions": [
        {
            "essential": true,
            "image": "XXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:init-ignore-firelens-2",
            "name": "log_router",
            "firelensConfiguration": {
                "type": "fluentbit"
            },
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "firelens-container",
                    "awslogs-region": "us-west-2",
                    "awslogs-create-group": "true",
                    "awslogs-stream-prefix": "init-ignore-firelens"
                }
            },
            "environment": [
                {
                    "name": "aws_fluent_bit_init_s3_1",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/dummy-filter.conf"
                },
                {
                    "name": "aws_fluent_bit_init_s3_2",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/dummy-input.conf"
                },
                {
                    "name": "aws_fluent_bit_init_s3_3",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/dummy-parser.conf"
                },
                {
                    "name": "aws_fluent_bit_init_s3_4",
                    "value": "arn:aws:s3:::firenosed-dolphin-bucket/stdout.conf"
                },
                {
                    "name": "aws_fluent_bit_init_ignore_firelens_config",
                    "value": "true"
                },
                {
                    "name": "FLB_LOG_LEVEL",
                    "value": "debug"
                }
            ],
            "memoryReservation": 50
        },
        {
            "essential": true,
            "image": "httpd",
            "name": "app",
            "logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "Name": "cloudwatch_logs",
                    "region": "${AWS_REGION}",
                    "log_group_name": "${ECS_CLUSTER}/application",
                    "auto_create_group": "true",
                    "log_stream_name": "${ECS_TASK_ID}",
                    "retry_limit": "2"
                }
            },
            "memoryReservation": 100
        }
    ]
}

Case 1: ignore set to true

Result from exec'ing into the task and checking the config:

sh-4.2# ls
dummy-filter.conf  dummy-input.conf  dummy-parser.conf	stdout.conf
sh-4.2# ls
dummy-filter.conf  dummy-input.conf  dummy-parser.conf	stdout.conf
sh-4.2# cd ..
sh-4.2# ls
fluent-bit-init-s3-files  fluent-bit-init.conf	fluent_bit_init_entrypoint.sh  fluent_bit_init_process	invoke_fluent_bit.sh
sh-4.2# cat fluent-bit-init.conf
@INCLUDE /init/fluent-bit-init-s3-files/stdout.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-input.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-filter.conf
sh-4.2# cat fluent_bit_init_entrypoint.sh
./init/fluent_bit_init_process
source /init/invoke_fluent_bit.sh
sh-4.2# cat invoke_fluent_bit.sh
export FLB_AWS_USER_AGENT=ecs-init
export AWS_REGION=us-west-2
export AWS_AVAILABILITY_ZONE=us-west-2a
export ECS_CLUSTER=firelens-private-subnets
export ECS_TASK_ARN=arn:aws:ecs:us-west-2:144718711470:task/firelens-private-subnets/002dd1eaa1544ac79cde672daedb4a83
export ECS_TASK_ID=002dd1eaa1544ac79cde672daedb4a83
export ECS_FAMILY=init-ignore-firelens
export ECS_LAUNCH_TYPE=FARGATE
export ECS_REVISION=3
export ECS_TASK_DEFINITION=init-ignore-firelens:3
exec /fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so -c /init/fluent-bit-init.conf -R /init/fluent-bit-init-s3-files/dummy-parser.conf

Note there is no include for the default FireLens Fluent Bit config path.

Case 2: Ignore set to false

Starting session with SessionId: ecs-execute-command-02fe7250f544ea00e
sh-4.2# ls
AWS_FOR_FLUENT_BIT_VERSION  bin   dev  entrypoint.sh  fluent-bit  init	lib64  managed-agents  mnt  proc  run	srv  tmp  var
Dockerfile.1.main-release   boot  ecs  etc	      home	  lib	local  media	       opt  root  sbin	sys  usr
sh-4.2# cd init
sh-4.2# ls
fluent-bit-init-s3-files  fluent-bit-init.conf	fluent_bit_init_entrypoint.sh  fluent_bit_init_process	invoke_fluent_bit.sh
sh-4.2# cat fluent-bit-init.conf
@INCLUDE /fluent-bit/etc/fluent-bit.conf
@INCLUDE /init/fluent-bit-init-s3-files/stdout.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-input.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-filter.conf
sh-4.2# cat fluent_bit_init_entrypoint.sh
./init/fluent_bit_init_process
source /init/invoke_fluent_bit.sh
sh-4.2#

Note the @INCLUDE /fluent-bit/etc/fluent-bit.conf.

Case 3: Ignore not defined (regression check)

sh-4.2# ls
dummy-filter.conf  dummy-input.conf  dummy-parser.conf	stdout.conf
sh-4.2# cd ..
sh-4.2# ls
fluent-bit-init-s3-files  fluent-bit-init.conf	fluent_bit_init_entrypoint.sh  fluent_bit_init_process	invoke_fluent_bit.sh
sh-4.2# cat fluent-bit-init.conf
@INCLUDE /fluent-bit/etc/fluent-bit.conf
@INCLUDE /init/fluent-bit-init-s3-files/stdout.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-input.conf
@INCLUDE /init/fluent-bit-init-s3-files/dummy-filter.conf

Normal behavior.

Unit tests

go test -v passes in the init directory.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[PettitWesley]

FireLens example: aws-samples/amazon-ecs-firelens-examples#127

[matthewfala]

It may be helpful to add an example with the appropriate match statement for docker logs.

    Match               Application-firelens-*

[matthewfala]

Looks good! One comment.