Live-check: allow filtering findings when emitting logs over otlp

[lmolkova]

Most of these findings are known, expected or something app developer has no control over (e.g. exception.escapted attribute is set by otel python for back-compat reasons).

I thought advice-preprocessor could help me, but it seems it only massages the data given to a policy and it does not seem I can use it to drop/filter things.

What would be great is some config (similar to linting suppressions) like

min_finding_level: WARN
suppress_findings:
  - ids: ["not_stable"]
  # some future advanced capabilities to suppress more granularly, but it might not be worth it

It can also be cmd line params (min finding level, finding ids to suppress) for the time being.

[lmolkova]

a workaround using collector is to send weaver logs back to collector, filter them in a separate pipeline and then then to the final destination. It creates circular dependency (weaver sends to collector, collector sends to weaver) and makes some data loss very likely (unless someone's willing to spin up another collector just as a weaver side-car).

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  otlp/weaver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4318

processors:
  filter/weaver-findings:
    logs:
      log_record:
        - 'severity_number < SEVERITY_NUMBER_WARN'
        - 'attributes["weaver.finding.id"] == "not_stable"'
        - 'attributes["weaver.finding.context.attribute_name"] == "exception.escaped"'

exporters:
  otlp:
    endpoint: lgtm:4317
    tls:
      insecure: true
  otlp/weaver:
    endpoint: weaver:4317
    compression: none # TODO: support gzip compression in weaver
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp, otlp/weaver]
    metrics:
      receivers: [otlp]
      exporters: [otlp, otlp/weaver]
    logs:
      receivers: [otlp]
      exporters: [otlp, otlp/weaver]
    logs/weaver-findings:
      receivers: [otlp/weaver]
      processors: [filter/weaver-findings]
      exporters: [otlp]

[jerbly]

More deeply controlling the configuration of live-check was on my todo list from the beginning. I will gladly work on this knowing that there's more interest in refining the output.