[receivercreator/discovery] Add support for default annotation-equivalent config in receivercreator

[ChrsMark]

Description

This PR adds support for defining default annotation-equivalent configuration. This can be useful when a default behaviour is required for all discovered Pods/containers without having to explicitly annotate all Pods.

Example: Enable log collection for all discovered Pods/containers unless they are explicitly annotated with io.opentelemetry.discovery.logs/enabled: "false". (to avoid collecting Collector's own logs make sure that Collector Pods are properly annotated with io.opentelemetry.discovery.logs/enabled: "false")

Link to tracking issue

Fixes #37436

Testing

Added

Documentation

Added

How to test this manually

1. Deploy Collector helm chart:

mode: daemonset

podAnnotations:
  io.opentelemetry.discovery.logs/enabled: "false"

image:
  repository:otelcontribcol-dev
  tag: "latest"
  pullPolicy: IfNotPresent

clusterRole:
  create: true
  rules:
    - apiGroups:
        - ''
      resources:
        - 'pods'
        - 'nodes'
      verbs:
        - 'get'
        - 'list'
        - 'watch'
    - apiGroups: [ "" ]
      resources: [ "nodes/proxy"]
      verbs: [ "get" ]
    - apiGroups:
        - ""
      resources:
        - nodes/stats
      verbs:
        - get
    - nonResourceURLs:
        - "/metrics"
      verbs:
        - get

extraVolumeMounts:
  - name: varlogpods
    mountPath: /var/log/pods
    readOnly: true

extraVolumes:
  - name: varlogpods
    hostPath:
      path: /var/log/pods

config:
  extensions:
    k8s_observer:
      auth_type: serviceAccount
      node: ${env:K8S_NODE_NAME}
    health_check:
      endpoint: ${env:MY_POD_IP}:13133
  exporters:
    debug/2:
      verbosity: basic
    debug:
      verbosity: detailed

  receivers:

    receiver_creator/logs:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
        default_annotations:
          io.opentelemetry.discovery.logs/enabled: "true"
      receivers:


  service:
    extensions: [health_check, k8s_observer]
    telemetry:
      logs:
        level: INFO
    pipelines:
      logs/discovery:
        receivers: [ receiver_creator/logs ]
        #processors: [ batch ]
        exporters: [ debug ]

2. Deploy a target Pod and ensure that logs are collected by default:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-deployment
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
      annotations:
        some: foo
    spec:
      containers:
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs at $(date +%H:%M:%S)" && sleep 2s; done

3. After ensuring that logs are collected by default annotate the Pod properly to ensure that log collection is explicitly disabled: kubectl annotate pod busybox-deployment-77b9b7cfbd-5lfmh io.opentelemetry.discovery.logs/enabled="false"

[ChrsMark]

@dmitryax please take a look when you get the chance.

[dmitryax]

LGTM. The comment isn't blocking 👍

[dmitryax]

I like how simple the interface is 👍

[andrzej-stencel]

Thanks for mentioning this in the documentation 👍