[receiver/kafka] Allow Kafka Receiver to accept Azure Resource Log format

[cparkins]

Component(s)

receiver/kafka

Is your feature request related to a problem? Please describe.

When trying to ingest Infrastructure logs and metrics from Azure Resources through Diagnostic Settings using the Kafka Receiver the format used by Azure is not in an accepted format.

Describe the solution you'd like

Allow the Kafka Receiver to ingest data from Azure Resource Logs and convert it into OpenTelemetry format.

Describe alternatives you've considered

Using the Azure Event Hub Receiver or Azure Blob Storage Receiver.

Additional context

No response

[atoulme]

@MovieStoreGuy we can make our codecs shared across kafka and azureeventhub so users can reuse the logic. Azure event hub looks very similar to kafka, to the point it can be exposed as a kafka endpoint apparently.

[atoulme]

This is the formatter @cparkins references fwiw: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/azureeventhubreceiver/azure_formatter_impl.go

[github-actions]

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for receiver/kafka: @pavolloffay @MovieStoreGuy. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[MovieStoreGuy]

@MovieStoreGuy we can make our codecs shared across kafka and azureeventhub so users can reuse the logic. Azure event hub looks very similar to kafka, to the point it can be exposed as a kafka endpoint apparently.

I had a proposal some time ago about having a an encoding package that provides a factory that can be passed around since other components like file receiver | exporter, kinesis exporter, kafka receiver | exporter, and even potentially the HTTP client could adopt.

I don't have an issue for it in principle, since we don't have this shared encoding package and we keep the kafka code generic enough that we can add other formats.

[martinjt]

I'm about to pickup expanding out the EventHub receiver to allow Diagnostic (audit etc.) logs and metrics to be taken by the eventhub receiver. This was likely to be quite hacky in the eventhub receiver. I was originally going to expand the kafka receiver to use the right authentication, and add the formats into that, but my feeling is that the eventhubs might be the better place?

My feeling is that with the eventhub receiver, we can detect what format the the event is and choose the right data type and transformation as it's more specific to event hubs?

I need this for a client, however, I'm more than happy for it to be work directly against the contrib repo. I'm no go-programmer, so it will likely need more of a thorough review.

[atoulme]

Nothing is hacky. Everything is a work in progress. @cparkins just landed metric support in azureeventhubreceiver, please take a look and feel free to open issues and PRs to make things better.