Add section on how to build an otelcol dist #1234

jpkrohling · 2020-10-09T14:04:19Z

Signed-off-by: Juraci Paixão Kröhling [email protected]

Description: Added readme section with links to resources on how to create custom OpenTelemetry Collector distributions.

Link to tracking Issue: open-telemetry/opentelemetry-collector#213

Testing: n/a

Documentation: n/a

README.md

Signed-off-by: Juraci Paixão Kröhling <[email protected]>

owais · 2020-10-12T16:42:18Z

README.md

+## Creating your own OpenTelemetry Collector distribution
+
+A guide on how to create your own distribution is available in this blog post: 
+["Building your own OpenTelemetry Collector distribution"](https://medium.com/p/42337e994b63)


I may be wrong but isn't medium content sometimes locked behind a paywall now? Not sure if a freely available project both as in beer and speech should rely on extended documentation that might not be accessible to everyone.

Nothing against medium or content they host. Just not sure if OpenTelemetry as a product should direct users to content/documentation/guides that they might not be able to read.

Great article BTW @jpkrohling

Oh, I see this is the official OpenTelemetry blog on medium. I suppose this decision has already been taken then by the maintainers and we are fine with publishin there? Please disregard if that's the case.

sometimes locked behind a paywall now

Each author/publication can choose to make money on Medium and have a paywall in front of individual posts, or to allow all users to have free access to the content, without any paywalls. I always publish articles without opting-in for the paywall, and I believe the OpenTelemetry publication also doesn't.

README.md

Updates #1234 Signed-off-by: Bogdan Drutu <[email protected]>

… add 'messaging.url' span attribute (#1234) * boto3sqs: Fix various issues * do not use 'otel' prefix for propagation keys to make propagation compatible with other SQS instrumentations like Node.Js Inject propergator.fields keys into the MessageAttributeNames argument for 'receive_message' calls to retreive the corresponding message attributes * add 'messaging.url' span attribute to SQS spans * add boto3sqs instrumentation to tox.ini to run tests in CI * add some basic unit tests * changelog * fix linting issues * unset instrumented flag on uninstrument

#### Description Removes `cmd/oteltestbedcol` from list since it has no Makefile  #### Link to tracking issue Fixes #35606

)  #### Description Removes `cmd/oteltestbedcol` from list since it has no Makefile  #### Link to tracking issue Fixes open-telemetry#35606

…atus (#35648)  #### Description Refactors the `extension/healthcheckv2extension/internal/status` into `pkg/status`. This exposes the aggregator to be used by other extensions to gather component status information using the `extension.StatusWatcher`.  #### Link to tracking issue Closes #34692  #### Testing Being it was a refactor and all the same tests provided coverage, no additional testing was added.  #### Documentation Added a `README.md` to the `pkg/status` to provide information on where this package can be used. --------- Co-authored-by: Matthew Wear <[email protected]>

#### Description Update OTel mode to implementation to serialize log body into body.* fields  #### Link to tracking issue  #### Testing  #### Documentation

…rning error to avoid upstream retries (#35740)  #### Description Log metrics validation error instead of returning to avoid upstream retries  #### Link to tracking issue  #### Testing  #### Documentation

#### Description  #### Link to tracking issue #35741 Fixes Wrong comparison func implement in translator/prometheusremotewrite.  #### Testing  #### Documentation  Signed-off-by: Juraj Michalek <[email protected]>

#### Description Follow up from #35703. Draft starting the work on adding support for remote write 2.0 in the translation package. Adding support for translating gauges. This is first iteration and to keep the PR small * we don't handle duplicate metrics * only support gauges * don't handle other labels than metric name * don't handle exemplars * don't handle metadata  #### Link to tracking issue #33661 Fixes  #### Testing  #### Documentation  --------- Signed-off-by: Juraj Michalek <[email protected]> Co-authored-by: Arthur Silva Sens <[email protected]> Co-authored-by: David Ashpole <[email protected]>

#### Description Follow [the contract](https://github.com/open-telemetry/opentelemetry-collector/blob/df3c9e38a80ccc3b14705462be2e2e51c628a3b3/receiver/doc.go#L10) for cloudflare receiver  #### Link to tracking issue Related #5909  #### Testing Added

#### Description Add a new receiver to handle systemd service state reporting as metrics.  #### Link to tracking issue #33532

… prefix conflict in OTel mapping mode (#35651)  #### Description Metric names should be flattened and exported as is, even when one metric name is a prefix of another. Same for attributes for all logs, metrics and traces.  #### Link to tracking issue  #### Testing  #### Documentation

#### Description Version of MySQL older than 8.0.22 use the `show slave status` vs `show replica status` view to retrieve replication metrics. This allows for both the newer & older views to work and retrieve data for the two actual replica metrics available (`mysql.replica.time_behind_source` & `mysql.replica.sql_delay`).  #### Link to tracking issue Fixes #35217  #### Testing Manual testing against MySQL versions 8.0, 8.0.37, & MariaDB Ver 15.1 Distrib 10.5.26-MariaDB  #### Documentation None needed

#### Description Adds initial implementation of ntpreceiver.  #### Link to tracking issue #34375

…only (#35839)  #### Description Make OTel mapping mode use RequireDataStream in docappender, which means it will only send to data streams. This prevents auto creating regular indices in OTel mapping mode due to a race condition in Elasticsearch where otel-data index templates are not ready.  #### Link to tracking issue  #### Testing  #### Documentation

#### Description Enable gzip compression by default, at hardcoded level BestSpeed. To disable compression, set `compression` to `none`.  #### Link to tracking issue  #### Testing  #### Documentation

#### Description Fixes an issue where the Kafka receiver would block on shutdown. There was an earlier fix for this issue [here](#32720). This does solve the issue, but it was only applied to the traces receiver, not the logs or metrics receiver. The issue is this go routine in the `Start()` functions for logs and metrics: ```go go func() { if err := c.consumeLoop(ctx, metricsConsumerGroup); err != nil { componentstatus.ReportStatus(host, componentstatus.NewFatalErrorEvent(err)) } }() ``` The `consumeLoop()` function returns a `context.Canceled` error when `Shutdown()` is called, which is expected. However `componentstatus.ReportStatus()` blocks while attempting to report this error. The reason/bug for this can be found [here](open-telemetry/opentelemetry-collector#9824). The previously mentioned PR fixed this for the traces receiver by checking if the error returned by `consumeLoop()` is `context.Canceled`: ```go go func() { if err := c.consumeLoop(ctx, consumerGroup); !errors.Is(err, context.Canceled) { componentstatus.ReportStatus(host, componentstatus.NewFatalErrorEvent(err)) } }() ``` Additionally, this is `consumeLoop()` for the traces receiver, with the logs and metrics versions being identical: ```go func (c *kafkaTracesConsumer) consumeLoop(ctx context.Context, handler sarama.ConsumerGroupHandler) error { for { // `Consume` should be called inside an infinite loop, when a // server-side rebalance happens, the consumer session will need to be // recreated to get the new claims if err := c.consumerGroup.Consume(ctx, c.topics, handler); err != nil { c.settings.Logger.Error("Error from consumer", zap.Error(err)) } // check if context was cancelled, signaling that the consumer should stop if ctx.Err() != nil { c.settings.Logger.Info("Consumer stopped", zap.Error(ctx.Err())) return ctx.Err() } } } ``` This does fix the issue, however the only error that can be returned by `consumeLoop()` is a canceled context. When we create the context and cancel function, we use `context.Background()`: ```go ctx, cancel := context.WithCancel(context.Background()) ``` This context is only used by `consumeLoop()` and the cancel function is only called in `Shutdown()`. Because `consumeLoop()` can only return a `context.Canceled` error, this PR removes this unused code for the logs, metrics, and traces receivers. Instead, `consumeLoop()` still logs the `context.Canceled` error but it does not return any error and the go routine simply just calls `consumeLoop()`. Additional motivation for removing the call to `componentstatus.ReportStatus()` is the underlying function called by it, `componentstatus.Report()` says it does not need to be called during `Shutdown()` or `Start()` as the service already does so for the given component, [comment here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/component/componentstatus/status.go#L21-L25). Even if there wasn't a bug causing this call to block, the component still shouldn't call it since it would only be called during `Shutdown()`.  #### Link to tracking issue Fixes #30789  #### Testing Tested in a build of the collector with these changes scraping logs from a Kafka instance. When the collector is stopped and `Shutdown()` gets called, the receiver did not block and the collector stopped gracefully as expected.

#### Description Use mdatagen to define and generate events functions in PostgreSQL receiver.  #### Link to tracking issue n/a  #### Testing Passed existing tests  #### Documentation Added  --------- Co-authored-by: Will Li <[email protected]>

@dmitryax

#### Description Annotation discovery should not add the default endpoint explicitly. Configuration should be left empty if users do not set anything. Main flow of the receiver_creator already adds the default endpoint, if not provided, at a later stage by also checking if target receiver supports it. This patch ensures this logic is unified for both annotation discovery and templated based discovery. Main flow of the receiver creator handles default endpoint at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/1ff442c73096b3770070d0fd8868d42467245134/receiver/receivercreator/observerhandler.go#L158-L163  #### Link to tracking issue Fixes #40712  #### Testing Fixed  #### Documentation Not needed /cc @dmitryax --------- Signed-off-by: ChrsMark <[email protected]> Co-authored-by: Dmitry Anoshin <[email protected]>

#### Description  #### Link to tracking issue Fixes #40680  #### Testing  #### Documentation

… with NewClientset (#40798)  #### Description replace deprecated NewSimpleClientset with NewClientset in test  #### Link to tracking issue Fixes  #### Testing  #### Documentation  Signed-off-by: Jared Tan <[email protected]>

#40799)  #### Description replace deprecated NewSimpleClientset with NewClientset in test  #### Link to tracking issue Fixes  #### Testing  #### Documentation  Signed-off-by: Jared Tan <[email protected]>

…t with NewClientset (#40796)  #### Description replace deprecated NewSimpleClientset with NewClientset in test  #### Link to tracking issue  #### Testing  #### Documentation  Signed-off-by: Jared Tan <[email protected]>

#### Description When setting pod_association to container.id it can not match at the moment, as it will look up pod.Attributes and fail to match. Update getIdentifiersFromAssoc() to generate PodIdentifier also from pod.Containers.ByID. This then also covers the case, where a single k8s.pod.UID covers multiple container.ids.  #### Link to tracking issue Fixes #40745  #### Testing  #### Documentation  --------- Signed-off-by: Florian Lehner <[email protected]>

…shaler (open-telemetry#40762)  #### Description  #### Link to tracking issue Fixes open-telemetry#40661  #### Testing  #### Documentation

…waiting on it (#40849) #### Description The `run()` goroutine closes the channel in defer. There are two sceanrios: 1. Best case scenario: no error encountered while calling `ConsumeMetrics`. 2. Worst case scenario: we encounter an error while calling `ConsumeMetrics`, causing the goroutine to exit early and as a result, close the `c.exit` chan. We close the channel, again, in `Shutdown` and this results in a "send on closed channel" panic in worst case scenario. In best case scenario, no panic is encountered. This PR adds waitgroup to avoid the panic.  #### Link to tracking issue Fixes #40845  #### Testing Added a test case

#### Support for Dynamic Log Indexing in OpenSearch This PR supports dynamic log index names using placeholders in the `logs_index` config. You can use any attribute or context key as a placeholder to construct index names dynamically per log record. #### `logs_index` - Placeholder: `%{key}` - Example: `otel-logs-%{service.name}` or `otel-logs-%{custom.label}` - The value is looked up from a context map (resource attributes, log attributes, etc.). - If the key is missing, the value from `logs_index_fallback` is used (or `unknown` if not set). - If `logs_index` is not set, the exporter will use the default naming pattern: `ss4o_{type}-{dataset}-{namespace}` (e.g., `ss4o_logs-default-namespace`). This ensures backward compatibility and a predictable index naming scheme. #### `logs_index_time_format` You can append a time-formatted suffix to the index name using the `logs_index_time_format` option. - `logs_index_time_format`: If set, appends a time suffix to the resolved index name using the specified format (default is no suffix). - Valid tokens (case-sensitive): - `yyyy` (4-digit year) - `yy` (2-digit year) - `MM` (2-digit month) - `dd` (2-digit day) - `HH` (2-digit hour, 24h) - `mm` (2-digit minute) - `ss` (2-digit second) - Allowed separators: `-`, `.`, `_`, `+` - Examples: - `yyyy.MM.dd` → `2024.06.07` - `yyyy-MM` → `2024-06` - `yyMMdd` → `240607` - `yyyy_MM_dd+HH` → `2024_06_07+15` - Any other characters or tokens will result in a configuration error.  #### Link to tracking issue #34746 #38595  #### Testing Unit tests for parsing and rendering `logs_index`, `logs_index_fallback`, and `logs_index_time_format`. Integration tests against a local OpenSearch instance to verify index names (including fallbacks and timestamp formatting).  #### Documentation Updated `exporter/opensearchexporter/README.md` ##### Example Configuration ```yaml exporters: opensearch: http: endpoint: http://opensearch.example.com:9200 logs_index: "otel-logs-%{service.name}" logs_index_fallback: "default-service" # optional, if not set default is `unknown` logs_index_time_format: "yyyy.MM.dd" # optional, if set appends time suffix ``` --------- Signed-off-by: Shenoy Pratik <[email protected]> Signed-off-by: Paul Sebastian <[email protected]> Co-authored-by: Andrzej Stencel <[email protected]> Co-authored-by: Paul Sebastian <[email protected]>

…amInfo (#40802)  #### Description remove unused EnvironmentInfo,AuthInfo,TeamInfo  #### Link to tracking issue Fixes #40669  #### Testing  #### Documentation  --------- Signed-off-by: Jared Tan <[email protected]>

…api which pulls in packages with known CVEs. (#40864)  #### Description There's a known vulnerability in `go-restful v2.9.5` which has CVE 2022-1996 - [BlackDuck OpenHub](https://openhub.net/p/go_restful/security?filter%5Bmajor_version%5D=2&filter%5Bperiod%5D=10&filter%5Bversion%5D=5682384252&filter%5Bseverity%5D=&filter%5Btype%5D=) - [CVE 2022-1996](https://nvd.nist.gov/vuln/detail/CVE-2022-1996) The fix for this is available from `go-restful v3.8.0` and later. The following two packages currently invoke `go-restful v2.9.5` via `internal/k8sconfig`: - openshift/client-go - openshift/api The [primary patch](e7f022d) updates these two dependencies in `internal/k8sconfig/go.mod`. The remaining files are dependent recursive module updates.  #### Link to tracking issue Fixes  #### Testing - `make gotidy` - `make otelcontribcol` - `make gotest`  #### Documentation

#### Description The default logs agent compression kind was [recently changed](DataDog/datadog-agent@db09bbc#diff-a7b6619528f155e64cefe12cc4ba753eb7688c87fdf9d96e1052a12eeb519df5L85) to zstd in agent v7.67.0. With the version upgrade in #40888, we want to change this back to gzip in contrib.  #### Link to tracking issue Fixes  #### Testing  #### Documentation

#### Description This PR adds the following internal telemetry metrics to improve the receiver telemetry to make it easier to operate. <details> <summary>Click to expand</summary> | Name | Description | Type | | -------- | ------- | ------- | | kafka_receiver_bytes | The number of received bytes, accounting for client compression | sum | | kafka_receiver_bytes_uncompressed | The number of received uncompressed bytes | sum | | kafka_receiver_latency | Time it took to receive a batch of records | histogram | | kafka_broker_connects | The number of connections opened | sum | | kafka_broker_disconnects | The number of connections closed | sum | | kafka_broker_throttling_duration | The throttling duration imposed by the broker when receiving records | histogram |histogram | </details>  #### Link to tracking issue Fixes #40816  #### Testing Unit tests are updated and tested manually.  #### Documentation Documentation is autogenerated with mdatagen.

#### Description This PR adds the following internal telemetry metrics to improve the exporter telemetry to make it easier to operate. <details> <summary>Click to expand</summary> | Name | Description | Type | | -------- | ------- | ------- | | kafka_exporter_records | The number of exported records | sum | | kafka_exporter_bytes | The number of exported bytes, accounting for client compression | sum | | kafka_exporter_bytes_uncompressed | The number of exported uncompressed bytes | sum | | kafka_exporter_latency | Time it took to export a batch of records | histogram | | kafka_broker_connects | The number of connections opened | sum | | kafka_broker_disconnects | The number of connections closed | sum | | kafka_broker_throttling_duration | The throttling duration imposed by the broker when exporting messages | histogram | </details>  #### Link to tracking issue Fixes #40815  #### Testing Unit tests are added, also tested manually.  #### Documentation Documentation is autogenerated with mdatagen.

#### Description Azure auth was only enabled when batch api was set to false. This issue fixes that.  #### Link to tracking issue Fixes #40872.  #### Testing Unit test was added.  #### Documentation README updated with a note on versions.  --------- Co-authored-by: Braydon Kains <[email protected]>

#### Description Adds support for adding histograms in the RW2 translation path.  #### Link to tracking issue Fixes Partially implements #33661 (when merging PR please don't close the tracing issue)  #### Testing * [x] Added unit test * [x] e2e ran with prometheus Histogram metrics in prometheus ui ![image](https://github.com/user-attachments/assets/6da34adf-b17e-4dc5-a132-5423d9df835e)

#### Description This PR adds two new histogram metrics to track the latency of Write-Ahead Log (WAL) operations in the Prometheus Remote Write Exporter: - `otelcol_exporter_prometheusremotewrite_wal_write_latency`: Tracks latency for WAL write operations - `otelcol_exporter_prometheusremotewrite_wal_read_latency`: Tracks latency for WAL read operations  #### Link to tracking issue Part of #39556  #### Testing Added UT  #### Documentation  --------- Signed-off-by: Nikos Angelopoulos <[email protected]>

#### Description The `Endpoint` struct is not part of the config, and therefore, should not be public. This PR just makes it private.  #### Link to tracking issue Closes #40663

…der (#40838)  #### Description The gzip reader was returned to the pool before we finished using it, which could cause multiple go routines using the same reader.  #### Link to tracking issue N/A.  #### Testing There was one unit test added to check this isn't happening anymore.  #### Documentation N/A

…gned groups (#40921)  #### Description The `OnPartitionLost` function can be called without the group ever being assigned ([ref](https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#OnPartitionsLost)). However, the kafka receiver code didn't protect against this. The PR adds a check to protect against this case. Quoting from the linked docs for `OnPartitionLost`: ``` OnPartitionsLost sets the function to be called on "fatal" group errors, such as IllegalGeneration, UnknownMemberID, and authentication failures. This function differs from [OnPartitionsRevoked](https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#OnPartitionsRevoked) in that it is unlikely that commits will succeed when partitions are outright lost, whereas commits likely will succeed when revoking partitions. Because this function is called on any fatal group error, it is possible for this function to be called without the group ever being joined. ```  #### Link to tracking issue Fixes #40920  #### Testing Since the test is a callback, I can't think of a good way to test it other than coding the exact scenario which I think would be too brittle.  #### Documentation Added godoc comments for clarity.

…40925)  #### Description This PR graduates the exporter.datadogexporter.UseLogsAgentExporter feature gate to stable. Logs agent exporter is now always used and logs exporter implementation was removed.  #### Link to tracking issue Fixes  #### Testing  #### Documentation

#### Description Add initial PR for Tinybird Exporter  #### Link to tracking issue Related to #40475  #### Testing Only config tests for now. No exporter tests because it doesn't have any logic yet.  #### Documentation - Basic README.md - chloggen  --------- Co-authored-by: Pablo Baeyens <[email protected]>

…#39864)  #### Description This PR partially implements the native histograms from prometheus v2 to the Otel exponential histograms. For now, it's missing some implementations that I take care on a next PR. Like: - [x] Deal with scale - [x] Deal with positive and negative spans - [ ] Maybe exemplars (I'm think that we can deal with exemplars in a next PR) - [x] Deal with positive and negative buckets  #### Link to tracking issue Related to #37277

#### Description Improve config documentation to align better with the code itself.  #### Link to tracking issue Fixes #40869  #### Testing No testing required - comments only  #### Documentation Changelogs

#### Description There are no active code owners so by default this component should be marked as seeking new code owners.  #### Link to tracking issue Part of #40970

#### Description Updates documentation for Datadog extension prior to v0.129.0 release   #### Documentation Readme and chloggen

#### Description Update the k8s resolver example to use v1beta1 OpentelemetryCollector CRD.  #### Link to tracking issue Fixes #40950  #### Testing Follow the modified instructions. Signed-off-by: Robbie Lankford <[email protected]>

…uery (#40809)  #### Description When a request like `http://localhost:8080/apitest?ip=myip&version=test` is processed by awsxrayexporter, the generated http.request.url found to be `http://localhost:8080/ip=myip&version=test` which appends url query parameters as path value. Expected: `http://localhost:8080/apitest?ip=myip&version=test`  #### Link to tracking issue Fixes  #### Testing Unit tests E2E manual tests  #### Documentation

…40167)  #### Description Add option `include_file_record_offset` to insert offset as attribute in log records as `log.file.record.offset`  #### Link to tracking issue Fixes #39684  #### Testing Add 2 tests 1. `TestAddOffset`: tests that the `log.file.record_offset` is correctly included when IncludeFileRecordOffset is set to true 2. `TestOffsetReadExistingLogs`: tests that the `log.file.record_offset` increments correctly  #### Documentation Document the new option in two places 1. `receiver/filelogreceiver/README.md` 2. `pkg/stanza/docs/operators/file_input.md`  --------- Co-authored-by: Braydon Kains <[email protected]> Co-authored-by: Andrzej Stencel <[email protected]>

jpkrohling requested a review from a team October 9, 2020 14:04

bogdandrutu reviewed Oct 12, 2020

View reviewed changes

README.md Outdated Show resolved Hide resolved

bogdandrutu changed the title ~~Added section on how to build an otelcol dist~~ Add section on how to build an otelcol dist Oct 12, 2020

jpkrohling added 2 commits October 12, 2020 17:31

Added section on how to build an otelcol dist

3166666

Signed-off-by: Juraci Paixão Kröhling <[email protected]>

s/OpenTracing/OpenTelemetry

acf4d51

Signed-off-by: Juraci Paixão Kröhling <[email protected]>

jpkrohling force-pushed the jpkrohling/add-link-to-blog-post-on-building-custom-otelcol branch from 73f9801 to acf4d51 Compare October 12, 2020 15:31

owais reviewed Oct 12, 2020

View reviewed changes

anuraaga reviewed Oct 13, 2020

View reviewed changes

README.md Show resolved Hide resolved

owais approved these changes Oct 13, 2020

View reviewed changes

anuraaga approved these changes Oct 14, 2020

View reviewed changes

bogdandrutu merged commit a987792 into open-telemetry:master Oct 14, 2020

kisieland referenced this pull request in kisieland/opentelemetry-collector-contrib Mar 16, 2021

Use cached tools instead of uploading to the artifacts (#2475)

9b2660f

Updates #1234 Signed-off-by: Bogdan Drutu <[email protected]>

ljmsc referenced this pull request in ljmsc/opentelemetry-collector-contrib Feb 21, 2022

Move asyncronous metrics state helper out of apimetric/metrictes (#1234)

f60f51d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add section on how to build an otelcol dist #1234

Add section on how to build an otelcol dist #1234

Uh oh!

jpkrohling commented Oct 9, 2020

Uh oh!

Uh oh!

owais Oct 12, 2020 •

edited

Loading

Uh oh!

owais Oct 12, 2020

Uh oh!

jpkrohling Oct 12, 2020 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Add section on how to build an otelcol dist #1234

Add section on how to build an otelcol dist #1234

Uh oh!

Conversation

jpkrohling commented Oct 9, 2020

Uh oh!

Uh oh!

owais Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

owais Oct 12, 2020

Choose a reason for hiding this comment

Uh oh!

jpkrohling Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

owais Oct 12, 2020 •

edited

Loading

jpkrohling Oct 12, 2020 •

edited

Loading