|
| 1 | +# Splunk Add-on for OpenTelemetry Collector |
| 2 | + |
| 3 | +This technical add-on for Splunk Universal Forwarder helps in deploying the OpenTelemetry Collector alongside Universal Forwarder. |
| 4 | + |
| 5 | +Splunk Add-on for OpenTelemetry Collector is supported on Linux and Windows on `amd64` (`x86_64`) |
| 6 | + |
| 7 | +## Contents |
| 8 | + |
| 9 | +The package contains the following folders and files - |
| 10 | + |
| 11 | +------------| |
| 12 | + |------windows_x86_64 |
| 13 | + |
| 14 | +------------| |
| 15 | + |------linux_x86_64 |
| 16 | + |
| 17 | +------------| |
| 18 | + |------README |
| 19 | + |
| 20 | +------------| |
| 21 | + |------default |
| 22 | + |
| 23 | +------------| |
| 24 | + |------configs |
| 25 | + |
| 26 | +------------| |
| 27 | + |------README.md |
| 28 | + |
| 29 | + |
| 30 | +The Windows and Linux folders contain platform specific binaries. The default folder contains the `app.conf` and `inputs.conf` files. The configs folder contains sample configuration files for collecting host metrics and traces using the OpenTelemetry Collector. |
| 31 | + |
| 32 | +## Install the add-on |
| 33 | + |
| 34 | +The installation process for the Splunk Add-on for OpenTelemetry Collector differs depending on the deployment method. |
| 35 | + |
| 36 | +### Splunk Deployment Server |
| 37 | + |
| 38 | +Follow these steps to install the add-on using Splunk Deployment Server: |
| 39 | + |
| 40 | +1. Extract the `Splunk_TA_otel.tgz` file to the `$SPLUNK_HOME/etc/deployment-apps` folder. |
| 41 | +2. Edit the configuration files. |
| 42 | +3. Follow the instructions for deploying apps. See https://docs.splunk.com/Documentation/Splunk/8.1.2/Updating/Updateconfigurations. |
| 43 | +4. Make sure to activate the restart of Universal Forwarder after deployment. |
| 44 | + |
| 45 | +### Splunk Universal Forwarder |
| 46 | + |
| 47 | +To install the add-on directly on Universal Forwarder, follow these steps: |
| 48 | + |
| 49 | +1. Log in to Universal Forwarder. |
| 50 | +2. Copy the tar file on the server. |
| 51 | +3. Extract the package to the `$SPLUNK_HOME/etc/apps` folder: |
| 52 | + |
| 53 | +``` |
| 54 | +tar -zxf Splunk_TA_otel.tgz |
| 55 | +``` |
| 56 | + |
| 57 | +## Configure the add-on |
| 58 | + |
| 59 | +Before using the add-on, create a `local` folder and write the contents of your Splunk Observability Cloud access token to this file. |
| 60 | +**WE STRONGLY RECOMMEND** that you restrict read and write to this token to the minimal possible permissions. |
| 61 | +Alternatively, you can set the `SPLUNK_ACCESS_TOKEN` environment variable to avoid needing to write this. |
| 62 | + |
| 63 | +```sh |
| 64 | +cd /opt/splunkforwarder/etc/apps/Splunk_TA_otel/ |
| 65 | +mkdir local |
| 66 | +cp -R config local |
| 67 | +touch local/access_token |
| 68 | +# make the contents of the access_token file your o11y access token. |
| 69 | +``` |
| 70 | + |
| 71 | +After all the steps are completed, restart Splunk Universal Forwarder by running `$SPLUNK_HOME/bin/splunk.exe restart` |
| 72 | + |
| 73 | +## Configure the OpenTelemetry Collector |
| 74 | + |
| 75 | +To configure the OpenTelemetry Collector within the add-on, follow these steps: |
| 76 | + |
| 77 | +1. Create a new configuration file in YAML format for the Collector. For more information, see [Configure the Collector](https://docs.splunk.com/Observability/gdi/opentelemetry/configure-the-collector.html) in the Splunk Observability Cloud documentation. |
| 78 | +2. Edit the inputs.conf file inside default to point to the new configuration file. |
| 79 | +3. Restart Splunk Universal Forwarder. |
| 80 | + |
| 81 | +Various settings in the `inputs.conf.spec` are "pass through" [environment variables](https://github.com/signalfx/splunk-otel-collector/blob/main/internal/settings/settings.go#L37-L64) to the default splunk distribution of the opentelemetry collector's agent configuration. We do not currently pass through things related to `gateway` nor for the splunk hec related settings. That said, the binary vended in the TA is identical to upstream |
| 82 | + |
| 83 | +Note that we additionally configure the following environment variables in the hopes of making configuration easier. |
| 84 | +- `$SPLUNK_OTEL_TA_HOME` References the location of the TA, ex `/opt/splunk/etc/apps/Splunk_TA_Otel` |
| 85 | +- `$SPLUNK_OTEL_TA_PLATFORM_HOME` references the location of the platform-specific configuration for the TA, ex `/opt/splunk/etc/apps/Splunk_TA_Otel/linux_x86_64`. By default the `splunk_bundle_dir` and `splunk_collectd_dir` options in `inputs.conf` references this environment variable. |
| 86 | + |
| 87 | +Finally, setting `$SPLUNK_OTEL_TA_DEBUG` to anything other than an empty string will provide detailed logging messages during TA start up. |
| 88 | + |
| 89 | + |
| 90 | +## Check operational status |
| 91 | + |
| 92 | +Both the add-on and the OpenTelemetry Collector generate log files to indicate operational status. |
| 93 | + |
| 94 | +You can find the log files in the `$SPLUNK_HOME/var/log/splunk/` folder. |
| 95 | + |
| 96 | +- The Splunk add-on log file is `Splunk_TA_otel.log`. |
| 97 | +- By default, the OpenTelemetry Collector log file is `otel.log` (although you can override this path to whatever you please). |
| 98 | + |
| 99 | +## Explore metrics and traces |
| 100 | + |
| 101 | +You can browse the metrics and traces collected by the add-on in in Splunk Observability Cloud. See the [Splunk Observability Cloud documentation for more information](https://docs.splunk.com/Observability). |
| 102 | + |
| 103 | +If you're using the default configuration, you can search [metrics finder](https://docs.splunk.com/observability/en/metrics-and-metadata/metrics-finder-metadata-catalog.html#metrics-finder-and-metadata-catalog) for [`splunk.distribution:otel-ta`](https://app.signalfx.com/#/metrics?sources%5B%5D=splunk.distribution:otel-ta). |
| 104 | + |
| 105 | +*note*: if you're using your own configuration, *please* continue to include the telemetry in the configuration. |
| 106 | + |
| 107 | + |
| 108 | +Under `processors` |
| 109 | +``` |
| 110 | + resource/telemetry: |
| 111 | + attributes: |
| 112 | + - action: insert |
| 113 | + key: splunk.distribution |
| 114 | + value: otel-ta |
| 115 | +``` |
| 116 | + |
| 117 | +Under `receivers` |
| 118 | +``` |
| 119 | + # This section is used to collect the OpenTelemetry Collector metrics |
| 120 | + # Even if just a Splunk APM customer, these metrics are included |
| 121 | + prometheus/internal: |
| 122 | + config: |
| 123 | + scrape_configs: |
| 124 | + - job_name: 'otel-collector' |
| 125 | + scrape_interval: 10s |
| 126 | + static_configs: |
| 127 | + - targets: ["${env:SPLUNK_LISTEN_INTERFACE}:8888"] |
| 128 | + metric_relabel_configs: |
| 129 | + - source_labels: [ __name__ ] |
| 130 | + regex: '.*grpc_io.*' |
| 131 | + action: drop |
| 132 | +``` |
| 133 | + |
| 134 | +Under `pipelines`, assuming you're using a `signalfx` exporter (only `otlp` is supported for our internal telemetry metrics) |
| 135 | +``` |
| 136 | + metrics/telemetry: |
| 137 | + receivers: [prometheus/internal] |
| 138 | + processors: [memory_limiter, batch, resourcedetection, resource/telemetry] |
| 139 | + exporters: [signalfx] |
| 140 | +``` |
| 141 | + |
| 142 | +# Binary File Declaration |
| 143 | +`linux_x86_64/bin/otelcol_linux_amd64` |
| 144 | +`windows_x86_64/bin/otelcol_windows_amd64.exe` |
0 commit comments