[tailsamplingprocessor] Fix the decision timer metric to measure > 50ms (#37722)

#### Description

Previously the decision timer metric was in microseconds. Because the
max histogram bucket was 50000, it capped the maximum measurable latency
at 50ms. This commit changes the metric to be in milliseconds, which
allows for measuring latencies of up to 50 seconds.

Since the default decision tick is 1s, it's necessary to observe when
the latency is approaching 1s.

I'm not sure how big of a breaking change it is to update the unit, but
given the previous value was not super useful, I thought it was better
to change it.

Here's an example of the current state:
![Screenshot 2025-02-05 at 3 23
21 PM](https://github.com/user-attachments/assets/0c39fad4-961b-44c0-b018-9633b9d2ff72)

Notice how the average latency is running much higher than the p99,
indicating that we are missing important data on how slow the decision
tick actually is.

Author: Logiraptor

.chloggen/logiraptor_fix-decision-timer-metric.yaml

@@ -0,0 +1,27 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: breaking
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: tailsamplingprocessor
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Fix the decision timer metric to capture longer latencies beyond 50ms.
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [37722]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext: This changes the unit of the decision timer metric from microseconds to milliseconds.
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: []

processor/tailsamplingprocessor/documentation.md

@@ -56,11 +56,11 @@ Latency (in microseconds) of a given sampling policy
 
 ### otelcol_processor_tail_sampling_sampling_decision_timer_latency
 
-Latency (in microseconds) of each run of the sampling decision timer
+Latency (in milliseconds) of each run of the sampling decision timer
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
-| µs | Histogram | Int |
+| ms | Histogram | Int |
 
 ### otelcol_processor_tail_sampling_sampling_late_span_age
 

processor/tailsamplingprocessor/internal/metadata/generated_telemetry.go

@@ -22,8 +22,8 @@ telemetry:
         bucket_boundaries: [1, 2, 5, 10, 25, 50, 75, 100, 150, 200, 300, 400, 500, 750, 1000, 2000, 3000, 4000, 5000, 10000, 20000, 30000, 50000]
 
     processor_tail_sampling_sampling_decision_timer_latency:
-      description: Latency (in microseconds) of each run of the sampling decision timer
-      unit: µs
+      description: Latency (in milliseconds) of each run of the sampling decision timer
+      unit: ms
       enabled: true
       histogram:
         value_type: int

processor/tailsamplingprocessor/internal/metadatatest/generated_telemetrytest.go

@@ -351,7 +351,7 @@ func (tsp *tailSamplingSpanProcessor) samplingPolicyOnTick() {
 
 		decision := tsp.makeDecision(id, trace, &metrics)
 
-		tsp.telemetry.ProcessorTailSamplingSamplingDecisionTimerLatency.Record(tsp.ctx, int64(time.Since(startTime)/time.Microsecond))
+		tsp.telemetry.ProcessorTailSamplingSamplingDecisionTimerLatency.Record(tsp.ctx, int64(time.Since(startTime)/time.Millisecond))
 		tsp.telemetry.ProcessorTailSamplingGlobalCountTracesSampled.Add(tsp.ctx, 1, decisionToAttribute[decision])
 
 		// Sampled or not, remove the batches

processor/tailsamplingprocessor/metadata.yaml

@@ -137,8 +137,8 @@ func TestMetricsAfterOneEvaluation(t *testing.T) {
 			opts: []metricdatatest.Option{metricdatatest.IgnoreTimestamp(), metricdatatest.IgnoreValue()},
 			m: metricdata.Metrics{
 				Name:        "otelcol_processor_tail_sampling_sampling_decision_timer_latency",
-				Description: "Latency (in microseconds) of each run of the sampling decision timer",
-				Unit:        "µs",
+				Description: "Latency (in milliseconds) of each run of the sampling decision timer",
+				Unit:        "ms",
 				Data: metricdata.Histogram[int64]{
 					Temporality: metricdata.CumulativeTemporality,
 					DataPoints:  []metricdata.HistogramDataPoint[int64]{{}},