Snowflake receiver doesn't seem to be collecting storage data #29750
Description
Component(s)
receiver/snowflake
What happened?
Description
When using the Snowflake receiver, I noticed that we weren't getting the storage metrics I'd expect:
snowflake.storage.stage_bytes.total, snowflake.storage.storage_bytes.total or snowflake.storage.failsafe_bytes.total. It does appear that the query for this is running, as I can see the query in the query history. I don't, however, see this getting sent to my OTLP endpoint (New Relic, in this case), or even logged out using the debug exporter.
All other metrics are received, including optional metrics.
Steps to Reproduce
Docker container with config for Snowflake connector and
Expected Result
I should see logs in the debug exporter that it's finding metrics that it's querying data, or at least
Actual Result
No logs, no errors, no data in otlp endpoint.
Collector version
0.90.1
Environment information
Official Docker Container
OpenTelemetry Collector configuration
receivers:
snowflake:
username: ${SNOWFLAKE_USERNAME}
password: ${SNOWFLAKE_PASSWORD}
account: my-snowflake-account
warehouse: my-warehouse
collection_interval: 1m
metrics:
[...]
# All other metrics are explicitly disabled, including default metrics.
snowflake.storage.stage_bytes.total:
enabled: true
snowflake.storage.storage_bytes.total:
enabled: true
snowflake.storage.failsafe_bytes.total:
enabled: true
[...]
processors:
batch:
exporters:
debug:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
otlp:
# New Relic Config
[...]
service:
telemetry:
logs:
level: "debug"
pipelines:
metrics:
receivers:
- snowflake
processors: [batch]
exporters: [otlp, debug]Log output
snowflake-otel-collector | 2023-12-12T00:08:07.346Z info [email protected]/telemetry.go:86 Setting up own telemetry...
snowflake-otel-collector | 2023-12-12T00:08:07.346Z info [email protected]/telemetry.go:203 Serving Prometheus metrics {"address": ":8888", "level": "Basic"}
snowflake-otel-collector | 2023-12-12T00:08:07.346Z debug [email protected]/extension.go:162 Beta component. May change in the future. {"kind": "extension", "name": "zpages"}
snowflake-otel-collector | 2023-12-12T00:08:07.346Z debug [email protected]/extension.go:162 Beta component. May change in the future. {"kind": "extension", "name": "health_check"}
snowflake-otel-collector | 2023-12-12T00:08:07.346Z debug [email protected]/exporter.go:273 Stable component. {"kind": "exporter", "data_type": "metrics", "name": "otlp"}
snowflake-otel-collector | 2023-12-12T00:08:07.346Z info [email protected]/exporter.go:275 Development component. May change in the future. {"kind": "exporter", "data_type": "metrics", "name": "debug"}
snowflake-otel-collector | 2023-12-12T00:08:07.346Z debug [email protected]/processor.go:287 Stable component. {"kind ": "processor", "name": "batch", "pipeline": "metrics"}
snowflake-otel-collector | 2023-12-12T00:08:07.346Z debug [email protected]/receiver.go:294 Alpha component. May change in the future. {"kind": "receiver", "name": "snowflake", "data_type": "metrics"} snowflake-otel-collector | 2023-12-12T00:08:07.346Z debug [email protected]/receiver.go:294 Alpha component. May change in the future. {"kind": "receiver", "name": "snowflake", "data_type": "metrics"} snowflake-otel-collector | 2023-12-12T00:08:07.347Z info [email protected]/service.go:148 Starting otelcol-contrib... {"Version": "0.90.1", "NumCPU": 20} snowflake-otel-collector | 2023-12-12T00:08:07.347Z info extensions/extensions.go:34 Starting extensions...
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info extensions/extensions.go:37 Extension is starting... {"kind": "extension", "name": "health_check"}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info [email protected]/healthcheckextension.go:35 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/status/check","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z warn [email protected]/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info extensions/extensions.go:45 Extension started. {"kind": "extension", "name": "health_check"}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info extensions/extensions.go:37 Extension is starting... {"kind": "extension", "name": "zpages"}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info [email protected]/zpagesextension.go:53 Registered zPages span processor on tracer provider {"kind": "extension", "name": "zpages"}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info [email protected]/zpagesextension.go:63 Registered Host's zPages {"kind": "extension", "name": "zpages"}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info [email protected]/zpagesextension.go:75 Starting zPages extension {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":":55679"}}}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info extensions/extensions.go:45 Extension started. {"kind": "extension", "name": "zpages"}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel created {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] original dial target is: "otlp.nr-data.net:4318" {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] parsed dial target is: {URL:{Scheme:otlp.nr-data.net Opaque:4318 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}} {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] fallback to scheme "passthrough" {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] parsed dial target is: {URL:{Scheme:passthrough Opaque: User: Host: Path:/otlp.nr-data.net:4318 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}} {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel authority set to "otlp.nr-data.net:4318" {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Resolver state updated: {
snowflake-otel-collector | "Addresses": [
snowflake-otel-collector | {
snowflake-otel-collector | "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector | "ServerName": "",
snowflake-otel-collector | "Attributes": null,
snowflake-otel-collector | "BalancerAttributes": null,
snowflake-otel-collector | "Metadata": null
snowflake-otel-collector | }
snowflake-otel-collector | ],
snowflake-otel-collector | "Endpoints": [
snowflake-otel-collector | {
snowflake-otel-collector | "Addresses": [
snowflake-otel-collector | {
snowflake-otel-collector | "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector | "ServerName": "",
snowflake-otel-collector | "Attributes": null,
snowflake-otel-collector | "BalancerAttributes": null,
snowflake-otel-collector | "Metadata": null
snowflake-otel-collector | }
snowflake-otel-collector | ],
snowflake-otel-collector | "Attributes": null
snowflake-otel-collector | }
snowflake-otel-collector | ],
snowflake-otel-collector | "ServiceConfig": null,
snowflake-otel-collector | "Attributes": null
snowflake-otel-collector | } (resolver returned new addresses) {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel switches to new LB policy "pick_first" {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [pick-first-lb 0xc002d81a10] Received new config {
snowflake-otel-collector | "shuffleAddressList": false
snowflake-otel-collector | }, resolver state {
snowflake-otel-collector | "Addresses": [
snowflake-otel-collector | {
snowflake-otel-collector | "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector | "ServerName": "",
snowflake-otel-collector | "Attributes": null,
snowflake-otel-collector | "BalancerAttributes": null,
snowflake-otel-collector | "Metadata": null
snowflake-otel-collector | }
snowflake-otel-collector | ],
snowflake-otel-collector | "Endpoints": [
snowflake-otel-collector | {
snowflake-otel-collector | "Addresses": [
snowflake-otel-collector | {
snowflake-otel-collector | "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector | "ServerName": "",
snowflake-otel-collector | "Attributes": null,
snowflake-otel-collector | "BalancerAttributes": null,
snowflake-otel-collector | "Metadata": null
snowflake-otel-collector | }
snowflake-otel-collector | ],
snowflake-otel-collector | "Attributes": null
snowflake-otel-collector | }
snowflake-otel-collector | ],
snowflake-otel-collector | "ServiceConfig": null,
snowflake-otel-collector | "Attributes": null
snowflake-otel-collector | } {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1 SubChannel #2] Subchannel created {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel Connectivity change to CONNECTING {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1 SubChannel #2] Subchannel picks a new address "otlp.nr-data.net:4318" to connect {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info zapgrpc/zapgrpc.go:178 [core] [pick-first-lb 0xc002d81a10] Received SubConn state update: 0xc002d81bc0, {ConnectivityState:CONNECTING ConnectionError:<nil>} {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info healthcheck/handler.go:132 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"}
snowflake-otel-collector | 2023-12-12T00:08:07.347Z info [email protected]/service.go:174 Everything is ready. Begin running and processing data.
snowflake-otel-collector | 2023-12-12T00:08:07.507Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.508Z info zapgrpc/zapgrpc.go:178 [core] [pick-first-lb 0xc002d81a10] Received SubConn state update: 0xc002d81bc0, {ConnectivityState:READY ConnectionError:<nil>} {"grpc_log": true}
snowflake-otel-collector | 2023-12-12T00:08:07.508Z info zapgrpc/zapgrpc.go:178 [core] [Channel #1] Channel Connectivity change to READY {"grpc_log": true}Additional context
I'm not sure if it matters, but the type in the client.go is of type int64, but it looks like the type returned from Snowflake might be of type float64 like some of the other types.
For example, this query (from the client.go):
storageMetricsQuery = "select STORAGE_BYTES, STAGE_BYTES, FAILSAFE_BYTES from STORAGE_USAGE ORDER BY USAGE_DATE DESC LIMIT 1;"
returns the following:
STORAGE_BYTES STAGE_BYTES FAILSAFE_BYTES
123412341234123.000000 123412341234.000000 12341234123412.000000
But frankly, I'm not sure if there is some type conversion happening somewhere that would impact this, or if it matters at all.
Just to be clear, these are the only three metrics that don't work. If I explicity enable all other metrics, or allow defaults with empty metric:, I see all of the other data points reflected in the OTLP endpoint.