Add metrics documentation

[saad-ali]

I need to add documentation to https://kubernetes-csi.github.io/docs/sidecar-containers.html

Background:

A new CSI Metrics Library was added to csi-lib-utils in and is part of v0.7.0 release. This library can be used to automatically generate Prometheus metrics for all CSI operations including total count, error count, and call latency. This library was integrated in to the following CSI Sidecar containers:

• Add prometheus metrics to CSI external-provisioner using new csi-lib-utils library external-provisioner#388
• Add prometheus metrics to CSI external-attacher using new csi-lib-utils library external-attacher#201
• Add prometheus metrics to CSI external-snapshotter using new csi-lib-utils library external-snapshotter#227
• Add prometheus metrics to CSI external-resizer using new csi-lib-utils library external-resizer#67

New flags “--metrics-address” or “--metrics-path” are now part of all 4 of those sidecars. Driver deployments should set those flags to ensure the metrics are being emitted.

[pohly]

It would be good have a short example how those metrics can be used. Not sure whether that belongs into that documentation (which is probably more reference-oriented) or into a blog post.

[pohly]

For a full example, integration with Prometheus and a Grafana dashboard would be useful. While investigating this, I found: https://github.com/helm/charts/tree/master/stable/prometheus#scraping-pod-metrics-via-annotations

But that only works for a single metrics endpoint per pod. When running external-provisioner, external-attacher, external-snapshotter and external-resizer all in the same statefulset and thus pod it won't be that easy, right?

[pohly]

See prometheus/prometheus#3756

[pohly]

CSI calls issued by kubelet are not exported yet?

[pohly]

Would it make sense for CSI drivers to export the same function count metric?

The code in https://github.com/saad-ali/csi-lib-utils/blob/e9a22428988a90ba8d833b5e235fcd22d16cd5fa/metrics/metrics.go currently doesn't support that:

• only has an interceptor for the gRPC client, but not the server
• hard-codes "csi_sidecar" as subsystem

The subsystem string then appears in metrics names like csi_sidecar_operations_seconds_count.

I could imagine that correlating those different counts may be useful, for example to detect when calls have problems at the transport level and don't reach the CSI driver.

[pohly]

After having read through the config documentation I believe I understand enough of it to replace or extend the example configuration such that it scrapes each sidecar container individually.

But then the problem remains that admins will have to add that to their Prometheus configuration. I don't see an easy way to do that when deploying through helm. If I understand it right, one can replace the entire default config, but not add to it.

[pohly]

If I understand it right, one can replace the entire default config, but not add to it.

That turned out to be wrong. There is some limited support for extending the default configuration.

I found a solution with an additional, generic scrape config and filed helm/charts#22899 to figure out whether that is something that should be supported by the Helm chart out-of-the-box.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[pohly]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[pohly]

/remove-lifecycle stale
/lifecycle frozen