Collect more runtime metrics

[pkositsyn]

Is your feature request related to a problem?
Sometimes it's useful to have goroutines count or gc related metrics exposed

Describe the solution you'd like
Add these metrics to collector internal metrics exporter

Describe alternatives you've considered
Adding configuration would be useful as well, but I don't see an elegant way to add it for internal exporter

[bogdandrutu]

I think we should remove the custom runtime metrics from the service, and use one of the way we do allow collecting of runtime metrics for any go lang binary.

I would suggest we expose expvar on our service and scrape these metrics using an expvar receiver. There are some reasons to do this:

1. This is a generic mechanism that allows us to monitor any golang binary not just the collector.
2. Avoid unnecessary code to scrape runtime metrics for this binary.

Thoughts?

[bogdandrutu]

@Vemmy124 alternative is to rely on opentelemetry-go library to produce these metrics

[rakyll]

@bogdandrutu The only issue with the expvar is that there are not many tools that knows how to parse it. Otherwise, I agree with your points.

[pkositsyn]

@bogdandrutu @rakyll I agree that expvar isn't popular and it's a real problem, which includes using an exporter almost for sure for monitoring (e.g. another collector with expvar receiver and prometheus exporter + still a problem of monitoring the last collector). Moreover, I doubt its efficiency with lock for every variable update but might be not that bad.

By the way, I don't mind writing some lines for collecting system metrics actually. Do you think that prometheus is not enough standardized for usage? If this is the case, I don't know a good alternative.

[bogdandrutu]

@Vemmy124 look into opentelemetry-go-contrib, already support for more runtime metrics. We should use that instead of OpenCensus for recording internal metrics and install the runtime plugin from there

[jpkrohling]

Instead of expvar, we should consider exposing this following the OpenMetrics format. Quite a lot of tools can understand it already, including the tool that is the most popular today in this area (Prometheus).

[pkositsyn]

To be honest even opentelemetry-go alone seems complicated for me as a client library. @bogdandrutu Do I understand correctly that you want to replace OpenCensus for metrics in the whole repository? Could you provide reasons for that and what's wrong with current approach? This issue is stale only because I cannot see the purpose of changing a lot of things. Think I need a vision for that