Collector: init memory allocation investigation, need to upgrade aws-sdk-go

[maxday]

Running the collector binary with GODEBUG="inittrace=1" allows to see init information.

An interesting part in the aws-sdk-go dependency where the alloc number is huge due to the endpoint static definition.

GODEBUG="inittrace=1" ./build/extensions/collector 2>&1 | grep "aws-sdk-go/"

init github.com/aws/aws-sdk-go/internal/sdkrand @0.88 ms, 0.038 ms clock, 5448 bytes, 3 allocs
init github.com/aws/aws-sdk-go/internal/ini @4.0 ms, 0.002 ms clock, 96 bytes, 1 allocs
init github.com/aws/aws-sdk-go/aws/credentials @4.5 ms, 0.001 ms clock, 512 bytes, 7 allocs
init github.com/aws/aws-sdk-go/aws/awsutil @4.6 ms, 0.032 ms clock, 3512 bytes, 31 allocs
-----> init github.com/aws/aws-sdk-go/aws/endpoints @4.7 ms, 3.2 ms clock, 2233936 bytes, 13063 allocs <-----
init github.com/aws/aws-sdk-go/aws @11 ms, 0.003 ms clock, 128 bytes, 2 allocs
init github.com/aws/aws-sdk-go/aws/request @11 ms, 0.010 ms clock, 832 bytes, 7 allocs
init github.com/aws/aws-sdk-go/aws/corehandlers @11 ms, 0.004 ms clock, 3184 bytes, 50 allocs
init github.com/aws/aws-sdk-go/private/protocol @11 ms, 0 ms clock, 0 bytes, 0 allocs
init github.com/aws/aws-sdk-go/private/protocol/json/jsonutil @11 ms, 0 ms clock, 48 bytes, 2 allocs
init github.com/aws/aws-sdk-go/private/protocol/rest @11 ms, 0.001 ms clock, 32 bytes, 2 allocs
init github.com/aws/aws-sdk-go/aws/signer/v4 @11 ms, 0.005 ms clock, 2136 bytes, 6 allocs
init github.com/aws/aws-sdk-go/service/sso @12 ms, 0 ms clock, 256 bytes, 2 allocs
init github.com/aws/aws-sdk-go/service/ssooidc @12 ms, 0 ms clock, 0 bytes, 0 allocs
init github.com/aws/aws-sdk-go/aws/credentials/ssocreds @12 ms, 0 ms clock, 0 bytes, 0 allocs
init github.com/aws/aws-sdk-go/service/sts @12 ms, 0 ms clock, 0 bytes, 0 allocs
init github.com/aws/aws-sdk-go/aws/session @12 ms, 0.003 ms clock, 320 bytes, 5 allocs

In addition to perform huge amount of allocation, it creates overhead during init time.

We should migrate to the sdk v2 which does not have this huge startup allocation.

Investigation which package is responsible to import this running go mod why github.com/aws/aws-sdk-go/aws/endpoints gives us

github.com/open-telemetry/opentelemetry-lambda/collector/internal/lifecycle
github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter
github.com/prometheus/prometheus/config
github.com/prometheus/common/sigv4
github.com/aws/aws-sdk-go/aws
github.com/aws/aws-sdk-go/aws/endpoints

So we need to upgrade github.com/open-telemetry/opentelemetry-collector-contrib dependency. Fortunately there is ongoing work to fix this issue here, so I created this issue to keep track of that not to forget about upgrading. open-telemetry/opentelemetry-collector-contrib#40123