Description
Discussed in #13051 (comment), connected with #7441, we have evidence that existing hard-coded parts of confighttp should be instead configured via middleware. In particular, we observe that rate-limiting is sometimes desirable via middleware both before and after the compression middleware (for network bytes and request count/bytes, respectively).
As for how this could be done -- here are some ideas:
At least one feature flag will be required.
Presently compression settings are built-in to both configgrpc and confighttp for standard codecs. We can keep standard settings in place by registering synthetic middlewares with names such as "compression" and "decompression" which, when inserted into the middlewares, cause the standard configuration to be used.
Ideally users are able to experiment with alternative compression codecs by registering alternative middleware, especially for clients. With the feature on, we should be able to name a compression extension instead of using a built-ins.
For servers, which typically handle multiple codecs, it's typical to register a static list of codecs. For gRPC this is already supported, extensions are able to register gRPC compression codecs. We could extend the server's CompressionAlgorithms field to recognize middleware extensions by name.