Deploy the Data Plane with Control Plane

[mpstefan]

As a user of NGF
I want the control plane to deploy the data plane
So that I do not need to manage it's deployment via Helm
And so that data plane instance management is done by the control plane.

Acceptance

• When a Gateway object is created within the cluster and targets NGF as the implementation, the data plane is deployed with the configuration specified via Gateway API.
• Data plane deployment and service are configurable with the same fields exposed via Helm via the NginxProxy CRD
  • Only the fields the user specifies in the NginxProxy CRD will be used to update a running data plane deployment.
  • If the NginxProxy is referenced, but is invalid or nonexistent, then default values are used.
• On startup, the control plane should ensure that any existing nginx deployments/services have the proper configuration
  • This ensures that if the control plane goes down, and any NginxProxy changes occurred during that downtime, that they are applied on restart
  • It also ensures that when the control plane is updated to a new version, we patch and rollout the updated nginx pods.
• Gateway resource should be able to exist in any namespace, and the nginx deployment is created in that namespace
  • Any bootstrap resources (secrets, configmaps, etc.) also need to be created in that namespace
  • If deleted by the user, our control plane should re-create them.
• Prometheus metrics endpoint is configured via agent conf, and nginx metrics can be scraped properly by Prometheus
• Enable GatewayInfrastructurePropagation conformance tests
• Remove temporary NGINX templates from the Helm chart.

Dev Notes

• There is a risk that we could be missing some details from our design doc. We should double and triple check this to ensure we don't miss any:
  • https://github.com/nginxinc/nginx-gateway-fabric/tree/main/docs/proposals/control-data-plane-split