podAnnotations not passed to new default-http-nginx pod in v2

[anwbtom]

Describe the bug
After upgrading from 1.6.2 to 2.0.2 the podAnnotation configuration is only applied to the data-plane of the setup.
Since we use Grafana cloud, we set the Grafana annotations ourselves through the helm chart those are not propagated to both pods only the data-plane.

To Reproduce
Steps to reproduce the behavior:
install the version 2.0.2 of nginx-gateway-fabric chart
We installed them with this template:

service:
  type: ClusterIP

terminationGracePeriodSeconds: 60


nginx:
  config:
    telemetry:
      exporter:
        endpoint: k8s-monitoring-alloy.k8s-monitoring.svc.cluster.local:4317
      spanAttributes:
      - key: k8s.cluster.name
        value: ${environment}-${cluster_name}
      - key: deployment.environment
        value: ${environment_short}
  lifecycle: # Lifecycle is necessary otherwise you'll get broken connections during downscale
    preStop:
      exec:
        command:
        - /bin/sleep
        - "60" # This flag is optional, the default is 30s
  service:
    type: ClusterIP

nginxGateway:
  configAnnotations:
    k8s.grafana.com/job: nginx-gateway-fabric

  gatewayClassAnnotations:
    k8s.grafana.com/job: nginx-gateway-fabric

  podAnnotations:
    k8s.grafana.com/metrics.portNumber: "9113"
    k8s.grafana.com/scrape: "true"
    k8s.grafana.com/job: nginx-gateway-fabric

  lifecycle: # Lifecycle is necessary otherwise you'll get broken connections during downscale
    preStop:
      exec:
        command:
        - /usr/bin/gateway
        - sleep
        - --duration=60s # This flag is optional, the default is 30s
  resources:
    requests:
      cpu: "0.00841"
      memory: 75Mi

  service:
    annotations:
      k8s.grafana.com/metrics.portNumber: "9113"
      k8s.grafana.com/scrape: "true"
      k8s.grafana.com/job: nginx-gateway-fabric

# Try to prevent upscaled pod to start on worker-node already housing one
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app.kubernetes.io/name
          operator: In
          values:
          - nginx-gateway-fabric
      topologyKey: "kubernetes.io/hostname"

gateways:
  - name: default-http
    annotations:
      k8s.grafana.com/metrics.portNumber: "9113"
      k8s.grafana.com/scrape: "true"
      k8s.grafana.com/job: nginx-gateway-fabric
    spec:
      infrastructure:
        annotations:
          k8s.grafana.com/metrics.portNumber: "9113"
          k8s.grafana.com/scrape: "true"
          k8s.grafana.com/job: nginx-gateway-fabric
      gatewayClassName: nginx
      listeners:
        - protocol: HTTP
          port: 80
          name: http
          allowedRoutes:
            kinds:
              - kind: HTTPRoute
            namespaces:
              from: All

Expected behavior
We expect to get metrics from both the control and the dataplane into our Grafana instance, but only see those of the data-plan (pods named: nginx-gateway-fabric-{hash}-{id}).
Meaning that the metric we used to scale our gateway with, no longer is available (active connections).

Your environment

• Version of the NGINX Gateway Fabric - 2.0.2
• Kubernetes platform EKS (Kubernetes 1.32)

Additional context
When we enable metrics setting, it seems the Prometheus annotations hard-coded in the chart are propagated to both control & data -plane pods.
We've tried all available keys which mention annotation in their key, but non seem to work in this specific case.
Currently we're missing the metrics in our cloud instances like:
nginx_http_connection_count_connections
nginx_http_connections_total
nginx_http_request_count_requests
nginx_http_requests_total

Which would be very helpful to have to determine our replica count and were available in v1 of this chart.

[nginx-bot]

Hi @anwbtom!

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.

[sjberman]

Hi @anwbtom, with 2.0 we split the control plane and data plane into separate Deployments, so they are now independent of each other in terms of configuration. To set annotations and labels on the data plane, you need to define these in your Gateway resource, in the spec.infrastructure fields. See https://docs.nginx.com/nginx-gateway-fabric/install/deploy-data-plane/#set-annotations-and-labels-on-provisioned-resources for more info and let me know if that helps!

[anwbtom]

@sjberman Thanks for the reply, but the keys I could find that will allow me to set annotations were the following.
None of those actually applied the label to the new control-plane pod (default-http-nginx).
The config bit I shared also already sets that infrastructure bit, and it sets the annoations on the gateway-resource but not the pod.

nginxGateway.podAnnotations
nginxGateway.gatewayClassAnnotations
nginxGateway.configAnnotations
nginxGateway.service.annotations
nginxGateway.serviceAccount.annotations
certGenerator.annotations

[sjberman]

Ah yes, you are setting them on the Gateway, I didn't look closely enough.

Just to be clear, the control plane is the controller pod running in the nginx-gateway namespace when you first install NGF, and the data plane is the nginx pod (default-http-nginx) running in your Gateway namespace. The annotations in the helm chart that you listed are all applied to control plane resources (which appear to be working as expected). But none of those are applied to the data plane.

Do any of the annotations that you specified in the Gateway infrastrcture field exist on the data plane (default-http-nginx) Deployment (not just the Pod)?

When I set the following in my Gateway resource:

infrastructure:
  annotations:
     test: value

I'm seeing the annotations on both the Deployment and Pod:

kubectl describe deployment <data-plane> and kubectl describe pod <data-plane>

Where data-plane would be default-http-nginx in your case.

...
Annotations:      prometheus.io/port: 9113
                  prometheus.io/scrape: true
                  test: value

[anwbtom]

Thanks for the response once again, thanks for the clarification on which is which, I assumed them the wrong way around.
Just as an extra ask, would there be a necessity based on traffic the scale the control-plane, or should scaling the data-plane be enough?

Just tried it again, but no it doesn't seem to work this way, through a get all, I see this is applied to the gateway:

gateways:
- annotations:
    k8s.grafana.com/job: nginx-gateway-fabric
    k8s.grafana.com/metrics.portNumber: "9113"
    k8s.grafana.com/scrape: "true"
  name: default-http
  spec:
    gatewayClassName: nginx
    infrastructure:
      annotations:
        test: tom
    listeners:
    - allowedRoutes:
        kinds:
        - kind: HTTPRoute
        namespaces:
          from: All
      name: http
      port: 80
      protocol: HTTP

So if I read your message correctly a annotation with the key test should exists with the value tom on both the deployment and the pod (changed it to make sure its not a complex key error or something), but that doesn't seem the case:

describe deployment (default-http):

Annotations:      prometheus.io/port: 9113
                  prometheus.io/scrape: true  

describe pod (default-http):

Annotations:          prometheus.io/port: 9113
                      prometheus.io/scrape: true 

[sjberman]

@anwbtom Regarding traffic, you can scale both independently. In general, scaling the data plane would suffice for handling more user traffic, but if you have a ton of data planes, scaling the control plane could be effective to handle the data plane connections better if you're seeing degraded performance in relation to configuration updates from the control plane.

And yes, you should be seeing that annotation. Not sure why that isn't being set. I'd be curious what your actual Gateway resource looks like (not the helm values one). Maybe the values aren't being set properly in the Gateway when it's created.

kubectl describe gateway <gateway-name>

[anwbtom]

It seems they indeed not propagate properly or something.
Describe from the gateway:

Name:         default-http                                                                                                                                                                                                                                                                                            Namespace:    nginx                                                                                                                                                                                                                                                                                                   Labels:       app.kubernetes.io/managed-by=Helm                                                                                                                                                                                                                                                                       Annotations:  k8s.grafana.com/job: nginx-gateway-fabric                                                                                                                                                                                                                                                                             k8s.grafana.com/metrics.portNumber: 9113                                                                                                                                                                                                                                                                              k8s.grafana.com/scrape: true
              meta.helm.sh/release-name: nginx-gateway-fabric
              meta.helm.sh/release-namespace: nginx
API Version:  gateway.networking.k8s.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2025-07-18T06:16:27Z
  Generation:          1
  Managed Fields:
    API Version:  gateway.networking.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:k8s.grafana.com/job:
          f:k8s.grafana.com/metrics.portNumber:
          f:k8s.grafana.com/scrape:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app.kubernetes.io/managed-by:
      f:spec:
        .:
        f:gatewayClassName:
        f:listeners:
          .:
          k:{"name":"http"}:
            .:
            f:allowedRoutes:
              .:
              f:kinds:
              f:namespaces:
                .:
                f:from:
            f:name:
            f:port:
            f:protocol:
    Manager:      terraform-provider-helm
    Operation:    Update
    Time:         2025-07-18T06:16:27Z
    API Version:  gateway.networking.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:addresses:
        f:conditions:
          k:{"type":"Accepted"}:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
          k:{"type":"Programmed"}:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
        f:listeners:
          .:
          k:{"name":"http"}:
            .:
            f:attachedRoutes:
            f:conditions:
              .:
              k:{"type":"Accepted"}:
                .:
                f:lastTransitionTime:
                f:message:
                f:observedGeneration:
                f:reason:
                f:status:
                f:type:
              k:{"type":"Conflicted"}:
                .:
                f:lastTransitionTime:
                f:message:
                f:observedGeneration:
                f:reason:
                f:status:
                f:type:
              k:{"type":"Programmed"}:
                .:
                f:lastTransitionTime:
                f:message:
                f:observedGeneration:
                f:reason:
                f:status:
                f:type:
              k:{"type":"ResolvedRefs"}:
                .:
                f:lastTransitionTime:
                f:message:
                f:observedGeneration:
                f:reason:
                f:status:
                f:type:
            f:name:
            f:supportedKinds:
    Manager:         gateway
    Operation:       Update
    Subresource:     status
    Time:            2025-07-18T06:22:10Z
  Resource Version:  13315
  UID:               5b2d50fc-8514-4a2e-a127-a4fc4a2fc2ee
Spec:
  Gateway Class Name:  nginx
  Listeners:
    Allowed Routes:
      Kinds:
        Group:  gateway.networking.k8s.io
        Kind:   HTTPRoute
      Namespaces:
        From:  All
    Name:      http
    Port:      80
    Protocol:  HTTP

[sjberman]

Good to know, we'll need to take a look at this bug in the helm chart. In the meantime, directly adding those annotations into the live Gateway resource instead of the helm chart should solve your problem.