Control Leader Election with Agent

[mpstefan]

As a cluster operator of NGF
I want the control plane of NGF to fail-over to another pod when the control plane pod goes down
So that I can ensure I am providing a highly available control plane to the product teams using my cluster.

Acceptance

• NGF needs to elect a lead control plane pod.
  • The lead control plane pod is marked as ready and all other pods as unready.
  • All Agent connections from the data plane should be to the control plane leader.
  • If the leader is not marked as ready, the leader should switch to another available control plane pod.
• Non-leader pods should not send any config.

Dev Notes

• Only the leader pod should be marked as ready by Kubernetes, all other pods are marked as unready.
• Any pod that tries to connect to NGF will only connect to the ready pod.

Dependencies

• Update Data Plane Config via Agent #2842