From d11a42611f447018b3c2ce19f7db32ed246a2784 Mon Sep 17 00:00:00 2001 From: Alex Russell Date: Mon, 7 Jul 2025 14:32:02 -0600 Subject: [PATCH 1/6] NLB-6740 Updated the SUPPORT and CONTRIBUTING guides --- CONTRIBUTING.md | 37 +++++++++++++++++++++++++------------ SUPPORT.md | 16 ++++++++-------- 2 files changed, 33 insertions(+), 20 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0e900ec6..72e6b218 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,7 +2,7 @@ The following is a set of guidelines for contributing to the nginx_loadbalancer_kubernetes. We really appreciate that you are considering contributing! -#### Table Of Contents +## Table Of Contents [Getting Started](#getting-started) @@ -24,11 +24,25 @@ We are not currently accepting contributions to this project. Please create an i ### Report a Bug -To report a bug, open an issue on GitHub with the label `bug` using the available bug report issue template. Please ensure the bug has not already been reported. **If the bug is a potential security vulnerability, please report using our security policy.** +To report a bug, open an issue on GitHub with the label `bug` using the available [bug report issue form](/.github/ISSUE_TEMPLATE/bug_report.yml). Please ensure the bug has not already been reported. **If the bug is a potential security vulnerability, please report it using our [security policy](/SECURITY.md).** ### Suggest a Feature or Enhancement -To suggest a feature or enhancement, please create an issue on GitHub with the label `feature` or `enhancement` using the available feature request issue template. Please ensure the feature or enhancement has not already been suggested. +To suggest a feature or enhancement, please create an issue on GitHub with the label `enhancement` using the available [feature request issue form](/.github/ISSUE_TEMPLATE/feature_request.yml). Please ensure the feature or enhancement has not already been suggested. + +### Open a Pull Request (PR) + +- Fork the repo, create a branch, implement your changes, add any relevant tests, and submit a PR when your changes are **tested** and ready for review. +- Fill in the [PR template](/.github/pull_request_template.md). + +> [!NOTE] +> If you'd like to implement a new feature, please consider creating a [feature request issue](/.github/ISSUE_TEMPLATE/feature_request.yml) first to start a discussion about the feature. + +#### F5 Contributor License Agreement (CLA) + +F5 requires all contributors to agree to the terms of the F5 CLA (available [here](https://github.com/f5/f5-cla/blob/main/docs/f5_cla.md)) before any of their changes can be incorporated into an F5 Open Source repository (even contributions to the F5 CLA itself!). + +If you have not yet agreed to the F5 CLA terms and submit a PR to this repository, a bot will prompt you to view and agree to the F5 CLA. You will have to agree to the F5 CLA terms through a comment in the PR before any of your changes can be merged. Your agreement signature will be safely stored by F5 and no longer be required in future PRs. ## Code Guidelines @@ -36,12 +50,11 @@ To suggest a feature or enhancement, please create an issue on GitHub with the l ### Git Guidelines -* Keep a clean, concise and meaningful git commit history on your branch (within reason), rebasing locally and squashing before submitting a PR. -* If possible and/or relevant, use the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) format when writing a commit message, so that changelogs can be automatically generated -* Follow the guidelines of writing a good commit message as described here and summarised in the next few points: - * In the subject line, use the present tense ("Add feature" not "Added feature"). - * In the subject line, use the imperative mood ("Move cursor to..." not "Moves cursor to..."). - * Limit the subject line to 67 characters or less. - * Limit the rest of the commit message to 72 characters or less. - * Reference issues and pull requests liberally after the subject line. - * Add more detailed description in the body of the git message (`git commit -a` to give you more space and time in your text editor to write a good message instead of `git commit -am`). +- Keep a clean, concise and meaningful git commit history on your branch (within reason), rebasing locally and squashing before submitting a PR. +- If possible and/or relevant, use the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) format when writing a commit message, so that changelogs can be automatically generated. +- Follow the guidelines of writing a good commit message as described here and summarized in the next few points: + - In the subject line, use the present tense ("Add feature" not "Added feature"). + - In the subject line, use the imperative mood ("Move cursor to..." not "Moves cursor to..."). + - Limit the subject line to 72 characters or less. + - Reference issues and pull requests liberally after the subject line. + - Add more detailed description in the body of the git message (`git commit -a` to give you more space and time in your text editor to write a good message instead of `git commit -am`). diff --git a/SUPPORT.md b/SUPPORT.md index 45bc32f7..ff353bab 100644 --- a/SUPPORT.md +++ b/SUPPORT.md @@ -10,13 +10,9 @@ Don't know how something in the nginx_loadbalancer_kubernetes works? Curious if This isn't the right place to get support for NGINX specific questions, but the following resources are available below. Thanks for your understanding! -### Community Slack +### Community Forum -We have a community [Slack](https://nginxcommunity.slack.com/)! - -If you are not a member click [here](https://join.slack.com/t/nginxcommunity/shared_invite/zt-1aaa22w80-~_~wSMNyPxLPLp5xunOC7w) to sign up (and let us know if the link does not seem to be working!) - -Once you join, check out the `#beginner-questions` and `nginx-users` channels :) +We have a community [forum](https://community.nginx.org/)! If you have any questions and/or issues, try checking out the [`Troubleshooting`](https://community.nginx.org/c/troubleshooting/8) and [`How do I...?`](https://community.nginx.org/c/how-do-i/9) categories. Both fellow community members and NGINXers might be able to help you! :) ### Documentation @@ -24,6 +20,10 @@ For a comprehensive list of all NGINX directives, check out . For a comprehensive list of admin and deployment guides for all NGINX products, check out . -### Mailing List +## Contributing + +Please see the [contributing guide](/CONTRIBUTING.md) for guidelines on how to best contribute to this project. + +## Community Support -Want to get in touch with the NGINX dev team directly? Try using the relevant mailing list found at ! +This project does **not** offer commercial support. Community support is offered on a best effort basis through either GitHub issues/PRs/discussions or through any of our active communities. From 66d559b548e8bbe597a52e1f4c164f3b482bec9f Mon Sep 17 00:00:00 2001 From: Alex Russell Date: Tue, 1 Jul 2025 15:05:34 -0600 Subject: [PATCH 2/6] NLB-6740 Removed values from configuration that were no longer referenced This includes references to the handler which was removed in an earlier code change. Also cleaned up references to annotations and namespaces that were used by the original NLK code, but are no longer used. --- internal/configuration/configuration_test.go | 28 +------- internal/configuration/settings.go | 68 ++----------------- .../testdata/multiple-nginx-hosts.yaml | 2 - .../testdata/one-nginx-host.yaml | 2 - internal/synchronization/synchronizer_test.go | 5 +- 5 files changed, 8 insertions(+), 97 deletions(-) diff --git a/internal/configuration/configuration_test.go b/internal/configuration/configuration_test.go index 45764d6e..c9aa4b69 100644 --- a/internal/configuration/configuration_test.go +++ b/internal/configuration/configuration_test.go @@ -22,20 +22,8 @@ func TestConfiguration_Read(t *testing.T) { LogLevel: "warn", NginxPlusHosts: []string{"https://10.0.0.1:9000/api"}, SkipVerifyTLS: false, - Handler: configuration.HandlerSettings{ - RetryCount: 5, - Threads: 1, - WorkQueueSettings: configuration.WorkQueueSettings{ - RateLimiterBase: time.Second * 2, - RateLimiterMax: time.Second * 60, - Name: "nlk-handler", - }, - }, Synchronizer: configuration.SynchronizerSettings{ - MaxMillisecondsJitter: 750, - MinMillisecondsJitter: 250, - RetryCount: 5, - Threads: 1, + Threads: 1, WorkQueueSettings: configuration.WorkQueueSettings{ RateLimiterBase: time.Second * 2, RateLimiterMax: time.Second * 60, @@ -54,20 +42,8 @@ func TestConfiguration_Read(t *testing.T) { LogLevel: "warn", NginxPlusHosts: []string{"https://10.0.0.1:9000/api", "https://10.0.0.2:9000/api"}, SkipVerifyTLS: true, - Handler: configuration.HandlerSettings{ - RetryCount: 5, - Threads: 1, - WorkQueueSettings: configuration.WorkQueueSettings{ - RateLimiterBase: time.Second * 2, - RateLimiterMax: time.Second * 60, - Name: "nlk-handler", - }, - }, Synchronizer: configuration.SynchronizerSettings{ - MaxMillisecondsJitter: 750, - MinMillisecondsJitter: 250, - RetryCount: 5, - Threads: 1, + Threads: 1, WorkQueueSettings: configuration.WorkQueueSettings{ RateLimiterBase: time.Second * 2, RateLimiterMax: time.Second * 60, diff --git a/internal/configuration/settings.go b/internal/configuration/settings.go index 75cec2e5..f34397ec 100644 --- a/internal/configuration/settings.go +++ b/internal/configuration/settings.go @@ -15,38 +15,16 @@ import ( ) const ( - // ConfigMapsNamespace is the value used to filter the ConfigMaps Resource in the Informer. - ConfigMapsNamespace = "nlk" - - // ConfigMapName is the name of the ConfigMap that contains the configuration for the application. - ConfigMapName = "nlk-config" - - // ResyncPeriod is the value used to set the resync period for the Informer. - ResyncPeriod = 0 - - // NlkPrefix is used to determine if a Port definition should be handled and used to update a Border Server. - // The Port name () must start with this prefix, e.g.: - // nlk- - NlkPrefix = ConfigMapsNamespace + "-" - - // PortAnnotationPrefix defines the prefix used when looking up a Port in the Service Annotations. - // The value of the annotation determines which BorderServer implementation will be used. - // See the documentation in the `application/application_constants.go` file for details. - PortAnnotationPrefix = "nginxinc.io" - - // ServiceAnnotationMatchKey is the key name of the annotation in the application's config map - // that identifies the ingress service whose events will be monitored. + // ServiceAnnotationMatchKey is the key name of the annotation that + // identifies the services whose events will be monitored. ServiceAnnotationMatchKey = "service-annotation-match" - // DefaultServiceAnnotation is the default name of the ingress service whose events will be + // DefaultServiceAnnotation is the default name of the services whose events will be // monitored. DefaultServiceAnnotation = "nginxaas" ) -// WorkQueueSettings contains the configuration values needed by the Work Queues. -// There are two work queues in the application: -// 1. nlk-handler queue, used to move messages between the Watcher and the Handler. -// 2. nlk-synchronizer queue, used to move message between the Handler and the Synchronizer. +// WorkQueueSettings contains the configuration for the nlk-synchronizer queue. // The queues are NamedDelayingQueue objects that use an ItemExponentialFailureRateLimiter // as the underlying rate limiter. type WorkQueueSettings struct { @@ -61,18 +39,6 @@ type WorkQueueSettings struct { RateLimiterMax time.Duration } -// HandlerSettings contains the configuration values needed by the Handler. -type HandlerSettings struct { - // RetryCount is the number of times the Handler will attempt to process a message before giving up. - RetryCount int - - // Threads is the number of threads that will be used to process messages. - Threads int - - // WorkQueueSettings is the configuration for the Handler's queue. - WorkQueueSettings WorkQueueSettings -} - // WatcherSettings contains the configuration values needed by the Watcher. type WatcherSettings struct { // ServiceAnnotation is the annotation of the ingress service whose events the watcher should monitor. @@ -84,15 +50,6 @@ type WatcherSettings struct { // SynchronizerSettings contains the configuration values needed by the Synchronizer. type SynchronizerSettings struct { - // MaxMillisecondsJitter is the maximum number of milliseconds that will be applied when adding an event to the queue. - MaxMillisecondsJitter int - - // MinMillisecondsJitter is the minimum number of milliseconds that will be applied when adding an event to the queue. - MinMillisecondsJitter int - - // RetryCount is the number of times the Synchronizer will attempt to process a message before giving up. - RetryCount int - // Threads is the number of threads that will be used to process messages. Threads int @@ -114,9 +71,6 @@ type Settings struct { // APIKey is the api key used to authenticate with the dataplane API. APIKey string - // Handler contains the configuration values needed by the Handler. - Handler HandlerSettings - // Synchronizer contains the configuration values needed by the Synchronizer. Synchronizer SynchronizerSettings @@ -157,20 +111,8 @@ func Read(configName, configPath string) (s Settings, err error) { NginxPlusHosts: v.GetStringSlice("nginx-hosts"), SkipVerifyTLS: skipVerifyTLS, APIKey: base64.StdEncoding.EncodeToString([]byte(v.GetString("NGINXAAS_DATAPLANE_API_KEY"))), - Handler: HandlerSettings{ - RetryCount: 5, - Threads: 1, - WorkQueueSettings: WorkQueueSettings{ - RateLimiterBase: time.Second * 2, - RateLimiterMax: time.Second * 60, - Name: "nlk-handler", - }, - }, Synchronizer: SynchronizerSettings{ - MaxMillisecondsJitter: 750, - MinMillisecondsJitter: 250, - RetryCount: 5, - Threads: 1, + Threads: 1, WorkQueueSettings: WorkQueueSettings{ RateLimiterBase: time.Second * 2, RateLimiterMax: time.Second * 60, diff --git a/internal/configuration/testdata/multiple-nginx-hosts.yaml b/internal/configuration/testdata/multiple-nginx-hosts.yaml index 2235c1ae..c7e52acf 100644 --- a/internal/configuration/testdata/multiple-nginx-hosts.yaml +++ b/internal/configuration/testdata/multiple-nginx-hosts.yaml @@ -1,5 +1,3 @@ -ca-certificate: "fakeCAKey" -client-certificate: "fakeCertKey" log-level: "warn" nginx-hosts: ["https://10.0.0.1:9000/api", "https://10.0.0.2:9000/api"] tls-mode: "no-tls" diff --git a/internal/configuration/testdata/one-nginx-host.yaml b/internal/configuration/testdata/one-nginx-host.yaml index 45e55eff..f722e4a8 100644 --- a/internal/configuration/testdata/one-nginx-host.yaml +++ b/internal/configuration/testdata/one-nginx-host.yaml @@ -1,5 +1,3 @@ -ca-certificate: "fakeCAKey" -client-certificate: "fakeCertKey" log-level: "warn" nginx-hosts: "https://10.0.0.1:9000/api" service-annotation-match: "fakeServiceMatch" diff --git a/internal/synchronization/synchronizer_test.go b/internal/synchronization/synchronizer_test.go index ba2253b2..c365e6a8 100644 --- a/internal/synchronization/synchronizer_test.go +++ b/internal/synchronization/synchronizer_test.go @@ -257,10 +257,7 @@ func defaultSettings(nginxHosts ...string) configuration.Settings { return configuration.Settings{ NginxPlusHosts: nginxHosts, Synchronizer: configuration.SynchronizerSettings{ - MaxMillisecondsJitter: 750, - MinMillisecondsJitter: 250, - RetryCount: 5, - Threads: 1, + Threads: 1, WorkQueueSettings: configuration.WorkQueueSettings{ RateLimiterBase: time.Second * 2, RateLimiterMax: time.Second * 60, From 313e1bdb69a8fa8c66d9fda59517dbceeeab79cd Mon Sep 17 00:00:00 2001 From: Alex Russell Date: Tue, 1 Jul 2025 16:26:15 -0600 Subject: [PATCH 3/6] NLB-6740 docstrings updates Updating the original NLK docstrings to bring them in line with the latest architectural changes. --- charts/nlk/values.yaml | 2 +- internal/application/application_constants.go | 22 +++++------- internal/application/doc.go | 34 +++++++------------ internal/observation/watcher.go | 8 +++-- internal/synchronization/doc.go | 2 +- internal/translation/doc.go | 2 +- internal/translation/translator.go | 15 ++++---- 7 files changed, 36 insertions(+), 49 deletions(-) diff --git a/charts/nlk/values.yaml b/charts/nlk/values.yaml index 1f2f1f2f..e6f81470 100644 --- a/charts/nlk/values.yaml +++ b/charts/nlk/values.yaml @@ -33,7 +33,7 @@ nlk: ## trace,debug,info,warn,error,fatal,panic logLevel: "info" - ## the nginx hosts (comma-separated) to send upstream updates to + ## the nginx hosts to send upstream updates to. For multiple hosts, use a sequence nginxHosts: "" ## Sets the annotation value that NLK is looking for to watch a Service # serviceAnnotationMatch: nginxaas diff --git a/internal/application/application_constants.go b/internal/application/application_constants.go index 4cb23a54..6c300867 100644 --- a/internal/application/application_constants.go +++ b/internal/application/application_constants.go @@ -5,20 +5,16 @@ package application -// These constants are intended for use in the Annotations field of the Service definition. -// They determine which Border Server client will be used. -// To use these values, add the following annotation to the Service definition: +// These constants determine which Border Server client will be used. The name +// of the port on the desired service needs to be formattted as follows: +// "http-tea", where the first part is the protocol ("http" or "stream") and the +// second part is the name of the upstream (in this case "tea"). // -// annotations: -// nginxinc.io/nlk-: -// -// where is the name of the upstream in the NGINX Plus configuration -// and is one of the constants below. -// -// Note, this is an extensibility point. To add a Border Server client... -// 1. Create a module that implements the BorderClient interface; -// 2. Add a new constant to this group that acts as a key for selecting the client; -// 3. Update the NewBorderClient factory method in border_client.go that returns the client; +// Note, this is an extensibility point. To add a Border Server client... 1. +// Create a module that implements the BorderClient interface; 2. Add a new +// constant to this group that acts as a key for selecting the client; 3. Update +// the NewBorderClient factory method in border_client.go that returns the +// client; const ( // ClientTypeNginxStream creates a NginxStreamBorderClient that uses the Stream* methods of the NGINX Plus client. diff --git a/internal/application/doc.go b/internal/application/doc.go index 296cb67c..36a01fe6 100644 --- a/internal/application/doc.go +++ b/internal/application/doc.go @@ -3,32 +3,22 @@ * Use of this source code is governed by the Apache License that can be found in the LICENSE file. */ -/* -Package application includes support for applying updates to the Border servers. - -"Border Servers" are servers that are exposed to the outside world and direct traffic into the cluster. - -The BorderClient module defines an interface that can be implemented to support other Border Server types. -To add a Border Server client... -1. Create a module that implements the BorderClient interface; -2. Add a new constant in application_constants.go that acts as a key for selecting the client; -3. Update the NewBorderClient factory method in border_client.go that returns the client; - -At this time the only supported Border Servers are NGINX Plus servers. +// Package application includes support for applying updates to the Border servers. -The two Border Server clients for NGINX Plus are: -- NginxHTTPBorderClient: updates NGINX Plus servers using HTTP Upstream methods on the NGINX Plus API. -- NginxStreamBorderClient: updates NGINX Plus servers using Stream Upstream methods on the NGINX Plus API. +// "Border Servers" are servers that are exposed to the outside world and direct traffic into the cluster. -Both of these implementations use the NGINX Plus client module to communicate with the NGINX Plus server. +// The BorderClient module defines an interface that can be implemented to support other Border Server types. +// To add a Border Server client... +// 1. Create a module that implements the BorderClient interface; +// 2. Add a new constant in application_constants.go that acts as a key for selecting the client; +// 3. Update the NewBorderClient factory method in border_client.go that returns the client; -Selection of the appropriate client is based on the Annotations present on the Service definition, e.g.: +// At this time the only supported Border Servers are NGINX Plus servers. - annotations: - nginxinc.io/nlk-: +// The two Border Server clients for NGINX Plus are: +// - NginxHTTPBorderClient: updates NGINX Plus servers using HTTP Upstream methods on the NGINX Plus API. +// - NginxStreamBorderClient: updates NGINX Plus servers using Stream Upstream methods on the NGINX Plus API. -where is the name of the upstream in the NGINX Plus configuration -and is one of the constants in application_constants.go. -*/ +// Both of these implementations use the NGINX Plus client module to communicate with the NGINX Plus server. package application diff --git a/internal/observation/watcher.go b/internal/observation/watcher.go index 9711347c..77efa0e2 100644 --- a/internal/observation/watcher.go +++ b/internal/observation/watcher.go @@ -21,9 +21,11 @@ import ( "k8s.io/client-go/tools/cache" ) -// Watcher is responsible for watching for changes to Kubernetes resources. -// Particularly, Services in the namespace defined in the WatcherSettings::NginxIngressNamespace setting. -// When a change is detected, an Event is generated and added to the Handler's queue. +// Watcher is responsible for watching for changes to kubernetes services marked +// with the correct annotation. NLK watches for changes to the services +// themselves, but also changes to the nodes, endpoint slices and load balancers +// associated with those services. When a change is detected, an Event is +// generated and added to the synchronizer's queue. type Watcher struct { synchronizer synchronization.Interface diff --git a/internal/synchronization/doc.go b/internal/synchronization/doc.go index 90f56023..ec5a69bd 100644 --- a/internal/synchronization/doc.go +++ b/internal/synchronization/doc.go @@ -4,7 +4,7 @@ */ /* -Package synchronization includes functionality to synchronize Ingress events to NGINX+ Configuration API commands. +Package synchronization includes functionality to synchronize kubernetes service events to NGINX upstream updates. */ package synchronization diff --git a/internal/translation/doc.go b/internal/translation/doc.go index c9dc86ca..8e1762d2 100644 --- a/internal/translation/doc.go +++ b/internal/translation/doc.go @@ -4,7 +4,7 @@ */ /* -Package translation includes functionality to translate Ingress events to target system definitions. +Package translation includes functionality to translate service events into NGINX upstream server updates. */ package translation diff --git a/internal/translation/translator.go b/internal/translation/translator.go index 8491a668..4eb2464c 100644 --- a/internal/translation/translator.go +++ b/internal/translation/translator.go @@ -33,20 +33,19 @@ func NewTranslator( } } -// Translate transforms event data into an intermediate format that can be consumed by the BorderClient implementations -// and used to update the Border Servers. +// Translate transforms service events into upstream server updates that can be implemented by the border client. func (t *Translator) Translate(event *core.Event) (core.ServerUpdateEvents, error) { slog.Debug("Translate::Translate") return t.buildServerUpdateEvents(event.Service.Spec.Ports, event) } -// buildServerUpdateEvents builds a list of ServerUpdateEvents based on the event type -// The NGINX+ Client uses a list of servers for Created and Updated events. -// The client performs reconciliation between the list of servers in the NGINX+ Client call -// and the list of servers in NGINX+. -// The NGINX+ Client uses a single server for Deleted events; -// so the list of servers is broken up into individual events. +// buildServerUpdateEvents builds a list of ServerUpdateEvents based on the +// event type. The NGINX+ Client uses a list of servers for Created and Updated +// events. The client performs reconciliation between the list of servers in the +// NGINX+ Client call and the list of servers in NGINX+. If a delete event is +// received, the translator generates an empty list of servers: the client +// reconciliation will then cause all servers to be deleted. func (t *Translator) buildServerUpdateEvents(ports []v1.ServicePort, event *core.Event, ) (events core.ServerUpdateEvents, err error) { slog.Debug("Translate::buildServerUpdateEvents", "ports", ports) From 432b45aec924a7749621ed1e172110f79886678a Mon Sep 17 00:00:00 2001 From: Alex Russell Date: Tue, 1 Jul 2025 16:32:16 -0600 Subject: [PATCH 4/6] NLB-6740 Updated the design doc Updating the original NLK design doc to bring it in line with the latest architectural changes. --- docs/DESIGN.md | 74 ++++++++++++++++---------------------------------- 1 file changed, 23 insertions(+), 51 deletions(-) diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 63f859c1..4d6e252b 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -1,22 +1,20 @@ # Overview -The nginx-loadbalancer-kubernetes runs in a Kubernetes Cluster and responds to changes in resources of interest, updating designated NGINX Plus hosts with the appropriate configuration. +The nginx-loadbalancer-kubernetes runs in a Kubernetes cluster and responds to changes in specified services by issuing requests to the designated NGINX Plus hosts to update the appropriate upstreams. ## Basic Architecture -The controller is deployed in a Kubernetes Cluster. Upon startup, it registers interest in changes to Service resources in the "nginx-ingress" namespace. -The Handler accepts the events raised by the Cluster and calls the Translator to convert the events into event definitions that are used to update NGINX Plus hosts. -Next, the Handler calls the Synchronizer with the list of translated events which are fanned-out for each NGINX host. -Lastly, the Synchronizer calls the [NGINX+ Configuration API](https://docs.nginx.com/nginx/admin-guide/load-balancer/dynamic-configuration-api/) using the [NGINX Plus Go client](https://github.com/nginxinc/nginx-plus-go-client) to update the target NGINX Plus host(s). +The controller is deployed in a Kubernetes cluster. Upon startup, it registers interest in changes to services with the appropriate annotation. +The Watcher listens for events raised by the cluster and then adds these events to the Synchronizer's queue. The Synchronizer makes use of the Translator to transform the events into specific updates that will be sent to the NGINX Plus hosts. +Lastly, the Synchronizer calls the [NGINX+ Configuration API](https://docs.nginx.com/nginx/admin-guide/load-balancer/dynamic-configuration-api/) using the [NGINX Plus Go client](https://github.com/nginxinc/nginx-plus-go-client) to update the target NGINX Plus host(s). ```mermaid stateDiagram-v2 Controller --> Watcher Controller --> Settings - Watcher --> Handler : "nlk-handler queue" - Handler --> Translator - Translator --> Handler - Handler --> Synchronizer : "nlk-synchronizer queue" + Watcher --> Synchronizer : "nlk-synchronizer queue" + Synchronizer --> Translator + Translator --> Synchronizer Synchronizer --> BorderClient : "HttpBorderClient | TcpBorderClient" BorderClient --> NGINXPlusLB1 BorderClient --> NGINXPlusLB2 @@ -26,38 +24,31 @@ stateDiagram-v2 ### Settings -The Settings module is responsible for loading the configuration settings from the "nlk-config" ConfigMap resource in the "nlk" namespace. -The Settings are loaded when the controller starts and are reloaded when the "nlk-config" ConfigMap resource is updated. +The Settings module is responsible for loading the configuration settings when the controller starts. Any changes made to the settings through helm at runtime will cause nginx-loadbalancer-kubernetes to be restarted. ### Watcher -The Watcher is responsible for monitoring changes to Service resources in the "nginx-ingress" namespace. -It registers methods that handle each event type. Events are handled by creating a `core.Event` instance and adding it to the "nlk-handler" queue. -When adding the event to the queue, the Watcher also retrieves the list of Node IPs and adds the list to the event. -The master node ip is excluded from the list. (NOTE: This should be configurable.) - -### Handler - -The Handler is responsible for taking the `core.Event` instances from the "nlk-handler" queue and calling the Translator to convert the event into a `core.ServerUpdateEvent` instance, -adding each `core.ServerUpdateEvent` to the "nlk-synchronizer" queue. +The Watcher is responsible for monitoring changes to Service resources that are annotated by the user. +It registers methods that handle each event type. An event is created for a particular service and is added to the "nlk-synchronizer" queue. ### Translator -The Translator is responsible for converting the `core.Event` event into an `nginxClient.UpstreamServer` event. -This involves filtering out the `core.Event` instances that are not of interest to the controller by accepting only Port names starting with the NlkPrefix value (currently _nlk-_). -The event is then fanned-out based on the defined Ports, one event per defined Port. Each port is then augmented with the Ingress name (the name configured in the Port definition with the NlkPrefix value removed), -and the list of the Node's IP addresses. +The Translator is responsible for converting a `core.Event` event into an `nginxClient.UpstreamServer` event. -The Translator passes the list of events to the Synchronizer by calling the `AddEvents` method. +The Translator assembles a list of IP addresses for the Kubernetes service, routing traffic to the NodePorts, ClusterIPs or external load balancer, depending on the Kubernetes service type. If the service is of type NodePort, the Translator excludes the master node IP from the list of the addresses. (NOTE: This should be configurable.) + +The Translator uses the port name to determine the protocol and the name of the user's upstream. For example, if the user has named their service port "http-tea", the translator will create an update event for the HTTP upstream named "tea". **NOTE: It is important that the Port names match the name of the defined NGINX Plus Upstreams.** -In the following example the NGINX Plus Upstreams are named "nlk-nginx-lb-http" and "nlk-nginx-lb-https". These match the name in the NGINX Plus configuration. +In the following example the service uses the `nginx.com/nginxaas: nginxaas` annotation so that NLK knows to monitor the service. NGINX Plus Upstreams are named "ingress" and "ingress-tls". These match the upstream names specified in the NGINX configuration on the NGINX Plus host. ```yaml apiVersion: v1 kind: Service metadata: + annotations: + nginx.com/nginxaas: nginxaas name: nginx-ingress namespace: nginx-ingress spec: @@ -66,11 +57,11 @@ spec: - port: 80 targetPort: 80 protocol: TCP - name: nlk-nginx-lb-http + name: http-ingress - port: 443 targetPort: 443 protocol: TCP - name: nlk-nginx-lb-https + name: http-ingress-tls selector: app: nginx-ingress ``` @@ -78,36 +69,17 @@ spec: ### Synchronizer The Synchronizer is responsible for fanning-out the given list of `core.ServerUpdateEvent` events, one for each configured NGINX Plus host. -The NGINX Plus hosts are configured using a ConfigMap resource named "nlk-config" in the "nlk" namespace. An example of the ConfigMap is shown below. +The NGINX Plus hosts are configured through the helm chart `nlk.config.nginxHosts` value. Use a string for a single host and a sequence for multiple hosts. See the comments in `charts/nlk/values.yaml` for details. -```yaml -apiVersion: v1 -kind: ConfigMap -data: - nginx-hosts: - "http://10.1.1.4:9000/api,http://10.1.1.5:9000/api" -metadata: - name: nlk-config - namespace: nlk -``` - -This example includes two NGINX Plus hosts to support High Availability. - -Additionally, the Synchronizer is responsible for taking the `core.ServerUpdateEvent` instances from the "nlk-synchronizer" queue and updating the target NGINX Plus host. The Synchronizer uses the [NGINX Plus Go client](https://github.com/nginxinc/nginx-plus-go-client) to communicate with each NGINX Plus host. - #### Retry Mechanism -The Synchronizer uses a retry mechanism to handle failures when updating the NGINX Plus hosts. -The retry mechanism is implemented in the workqueue using the `workqueue.NewItemExponentialFailureRateLimiter`, +The Synchronizer uses a retry mechanism to handle failures when updating the NGINX Plus hosts. +The retry mechanism is implemented in the workqueue using the `workqueue.NewItemExponentialFailureRateLimiter`, having defaults set to a base of 2 seconds, and a maximum of 60 seconds. -### Jitter - -The Synchronizer uses a jitter mechanism to avoid thrashing the NGINX Plus hosts. Each `core.ServerUpdateEvent` instance -is added to the "nlk-synchronizer" queue with a random jitter value between 250 and 750 milliseconds. - ## Authors + - Steve Wagner - Solutions Architect - Community and Alliances @ F5, Inc. - Chris Akker - Solutions Architect - Community and Alliances @ F5, Inc. From a6bd4a969e3d0d99b95b52b4b5a37c745df025fe Mon Sep 17 00:00:00 2001 From: Alex Russell Date: Wed, 2 Jul 2025 13:57:54 -0600 Subject: [PATCH 5/6] NLB-6740 Updated the README doc Updating the original NLK README to bring it in line with the latest architectural changes. --- README.md | 31 +++++++++---------- docs/README.md | 80 +++++++++++++++----------------------------------- 2 files changed, 38 insertions(+), 73 deletions(-) diff --git a/README.md b/README.md index 226a9738..d2467830 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,4 @@ + [![CI](https://github.com/nginxinc/nginx-loadbalancer-kubernetes/actions/workflows/build-test.yml/badge.svg)](https://github.com/nginxinc/nginx-loadbalancer-kubernetess/actions/workflows/build-test.yml) [![Go Report Card](https://goreportcard.com/badge/github.com/nginxinc/nginx-loadbalancer-kubernetes)](https://goreportcard.com/report/github.com/nginxinc/nginx-loadbalancer-kubernetes) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) @@ -8,8 +9,6 @@ [![Community Support](https://badgen.net/badge/support/community/cyan?icon=awesome)](https://github.com/nginxinc/nginx-loadbalancer-kubernetes/discussions) [![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active) - -
@@ -17,7 +16,6 @@
- The NGINX Loadbalancer for Kubernetes, or _NLK_, is a Kubernetes controller that provides TCP load balancing external to a Kubernetes cluster running on-premise. ## Requirements @@ -70,7 +68,7 @@ NLK itself does not perform load balancing. Rather, NLK allows you to manage Ser There are few bits of administrivia to get out of the way before you can start leveraging NLK for your load balancing needs. As noted above, NLK is intended for when you have one or more Kubernetes clusters running on-premise. In addition to this, -you need to have at least one NGINX Plus host running outside your cluster (Please refer to the [Roadmap](#Roadmap) for information about other load balancer servers). +you need to have at least one NGINX Plus host running outside your cluster (Please refer to the [Roadmap](#roadmap) for information about other load balancer servers). ### Deployment @@ -107,27 +105,27 @@ There is a much more detailed [Installation Reference](docs/README.md) available 1. Clone this repo (optional, you can simply copy the `deployments/` directory) -```git clone git@github.com:nginxinc/nginx-loadbalancer-kubernetes.git``` + ```git clone git@github.com:nginxinc/nginx-loadbalancer-kubernetes.git``` -2. Apply the Namespace +1. Apply the Namespace -```kubectl apply -f deployments/deployment/namespace.yaml``` + ```kubectl apply -f deployments/deployment/namespace.yaml``` -3. Apply the RBAC resources +1. Apply the RBAC resources -```./deployments/rbac/apply.sh``` + ```./deployments/rbac/apply.sh``` -4. Update / Apply the ConfigMap (For best results update the `nginx-hosts` values first) +1. Update / Apply the ConfigMap (For best results update the `nginx-hosts` values first) -```kubectl apply -f deployments/deployment/configmap.yaml``` + ```kubectl apply -f deployments/deployment/configmap.yaml``` -5. Apply the Deployment +1. Apply the Deployment -```kubectl apply -f deployments/deployment/deployment.yaml``` + ```kubectl apply -f deployments/deployment/deployment.yaml``` -6. Check the logs +1. Check the logs -```kubectl -n nlk logs -f $(kubectl -n nlk get po -l "app=nlk" --no-headers -o custom-columns=":metadata.name")``` + ```kubectl -n nlk logs -f $(kubectl -n nlk get po -l "app=nlk" --no-headers -o custom-columns=":metadata.name")``` At this point NLK should be up and running. Now would be a great time to go over to the [Installation Reference](docs/README.md) and follow the instructions to deploy a demo application. @@ -164,11 +162,10 @@ diverse ideas will be key to NLK becoming a solution that is useful to the commu when we are able to accept pull requests from the community. ## Authors + - Chris Akker - Solutions Architect - Community and Alliances @ F5, Inc. - Steve Wagner - Solutions Architect - Community and Alliances @ F5, Inc. -
- ## License [Apache License, Version 2.0](https://github.com/nginxinc/nginx-loadbalancer-kubernetes/blob/main/LICENSE) diff --git a/docs/README.md b/docs/README.md index 7335ab39..6348a02f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,110 +1,78 @@ # nginx-loadbalancer-kubernetes -
- -# Welcome to the Nginx LoadBalancer for Kubernetes Solution! - -
+## Welcome to the NGINX LoadBalancer for Kubernetes Solution ![Nginx K8s LB](media/nlk-logo.png) | ![Nginx K8s LB](media/nginx-2020.png) --- | --- -
- -This repo contains source code and documents for a new `Kubernetes Controller from Nginx`, that provides TCP and HTTP load balancing external to a Kubernetes Cluster running On Premises. - -
- ->>**This is a replacement for a Cloud Providers `Service Type Loadbalancer`, that is not available for On Premises Kubernetes Clusters.** +This repo contains source code and documents for a `Kubernetes Controller from NGINX`, that provides TCP and HTTP load balancing external to a Kubernetes cluster running on premises. -
-
+>>**This is a replacement for a cloud provider's `Service Type Loadbalancer`, that is not available for on premises Kubernetes clusters.** +## Overview -# Overview - -- `NLK - NGINX Loadbalancer for Kubernetes` is a new K8s Controller from Nginx, that monitors specified K8s Services, and then sends API calls to an external Nginx Plus server to manage Nginx Upstream servers dynamically. -- This will `synchronize` the K8s Service Endpoint list, with the Nginx LB Server's upstream list. -- The primary use case and Solution provided is for tracking the K8s` NodePort` IP:Port definitions for the Nginx Ingress Controller's `nginx-ingress Service`. -- NLK is a native Kubernetes Controller, running, configured and managed with standard K8s commands. -- NLK paired with the Nginx Plus Server located external to the K8s cluster, this new controller LB function will provide a `TCP Load Balancer Service` for On Premises K8s clusters, which do not have access to a Cloud providers "Service Type LoadBalancer". -- NLK paired with the Nginx Plus Server located external to the Cluster, using Nginx's advanced HTTP features, provide an `HTTP Load Balancer Service` for Enterprise traffic management solutions, such as: - - MultiCluster Active/Active Load Balancing - - Horizontal Cluster Scaling - - HTTP Split Clients - for A/B, Blue/Green, and Canary test and production traffic steering. Allows Cluster operations/maintainence like upgrades, patching, expansion and troubleshooting with no downtime or reloads - - Advanced TLS Processing - MutualTLS, OCSP, FIPS, dynamic cert loading - - Advanced Security features - Oauth, JWT, App Protect WAF Firewall, Rate and Bandwidth limits - - Nginx Java Script (NJS) for custom solutions - - Nginx Zone Sync of KeyVal data - -
+- `NLK - NGINX Loadbalancer for Kubernetes` is a Kubernetes controller from NGINX that monitors specified Kubernetes services, and then sends API calls to an external NGINX Plus server to manage NGINX Upstream servers dynamically. +- This will `synchronize` the Kubernetes service's endpoints with the NGINX server's upstream list. +- One use case is to track the`NodePort` IP:Port definitions for the NGINX Ingress Controller's `nginx-ingress Service`. +- NLK is a native Kubernetes controller, running, configured and managed with standard Kubernetes commands. +- When NLK is paired with the NGINX Plus Server located external to the Kubernetes cluster, this controller will provide a `TCP Load Balancer Service` for on premises Kubernetes clusters, which do not have access to a cloud provider's "Service Type LoadBalancer". +- NLK paired with the NGINX Plus Server located external to the Cluster, using NGINX's advanced HTTP features, provides an `HTTP Load Balancer Service` for enterprise traffic management solutions, such as: + - MultiCluster Active/Active Load Balancing + - Horizontal Cluster Scaling + - HTTP Split Clients - for A/B, Blue/Green, and Canary test and production traffic steering. Allows Cluster operations/maintainence like upgrades, patching, expansion and troubleshooting with no downtime or reloads + - Advanced TLS Processing - MutualTLS, OCSP, FIPS, dynamic cert loading + - Advanced Security features - Oauth, JWT, App Protect WAF Firewall, Rate and Bandwidth limits + - NGINX Java Script (NJS) for custom solutions + - NGINX Zone Sync of KeyVal data ## NLK Controller Software Design Overview - How it works [NLK Controller DESIGN and Architecture](DESIGN.md) -
- ## Reference Diagram for NLK TCP Load Balancer Service -
- ![NLK Stream Diagram](media/nlk-blog-diagram-v1.png) -
- ## Sample Screenshots of Solution at Runtime -
- ![NGINX LB ConfigMap](media/nlk-configmap.png) -### ConfigMap with 2 Nginx LB Servers defined for HA -
+### ConfigMap with 2 NGINX LB Servers defined for HA ![NGINX LB Create Nodeport](media/nlk-stream-create-nodeport.png) -### Nginx LB Server Dashboard, NodePort, and NLK Controller Logging -### Legend: +### NGINX LB Server Dashboard, NodePort, and NLK Controller Logging + +### Legend + - Red - kubectl nodeport commands - Blue - nodeport and upstreams for http traffic - Indigo - nodeport and upstreams for https traffic - Green - NLK log for api calls to LB Server #1 -- Orange - Nginx LB Server upstream dashboard details +- Orange - NGINX LB Server upstream dashboard details - Kubernetes Worker Nodes are 10.1.1.8 and 10.1.1.10 -
- The `Installation Guide` for TCP Loadbalancer Solution is located in the tcp folder: [TCP Installation Guide](tcp/tcp-installation-guide.md) -
- The `Installation Guide` for HTTP Loadbalancer Solution is located in the http folder: [HTTP Installation Guide](http/http-installation-guide.md) -
- ## Requirements Please see the /docs folder and Installation Guides for detailed documentation. -
- ## Development Read the [`CONTRIBUTING.md`](https://github.com/nginxinc/nginx-loadbalancer-kubernetes/blob/main/CONTRIBUTING.md) file. -
- ## Authors + - Chris Akker - Solutions Architect - Community and Alliances @ F5, Inc. - Steve Wagner - Solutions Architect - Community and Alliances @ F5, Inc. -
- ## License [Apache License, Version 2.0](https://github.com/nginxinc/nginx-loadbalancer-kubernetes/blob/main/LICENSE) From db8f162245cc33fbb23bb169fcff9db2f71357bd Mon Sep 17 00:00:00 2001 From: Alex Russell Date: Mon, 7 Jul 2025 11:19:08 -0600 Subject: [PATCH 6/6] NLB-6740 Added copyright declaration to files without it --- internal/configuration/configuration_test.go | 5 +++++ internal/core/event_test.go | 5 +++++ internal/observation/register.go | 5 +++++ internal/synchronization/cache.go | 5 +++++ pkg/buildinfo/buildinfo.go | 5 +++++ pkg/pointer/pointer.go | 5 +++++ pkg/pointer/pointer_test.go | 5 +++++ 7 files changed, 35 insertions(+) diff --git a/internal/configuration/configuration_test.go b/internal/configuration/configuration_test.go index c9aa4b69..add7dbc0 100644 --- a/internal/configuration/configuration_test.go +++ b/internal/configuration/configuration_test.go @@ -1,3 +1,8 @@ +/* + * Copyright (c) 2023 F5 Inc. All rights reserved. + * Use of this source code is governed by the Apache License that can be found in the LICENSE file. + */ + package configuration_test import ( diff --git a/internal/core/event_test.go b/internal/core/event_test.go index 09724cfa..572b1b0b 100644 --- a/internal/core/event_test.go +++ b/internal/core/event_test.go @@ -1,3 +1,8 @@ +/* + * Copyright (c) 2023 F5 Inc. All rights reserved. + * Use of this source code is governed by the Apache License that can be found in the LICENSE file. + */ + package core import ( diff --git a/internal/observation/register.go b/internal/observation/register.go index bfe61f80..76cc6b73 100644 --- a/internal/observation/register.go +++ b/internal/observation/register.go @@ -1,3 +1,8 @@ +/* + * Copyright (c) 2023 F5 Inc. All rights reserved. + * Use of this source code is governed by the Apache License that can be found in the LICENSE file. + */ + package observation import ( diff --git a/internal/synchronization/cache.go b/internal/synchronization/cache.go index 14effb99..87722f35 100644 --- a/internal/synchronization/cache.go +++ b/internal/synchronization/cache.go @@ -1,3 +1,8 @@ +/* + * Copyright (c) 2023 F5 Inc. All rights reserved. + * Use of this source code is governed by the Apache License that can be found in the LICENSE file. + */ + package synchronization import ( diff --git a/pkg/buildinfo/buildinfo.go b/pkg/buildinfo/buildinfo.go index 5d8839d1..2896fafd 100644 --- a/pkg/buildinfo/buildinfo.go +++ b/pkg/buildinfo/buildinfo.go @@ -1,3 +1,8 @@ +/* + * Copyright (c) 2023 F5 Inc. All rights reserved. + * Use of this source code is governed by the Apache License that can be found in the LICENSE file. + */ + package buildinfo var semVer string diff --git a/pkg/pointer/pointer.go b/pkg/pointer/pointer.go index 08ff6672..9fd6c696 100644 --- a/pkg/pointer/pointer.go +++ b/pkg/pointer/pointer.go @@ -1,3 +1,8 @@ +/* + * Copyright (c) 2023 F5 Inc. All rights reserved. + * Use of this source code is governed by the Apache License that can be found in the LICENSE file. + */ + // Package pointer provides utilities that assist in working with pointers. package pointer diff --git a/pkg/pointer/pointer_test.go b/pkg/pointer/pointer_test.go index e929e58a..b0dd50f9 100644 --- a/pkg/pointer/pointer_test.go +++ b/pkg/pointer/pointer_test.go @@ -1,3 +1,8 @@ +/* + * Copyright (c) 2023 F5 Inc. All rights reserved. + * Use of this source code is governed by the Apache License that can be found in the LICENSE file. + */ + package pointer_test import (