Skip to content

Add CEL validation test for targetRef in ClientSettingsPolicy #3623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 60 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
dbb5b5c
Add CEL validation test for targetRef in ClientSettingsPolicy
shaun-nx Jul 14, 2025
48ec86b
gofumpt
shaun-nx Jul 14, 2025
c7129bd
Add tests for targetRefGroup
shaun-nx Jul 14, 2025
87a4e0e
Rename tests
shaun-nx Jul 14, 2025
29f201e
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 15, 2025
d921ed2
Move tests into clientsettingspolicy_test.go
shaun-nx Jul 15, 2025
f22269a
Update tests to create a ClientSettingsPolicy resource during tests
shaun-nx Jul 15, 2025
7fddffa
make lint in tests
shaun-nx Jul 15, 2025
dd66a79
Update tests to create a ClientSettingsPolicy object during validation
shaun-nx Jul 17, 2025
d664f35
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 17, 2025
cd9bada
Update TargetRegGroup tests
shaun-nx Jul 17, 2025
7d9989d
Fix lint errors
shaun-nx Jul 17, 2025
9274e57
Group valid and invalid test cases into single test function
shaun-nx Jul 21, 2025
506b01c
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 21, 2025
f99ab29
Revert dependency version
shaun-nx Jul 21, 2025
ad80b70
Move imports
shaun-nx Jul 21, 2025
e5479fb
Define constants for Kinds and Groups
shaun-nx Jul 21, 2025
1789701
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 22, 2025
aa4a6b2
Use controller-runtime library to get cluster information
shaun-nx Jul 22, 2025
8f74214
Add Makefile targets to run CEL test and re format test
shaun-nx Jul 22, 2025
4486bd5
Ensure TestClientSettingsPoliciesTargetRefKind and TestClientSettings…
shaun-nx Jul 22, 2025
1adf168
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 22, 2025
7c6dba4
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 23, 2025
69f485b
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 23, 2025
1ad24c4
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 25, 2025
aa5adc6
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 28, 2025
29f26ef
Update test to use `gomega` test library
shaun-nx Jul 28, 2025
f26eea3
Add ci workflow job for cel tests
shaun-nx Jul 28, 2025
efa4a81
Add new line
shaun-nx Jul 28, 2025
d25661a
Add `Deploy Kubernetes` step
shaun-nx Jul 28, 2025
c413895
Add step to apply CRDs in cel-tests CI job
shaun-nx Jul 28, 2025
9962e03
Add `--server-side` to apply command
shaun-nx Jul 28, 2025
feca4ed
Add `working-directory` to test run and remove code coverage steps
shaun-nx Jul 28, 2025
8d84e7b
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 28, 2025
ddce60f
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 29, 2025
d518ada
Add t.Parallel()
shaun-nx Jul 29, 2025
39a6b38
Use g.Expect for errors and ensure unique policy names
shaun-nx Jul 29, 2025
5079c21
Return and assert errors from `getKubernetesClient`
shaun-nx Jul 29, 2025
ce0960b
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 29, 2025
248df53
Remove nil error checks
shaun-nx Jul 29, 2025
c5860e2
Create and return client on same line
shaun-nx Jul 29, 2025
83139ce
Use `rand.Prime` to attempt to keep policy names unique for each para…
shaun-nx Jul 29, 2025
a4e58ee
Remove unused dependency
shaun-nx Jul 29, 2025
339e6b8
Fix imports
shaun-nx Jul 29, 2025
716d051
Add helper functions to gernated unique resource names
shaun-nx Jul 30, 2025
d273281
Move `getKubernetesClient` function to helpers
shaun-nx Jul 30, 2025
4a6e84e
Fix test failures
shaun-nx Jul 30, 2025
66b7105
Update `TestMustCreateKubernetesClient` to only assert client creation
shaun-nx Jul 30, 2025
510aac1
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 30, 2025
c45403a
Remove test for creating kubernetes cluster
shaun-nx Jul 30, 2025
e284740
Add common finctions for cel tests
shaun-nx Jul 30, 2025
06adc0e
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 30, 2025
cb43b52
Move common constatns to `common.go` and add v1alpha2 to client schema
shaun-nx Jul 30, 2025
9217339
Move helper functions to common.go and add tests
shaun-nx Jul 30, 2025
f30f117
unexport consnts and functions in cel common file
shaun-nx Jul 30, 2025
e18b9dc
unexport randomPrimeNumber and uniqueResourceName
shaun-nx Jul 30, 2025
869a091
Fix casing in comments
shaun-nx Jul 30, 2025
92caf12
Remove duplicate error checks
shaun-nx Jul 31, 2025
81fd868
Initialise ginko and k8sClient at the start of each test
shaun-nx Jul 31, 2025
94a8dd0
Merge branch 'main' into tests/cel-clientsettingspolicies
shaun-nx Jul 31, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions tests/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ STANDARD_CONFORMANCE_PROFILES = GATEWAY-HTTP,GATEWAY-GRPC
EXPERIMENTAL_CONFORMANCE_PROFILES = GATEWAY-TLS
CONFORMANCE_PROFILES = $(STANDARD_CONFORMANCE_PROFILES) # by default we use the standard conformance profiles. If experimental is enabled we override this and add the experimental profiles.
SKIP_TESTS =
CEL_TEST_TARGET =

# Check if ENABLE_EXPERIMENTAL is true
ifeq ($(ENABLE_EXPERIMENTAL),true)
Expand Down Expand Up @@ -186,3 +187,22 @@ uninstall-ngf: ## Uninstall NGF on configured kind cluster
-make uninstall-gateway-crds
-kubectl delete namespace nginx-gateway
-kubectl kustomize ../config/crd | kubectl delete -f -

# Run CEL validation integration tests against a real cluster
.PHONY: test-cel-validation
test-cel-validation:
@if [ -z "$(CEL_TEST_TARGET)" ]; then \
echo "Running all tests in ./cel"; \
go test -v ./cel; \
else \
echo "Running test: $(CEL_TEST_TARGET) in ./cel"; \
go test -v ./cel -run "$(CEL_TEST_TARGET)"; \
fi


# Set up a kind cluster, install NGF CRDs, and run CEL validation tests
.PHONY: setup-and-test-cel-validation
setup-and-test-cel-validation:
kind create cluster --name $(CLUSTER_NAME) || true
kubectl kustomize ../config/crd | kubectl apply -f -
$(MAKE) test-cel-validation
226 changes: 226 additions & 0 deletions tests/cel/clientsettingspolicy_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
package cel

import (
"context"
"fmt"
"strings"
"testing"
"time"

"k8s.io/apimachinery/pkg/runtime"
controllerruntime "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
gatewayv1alpha2 "sigs.k8s.io/gateway-api/apis/v1alpha2"

ngfAPIv1alpha1 "github.com/nginx/nginx-gateway-fabric/apis/v1alpha1"
)

const (
GatewayKind = "Gateway"
HTTPRouteKind = "HTTPRoute"
GRPCRouteKind = "GRPCRoute"
TCPRouteKind = "TCPRoute"
InvalidKind = "InvalidKind"
)

const (
GatewayGroup = "gateway.networking.k8s.io"
InvalidGroup = "invalid.networking.k8s.io"
DiscoveryGroup = "discovery.k8s.io/v1"
)

const (
ExpectedTargetRefKindError = `TargetRef Kind must be one of: Gateway, HTTPRoute, or GRPCRoute`
ExpectedTargetRefGroupError = `TargetRef Group must be gateway.networking.k8s.io.`
)

const (
PolicyName = "test-policy"
TargetRefFormat = "targetRef-name-%d"
)

func TestClientSettingsPoliciesTargetRefKind(t *testing.T) {
tests := []struct {
policySpec ngfAPIv1alpha1.ClientSettingsPolicySpec
name string
wantErrors []string
}{
{
name: "Validate TargetRef of kind Gateway is allowed",
policySpec: ngfAPIv1alpha1.ClientSettingsPolicySpec{
TargetRef: gatewayv1alpha2.LocalPolicyTargetReference{
Kind: GatewayKind,
Group: GatewayGroup,
},
},
},
{
name: "Validate TargetRef of kind HTTPRoute is allowed",
policySpec: ngfAPIv1alpha1.ClientSettingsPolicySpec{
TargetRef: gatewayv1alpha2.LocalPolicyTargetReference{
Kind: HTTPRouteKind,
Group: GatewayGroup,
},
},
},
{
name: "Validate TargetRef of kind GRPCRoute is allowed",
policySpec: ngfAPIv1alpha1.ClientSettingsPolicySpec{
TargetRef: gatewayv1alpha2.LocalPolicyTargetReference{
Kind: GRPCRouteKind,
Group: GatewayGroup,
},
},
},
{
name: "Validate Invalid TargetRef Kind is not allowed",
wantErrors: []string{ExpectedTargetRefKindError},
policySpec: ngfAPIv1alpha1.ClientSettingsPolicySpec{
TargetRef: gatewayv1alpha2.LocalPolicyTargetReference{
Kind: InvalidKind,
Group: GatewayGroup,
},
},
},
{
name: "Validate TCPRoute TargetRef Kind is not allowed",
wantErrors: []string{ExpectedTargetRefKindError},
policySpec: ngfAPIv1alpha1.ClientSettingsPolicySpec{
TargetRef: gatewayv1alpha2.LocalPolicyTargetReference{
Kind: TCPRouteKind,
Group: GatewayGroup,
},
},
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
validateClientSettingsPolicy(t, tt)
})
}
}

func TestClientSettingsPoliciesTargetRefGroup(t *testing.T) {
tests := []struct {
policySpec ngfAPIv1alpha1.ClientSettingsPolicySpec
name string
wantErrors []string
}{
{
name: "Validate gateway.networking.k8s.io TargetRef Group is allowed",
policySpec: ngfAPIv1alpha1.ClientSettingsPolicySpec{
TargetRef: gatewayv1alpha2.LocalPolicyTargetReference{
Kind: GatewayKind,
Group: GatewayGroup,
},
},
},
{
name: "Validate invalid.networking.k8s.io TargetRef Group is not allowed",
wantErrors: []string{ExpectedTargetRefGroupError},
policySpec: ngfAPIv1alpha1.ClientSettingsPolicySpec{
TargetRef: gatewayv1alpha2.LocalPolicyTargetReference{
Kind: GatewayKind,
Group: InvalidGroup,
},
},
},
{
name: "Validate discovery.k8s.io/v1 TargetRef Group is not allowed",
wantErrors: []string{ExpectedTargetRefGroupError},
policySpec: ngfAPIv1alpha1.ClientSettingsPolicySpec{
TargetRef: gatewayv1alpha2.LocalPolicyTargetReference{
Kind: GatewayKind,
Group: DiscoveryGroup,
},
},
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
validateClientSettingsPolicy(t, tt)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also add t.Parallel() to each loop iteration.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, I've never seen that before.

Copy link
Author

@shaun-nx shaun-nx Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sjberman I made this change so that each policy created has a unique name during the parallel test runs. Even with this, the test runs are inconsistent. It looks like some of them are being created in parallel so fast that they still get the same names

I noticed it when I tried making new tests for the ClientKeepAliveTimeout validation

=== NAME  TestClientSettingsPoliciesKeepAliveTimeout/Validate_KeepAliveTimeout_is_not_set
    clientsettingspolicy_test.go:205: 
        Unexpected error:
            <*errors.StatusError | 0x14000516a00>: 
            clientsettingspolicies.gateway.nginx.org "test-policy-1753786044059919000" already exists
            {
                ErrStatus: {
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ListMeta: {
                        SelfLink: "",
                        ResourceVersion: "",
                        Continue: "",
                        RemainingItemCount: nil,
                    },
                    Status: "Failure",
                    Message: "clientsettingspolicies.gateway.nginx.org \"test-policy-1753786044059919000\" already exists",
                    Reason: "AlreadyExists",
                    Details: {
                        Name: "test-policy-1753786044059919000",
                        Group: "gateway.nginx.org",
                        Kind: "clientsettingspolicies",
                        UID: "",
                        Causes: nil,
                        RetryAfterSeconds: 0,
                    },
                    Code: 409,
                },
            }
        occurred
--- FAIL: TestClientSettingsPoliciesKeepAliveTimeout (0.00s)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The results are more consistent if I remove t.Parallel() from each loop of t.Run()

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, okay. Maybe because this is running on a live system. that's fine then.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, this is interesting.
The pre-commit linter errors if you don't put t.Parallel() in each t.Run() loop

golangci-lint-tests......................................................Failed
- hook id: golangci-lint-full
- exit code: 1

tests/cel/clientsettingspolicy_test.go:42:6: TestClientSettingsPoliciesTargetRefKind's subtests should call t.Parallel (tparallel)
func TestClientSettingsPoliciesTargetRefKind(t *testing.T) {
     ^
tests/cel/clientsettingspolicy_test.go:105:6: TestClientSettingsPoliciesTargetRefGroup's subtests should call t.Parallel (tparallel)
func TestClientSettingsPoliciesTargetRefGroup(t *testing.T) {
     ^
2 issues:
* tparallel: 2

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh fun, well then let's see about making this work. Maybe instead of using the unix time in the name we just use a random uuid.

Copy link
Author

@shaun-nx shaun-nx Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sjberman I swapped out the use of time.Now().UnixNano() in place of rand.Prime for generating unique policy names in each parallel test run. This seems to be more consistent from what I've tested.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would an index id or counter variable appended to the name work?

})
}
}

func validateClientSettingsPolicy(t *testing.T, tt struct {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something to just keep in mind for when we write more CEL tests for other policies, is that a lot of this function (and getKubernetesClient) will likely be re-used. So we can probably make it more generic in a separate file to handle all types. No action needed in this PR.

policySpec ngfAPIv1alpha1.ClientSettingsPolicySpec
name string
wantErrors []string
},
) {
t.Helper()

// Get Kubernetes client from test framework
// This should be set up by your test framework to connect to a real cluster
k8sClient := getKubernetesClient(t)

policySpec := tt.policySpec
policySpec.TargetRef.Name = gatewayv1alpha2.ObjectName(fmt.Sprintf(TargetRefFormat, time.Now().UnixNano()))

clientSettingsPolicy := &ngfAPIv1alpha1.ClientSettingsPolicy{
ObjectMeta: controllerruntime.ObjectMeta{
Name: PolicyName,
Namespace: "default",
},
Spec: policySpec,
}

err := k8sClient.Create(context.Background(), clientSettingsPolicy)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the k8sClient is using the actual Kubernetes API Server, I wonder if we need to add a timeout

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It probably is fine without it but just curious

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, when creating we typically do have a context to make sure it doesn't block there. Like these constants https://github.com/nginx/nginx-gateway-fabric/blob/main/tests/framework/timeout.go


// Clean up after test
defer func() {
_ = k8sClient.Delete(context.Background(), clientSettingsPolicy)
}()

// Check if we expected errors
if len(tt.wantErrors) == 0 {
if err != nil {
t.Errorf("expected no error but got: %v", err)
}
return
}

// We expected errors - validation should have failed
if err == nil {
t.Errorf("expected validation error but policy was accepted")
return
}

// Check that we got the expected error messages
var missingErrors []string
for _, wantError := range tt.wantErrors {
if !strings.Contains(err.Error(), wantError) {
missingErrors = append(missingErrors, wantError)
}
}
if len(missingErrors) != 0 {
t.Errorf("missing expected errors: %v, got: %v", missingErrors, err)
}
}

// getKubernetesClient returns a client connected to a real Kubernetes cluster.
func getKubernetesClient(t *testing.T) client.Client {
t.Helper()
// Use controller-runtime to get cluster connection
k8sConfig, err := controllerruntime.GetConfig()
if err != nil {
t.Skipf("Cannot connect to Kubernetes cluster: %v", err)
}

// Set up scheme with NGF types
scheme := runtime.NewScheme()
if err := ngfAPIv1alpha1.AddToScheme(scheme); err != nil {
t.Fatalf("Failed to add NGF schema: %v", err)
}

// Create client
k8sClient, err := client.New(k8sConfig, client.Options{Scheme: scheme})
if err != nil {
t.Skipf("Cannot create k8s client: %v", err)
}

return k8sClient
}
Loading