GRPCRoute with header-based match results in invalid client policy error on linkerd sidecar and result in connection errors and failures

[adleong]

Discussed in #14010

^{Originally posted by sina-grz May 10, 2025}

Linkerd Unable to Route gRPC Requests with Header-Based Matches

Issue Description

I've discovered that Linkerd appears unable to route gRPC requests when a GRPCRoute definition contains match sections based on headers.

Environment

• Kubernetes version: v1.33.0
• Gateway API version: v1.2.1
• Linkerd version: edge-25.5.2

How to Reproduce

I've developed three test services:

• service-a (client)
• service-b (server)
• service-c (second server)
  Here are the manifests:

manifest.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-a
  namespace: test
  labels:
    app: service-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-a
  template:
    metadata:
      labels:
        app: service-a
    spec:
      containers:
      - name: service-a
        image: sinagdev/test:service-a 
        ports:
        - containerPort: 8001 
        env:
        - name: GRPC_SERVER
          value: "service-b:6000" # Service B's gRPC endpoint
      imagePullSecrets:
      - name: reg-pegah-credit

---
apiVersion: v1
kind: Service
metadata:
  name: service-a
  namespace: test
spec:
  selector:
    app: service-a
  ports:
  - name: metrics
    protocol: TCP
    port: 8001
    targetPort: 8001
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-b
  namespace: test
  labels:
    app: service-b
spec:
  replicas: 3
  selector:
    matchLabels:
      app: service-b
  template:
    metadata:
      labels:
        app: service-b
    spec:
      containers:
      - name: service-b
        image: sinagdev/test:service-b 
        ports:
        - containerPort: 6000 # gRPC port
        - containerPort: 8000 # Metrics port
        env:
        - name: PYTHONUNBUFFERED
          value: "1"
      imagePullSecrets:
      - name: reg-pegah-credit
---
apiVersion: v1
kind: Service
metadata:
  name: service-b
  namespace: test
spec:
  selector:
    app: service-b
  ports:
  - name: grpc
    protocol: TCP
    port: 6000
    targetPort: 6000
  - name: metrics
    protocol: TCP
    port: 8000
    targetPort: 8000
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-c
  namespace: test
  labels:
    app: service-c
  annotations:
    linkerd.io/inject: enabled
spec:
  replicas: 2
  selector:
    matchLabels:
      app: service-c
  template:
    metadata: 
      labels:
        app: service-c
      annotations:
        linkerd.io/inject: enabled
    spec:
      containers:
      - name: service-c
        image:  sinagdev/test:service-b # Using same image as service-b
        ports:
        - containerPort: 6000 # gRPC port
        - containerPort: 8000 # Metrics port
        env:
        - name: PYTHONUNBUFFERED
          value: "1"
      imagePullSecrets:
      - name: reg-pegah-credit
---
apiVersion: v1
kind: Service
metadata:
  namespace: test
  name: service-c
spec:
  selector:
    app: service-c
  ports:
  - name: grpc
    protocol: TCP
    port: 6000
    targetPort: 6000
  - name: metrics
    protocol: TCP
    port: 8000
    targetPort: 8000
  type: ClusterIP

---
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: service-split-route
  namespace: test
spec:
  parentRefs:
  - name: service-b
    kind: Service            
    group: ""              
    port: 6000
  rules:
  - matches:
    - headers:
      - name: "session-id"
        value: "user-1"
      rpc:
        methodPrefix: "/"
    backendRefs:
    - name: service-c
      group: ""
      kind: Service
      port: 6000
  - backendRefs:
    - name: service-b
      group: ""
      kind: Service
      port: 6000

The Docker images are available on Docker Hub (sinagdev/test).

Error Symptoms

When trying to route based on headers, my gRPC client crashes with:

Traceback (most recent call last):
  File "/app/serviceA.py", line 30, in <module>
    run()
  File "/app/serviceA.py", line 23, in run
    response = stub.ProcessSession(request, metadata=metadata)
  File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 1181, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "unexpected error"
        debug_error_string = "UNKNOWN:Error received from peer {created_time:"2025-05-10T19:01:13.827742158+00:00", grpc_status:13, grpc_message:"unexpected error"}"
>

Linkerd Logs

The Linkerd proxy logs show:

[     6.945562s]  WARN ThreadId(01) outbound:proxy{addr=172.20.190.246:6000}: linkerd_app_outbound::policy::api: Client policy misconfigured error=invalid gRPC route: invalid route match: missing RPC match
[     6.946248s]  INFO ThreadId(01) outbound:proxy{addr=172.20.190.246:6000}:rescue{client.addr=172.17.1.3:37990}: linkerd_app_core::errors::respond: gRPC request failed error=logical service 172.20.190.246:6000: route default.invalid: invalid client policy: invalid client policy configuration error.sources=[route default.invalid: invalid client policy: invalid client policy configuration, invalid client policy: invalid client policy configuration]

The key error message is:

invalid gRPC route: invalid route match: missing RPC match

Configuration

GRPCRoute

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: service-split-route
  namespace: test
spec:
  parentRefs:
  - name: service-b
    kind: Service            
    group: ""              
    port: 6000
  rules:
  - matches:
    - headers:
      - name: "session-id"
        value: "user-1"
    backendRefs:
    - name: service-c
      group: ""
      kind: Service
      port: 6000
  - backendRefs:
    - name: service-b
      group: ""
      kind: Service
      port: 6000

The GRPCRoute is successfully created and shows as "Accepted" and "ResolvedRefs: True" in status:

Status:
  Parents:
    Conditions:
      Last Transition Time:  2025-05-10T18:39:58Z
      Message:               
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2025-05-10T18:39:58Z
      Message:               
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         linkerd.io/policy-controller

Expected Behavior

The GRPCRoute should route traffic with the "session-id: user-1" header to service-c, and all other traffic to service-b.

Fix option

To resolve the 'missing RPC match' error, I added the required 'service' and 'method' fields in the match section.

- matches:
    - headers:
        - name: "session-id"
          value: "user-1"
      method:
        service: "SessionService"  
        method: "ProcessSession"

However, this approach requires explicitly specifying the gRPC service and method, which is inconvenient when trying to match all requests. Additionally, Linkerd documentation does not mention this strict requirement for GRPCRoute."

Question

Why Does Linkerd require additional RPC match configuration for header-based routing in GRPCRoutes? The error suggests it's looking for an RPC match, but I'm only trying to use header-based matching as defined in the Gateway API spec.