KServe

Use KServe with agentgateway.

KServe is a Kubernetes-native platform for serving machine learning models. KServe can create an InferencePool and run the llm-d Router Endpoint Picker (EPP) for an LLMInferenceService. Agentgateway routes requests to that pool through the Gateway API Inference Extension.

Use the standard InferencePool backend when you need inference-aware endpoint selection. Add an AgentgatewayBackend only when you also need agentgateway LLM processing, such as token-based rate limiting, guardrails, or LLM observability.

    graph LR
    Client --> Gateway
    Gateway --> HTTPRoute
    HTTPRoute --> InferencePool
    InferencePool --> EPP["KServe-managed llm-d Router EPP"]
    EPP --> ModelServer["model server"]

KServe also maintains an LLMInferenceService with agentgateway guide. For production llm-d deployment patterns, see the llm-d gateway documentation and agentgateway integration.

Before you begin

This guide is tested with the following versions.

Component	Version
Kubernetes Gateway API	v1.6.0
Gateway API Inference Extension	v1.5.0
agentgateway	v1.4.1
KServe	v0.20.0-rc0

You need a Kubernetes cluster, Helm, and kubectl.

Install the APIs and controllers

Install the Kubernetes Gateway API Custom Resource Definitions (CRDs).

kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.0/standard-install.yaml

Install cert-manager, which KServe uses for webhook certificates.

kubectl apply -f \
  https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml

kubectl wait --for=condition=available deployment --all \
  --namespace cert-manager \
  --timeout=180s

Install KServe and configure its LLMInferenceService controller to use a shared agentgateway Gateway. The KServe chart also installs the transitional Inference Extension CRDs that its migration controller requires.

kubectl create namespace kserve

helm upgrade -i kserve-llmisvc-crd \
  oci://ghcr.io/kserve/charts/kserve-llmisvc-crd \
  --version v0.20.0-rc0 \
  --namespace kserve

helm upgrade -i kserve-llmisvc-resources \
  oci://ghcr.io/kserve/charts/kserve-llmisvc-resources \
  --version v0.20.0-rc0 \
  --namespace kserve \
  --set kserve.controller.deploymentMode=Standard \
  --set kserve.controller.gateway.ingressGateway.enableGatewayApi=true \
  --set kserve.controller.gateway.ingressGateway.createGateway=false \
  --set kserve.controller.gateway.ingressGateway.kserveGateway=kserve/kserve-ingress-gateway \
  --set kserve.controller.gateway.ingressGateway.className=agentgateway \
  --set kserve.controller.gateway.disableIstioVirtualHost=true \
  --set kserve.controller.gateway.disableIngressCreation=false \
  --set kserve.controller.knativeAddressableResolver.enabled=false \
  --set kserve.controller.gateway.localGateway.gateway="" \
  --set kserve.controller.gateway.localGateway.gatewayService=""

kubectl rollout status deployment/llmisvc-controller-manager \
  --namespace kserve \
  --timeout=240s

helm upgrade -i kserve-runtime-configs \
  oci://ghcr.io/kserve/charts/kserve-runtime-configs \
  --version v0.20.0-rc0 \
  --namespace kserve \
  --set kserve.llmisvcConfigs.enabled=true

Apply the final GAIE v1.5.0 CRD bundle. Applying it after the KServe chart updates the stable API definitions while retaining the transitional CRDs that KServe needs during the migration.
```
kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.5.0/manifests.yaml
```

Install agentgateway with Inference Extension support. Installing it after the GAIE CRDs ensures that the controller discovers InferencePool.

helm upgrade -i agentgateway-crds \
  oci://cr.agentgateway.dev/charts/agentgateway-crds \
  --create-namespace \
  --namespace agentgateway-system \
  --version v1.4.1

helm upgrade -i agentgateway \
  oci://cr.agentgateway.dev/charts/agentgateway \
  --namespace agentgateway-system \
  --version v1.4.1 \
  --set inferenceExtension.enabled=true

Create an agentgateway Gateway. KServe attaches generated HTTPRoute resources from model namespaces to this shared Gateway.

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kserve-ingress-gateway
  namespace: kserve
spec:
  gatewayClassName: agentgateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
  infrastructure:
    labels:
      serving.kserve.io/gateway: kserve-ingress-gateway
EOF

kubectl wait --for=condition=Programmed \
  gateway/kserve-ingress-gateway \
  --namespace kserve \
  --timeout=180s

Do not install the llm-d Router Helm chart in this workflow. KServe owns the router deployment and uses the llm-d endpoint-picker image from its runtime configuration.

Deploy a simulated LLM

Create a namespace for the model.
```
kubectl create namespace kserve-test
```

Deploy an LLMInferenceService with its managed scheduler enabled. This example uses llm-d-inference-sim instead of downloading model weights or requiring GPUs.

kubectl apply -f - <<EOF
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: mock-llm
  namespace: kserve-test
spec:
  model:
    name: mock-llm
    uri: hf://mock/mock-llm
  replicas: 1
  storageInitializer:
    enabled: false
  router:
    route: {}
    scheduler: {}
  template:
    containers:
      - name: main
        image: ghcr.io/llm-d/llm-d-inference-sim:v0.9.0-rc3
        command:
          - /app/llm-d-inference-sim
        args:
          - --model
          - mock-llm
          - --port
          - "8000"
          - --mode
          - echo
        ports:
          - name: http
            containerPort: 8000
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
EOF

Wait for the service and generated routing resources.

kubectl wait --for=condition=Ready \
  llminferenceservice/mock-llm \
  --namespace kserve-test \
  --timeout=300s

kubectl get inferencepool mock-llm-inference-pool \
  --namespace kserve-test
kubectl get httproute mock-llm-kserve-route \
  --namespace kserve-test

Confirm that the generated route uses the standard InferencePool backend and that KServe runs the migrated llm-d Router image.

kubectl get httproute mock-llm-kserve-route \
  --namespace kserve-test \
  -o jsonpath='{.spec.rules[?(@.name=="v1-chat-completions-path")].backendRefs[0]}{"\n"}'

kubectl get deployment mock-llm-kserve-router-scheduler \
  --namespace kserve-test \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="main")].image}{"\n"}'

The backend is mock-llm-inference-pool. The EPP image repository is ghcr.io/llm-d/llm-d-router-endpoint-picker.

Test the standard InferencePool backend

Port-forward the Gateway service.

kubectl port-forward \
  --namespace kserve \
  service/kserve-ingress-gateway \
  8080:80

In another terminal, send a request through the path generated by KServe.

curl -i http://localhost:8080/kserve-test/mock-llm/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mock-llm",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

The response has an HTTP 200 OK status. No AgentgatewayBackend is required for this path.

Optional: Apply an AI policy

Token-based rate limiting and other supported AI policies require agentgateway to parse the LLM request or response. To apply these policies, create an AgentgatewayBackend that wraps the same InferencePool, then override the KServe route to use that backend.

Create the backend.

kubectl apply -f - <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: mock-llm-backend
  namespace: kserve-test
spec:
  ai:
    provider:
      custom:
        backendRef:
          group: inference.networking.k8s.io
          kind: InferencePool
          name: mock-llm-inference-pool
        model: mock-llm
        formats:
          - type: Completions
            path: /v1/chat/completions
EOF

Update the LLMInferenceService route. The managed scheduler remains enabled, so KServe continues to own the InferencePool and llm-d Router EPP.

kubectl patch llminferenceservice mock-llm \
  --namespace kserve-test \
  --type merge \
  --patch '{
    "spec": {
      "router": {
        "route": {
          "http": {
            "spec": {
              "parentRefs": [{
                "group": "gateway.networking.k8s.io",
                "kind": "Gateway",
                "name": "kserve-ingress-gateway",
                "namespace": "kserve"
              }],
              "rules": [{
                "backendRefs": [{
                  "group": "agentgateway.dev",
                  "kind": "AgentgatewayBackend",
                  "name": "mock-llm-backend"
                }],
                "matches": [{
                  "path": {
                    "type": "PathPrefix",
                    "value": "/v1/chat/completions"
                  }
                }],
                "timeouts": {
                  "backendRequest": "0s",
                  "request": "0s"
                }
              }]
            }
          }
        },
        "scheduler": {}
      }
    }
  }'

Apply a token-based rate limit to the generated route.

kubectl apply -f - <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: mock-llm-token-budget
  namespace: kserve-test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: mock-llm-kserve-route
  traffic:
    rateLimit:
      local:
        - tokens: 10
          unit: Minutes
EOF

Send requests to the policy-enabled route.

for i in $(seq 1 5); do
  curl -s -o /dev/null -w "%{http_code}\n" \
    http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "mock-llm",
      "messages": [{"role": "user", "content": "Hello"}]
    }'
done

Successful requests return 200. After the response token usage consumes the budget, agentgateway returns 429 Too Many Requests.

Cleanup

Remove the resources created in this guide.

kubectl delete namespace kserve-test
helm uninstall kserve-runtime-configs --namespace kserve
helm uninstall kserve-llmisvc-resources --namespace kserve
helm uninstall kserve-llmisvc-crd --namespace kserve
kubectl delete gateway kserve-ingress-gateway --namespace kserve
kubectl delete namespace kserve
helm uninstall agentgateway --namespace agentgateway-system
helm uninstall agentgateway-crds --namespace agentgateway-system

Argo Rollouts LLM client configuration

Was this page helpful?