NeMo Guardrails × OSSM TrafficExtension

What this is

A proof-of-concept that applies NeMo Guardrails to AI workloads on RHOAI by adding a single Kubernetes pod label — no application code changes, no gateway dependency.

oc label pod <predictor-pod> -n <namespace> guardrails.trustyai.io/config=pii

An Istio TrafficExtension with a Wasm plugin intercepts every inbound request and outbound response, calls NeMo's /v1/guardrail/checks endpoint, and blocks flagged content with HTTP 403 before it reaches the model or the client.

Multiple pods can each enforce a different guardrail policy simultaneously, all served by a single NeMo instance. Policy routing uses guardrails.config_id (single config) or guardrails.config_ids (merged union of configs).

---

Validated results (2026-08-25, OCP 4.20.32 / OSSM 3.4.1 / Istio 1.30.3)

Single-policy — glm4-9b-chat (vLLM InferenceService)

Test	Result
Clean request	200 ✅
Email PII in request	403 blocked_by_guardrail ✅
Credential (api_key) in request	403 ✅
SSN in request (regex)	403 ✅
Clean model response	200 ✅
Email PII echoed in response	403 ✅

Multi-policy — 4 policies, 1 NeMo pod, 13/13 tests pass

Label	Policy	Email	SSN	api_key	Clean
config=pii	pii	403 ✅	403 ✅	403 ✅	200 ✅
config=ssn-only	ssn-only	200 ✅	403 ✅	200 ✅	200 ✅
config=credentials	credentials	200 ✅	200 ✅	403 ✅	200 ✅
config=ssn-credentials	ssn-only + credentials merged	200 ✅	403 ✅	403 ✅	200 ✅

Latency impact

NeMo adds two synchronous callouts per transaction: one for the request body, one for the response body (~38ms each at rest, single-node cluster).

Scenario	Baseline avg	Guardrailed avg	Overhead
max_tokens=5 (minimal inference)	260ms	337ms	+77ms (+30%)
max_tokens=100 (realistic)	6927ms	6348ms	within variance (±600ms)

At real inference lengths (100+ tokens), model inference variance (~±600ms) exceeds the NeMo overhead (~77ms). The guardrail cost is effectively invisible in production.

Regex-only policies (e.g. credentials) are 39ms faster than Presidio NER policies — the ML entity scanner is skipped entirely.

---

Architecture

Inference request
  │
  ▼
Envoy sidecar (OSSM 3.4.1 / Istio 1.30.3) on model pod
  └── TrafficExtension: nemo-<policy>-guard (phase: AUTHZ, Wasm)
       │
       ├── OnHttpRequestBody
       │    ├── inject guardrails.config_id into NeMo payload
       │    ├── POST /v1/guardrail/checks → nemo-pii.guardrails:80
       │    ├── "success"  → forward to model ──────────────────► Model
       │    └── "blocked"  → return 403 ──────────────────────► Client
       │
       ├── OnHttpResponseBody
       │    ├── inject guardrails.config_id into NeMo payload (role: assistant)
       │    ├── POST /v1/guardrail/checks → nemo-pii.guardrails:80
       │    ├── "success"  → forward response to client ────────► Client
       │    └── "blocked"  → return 403 ──────────────────────► Client
       │
       └── NeMo unreachable → return 403 (fail-closed)

Multiple TrafficExtensions → same nemo-pii Service → different configs via config_id

No AI Gateway involvement. The RHOAI AI Gateway continues routing inference traffic normally; enforcement happens at the sidecar level.

---

Prerequisites

Requirement	Check
OCP 4.20+	oc version
OSSM 3 Operator (Sail Operator)	oc get csv -n openshift-operators | grep servicemeshoperator3
TrustyAI Operator	oc get crd nemoguardrails.trustyai.opendatahub.io
oc CLI, cluster-admin role	oc whoami
Go 1.24+ (Wasm build)	go version

---

Quick start

git clone https://github.com/williamcaban/istio-guardrails
cd istio-guardrails

./scripts/00-prereqs.sh          # deploy OSSM control plane + label namespaces
./scripts/01-deploy-nemo.sh      # deploy NeMo + validate endpoint
./scripts/02-get-cluster-name.sh # confirm Envoy cluster name
./scripts/03-deploy-wasm.sh      # build Wasm image, deploy TrafficExtension, run tests

---

Manifests

deploy/
├── ossm/
│   ├── 00-istio.yaml                              # Istio CR (v1.30.3, discoverySelectors)
│   ├── 01-istiocni.yaml                           # IstioCNI CR
│   ├── wasm/
│   │   ├── traffic-extension-wasm.yaml            # pii policy         (config_id: pii)
│   │   ├── traffic-extension-ssn-only.yaml        # ssn-only policy    (config_id: ssn-only)
│   │   ├── traffic-extension-credentials.yaml     # credentials policy (config_id: credentials)
│   │   └── traffic-extension-ssn-credentials.yaml # merged policy      (config_ids: [ssn-only, credentials])
│   └── glm4-baseline/
│       └── traffic-extension-wasm.yaml            # TrafficExtension for glm4-9b-chat namespace
├── nemo/
│   ├── 00-namespace.yaml                          # guardrails namespace
│   ├── 01-configmap-pii.yaml                      # PII + regex rails (input + output)
│   ├── 01-configmap-ssn-only.yaml                 # SSN/ITIN rails only
│   ├── 01-configmap-credentials.yaml              # credential regex rails only
│   ├── 02-nemoguardrails-cr.yaml                  # single NeMo CR hosting all three configs
│   └── 03-authorizationpolicy.yaml                # restricts NeMo to mesh-internal access
└── app-ns/
    ├── 00-namespace.yaml                          # app-ns namespace
    ├── mock-model.yaml                            # Python HTTP echo server (test target)
    ├── test-app.yaml                              # curl probe pod
    ├── wasm-server.yaml                           # HTTP server delivering plugin.wasm
    └── pii-model.yaml                             # test fixture — always returns PII in response

---

Step-by-step deployment

Step 1 — Deploy OSSM control plane

Critical: create namespaces before applying the Istio CR. The Sail Operator returns ReconcileError: namespace does not exist if absent.

oc create namespace istio-system
oc create namespace istio-cni

oc apply -f deploy/ossm/00-istio.yaml
oc apply -f deploy/ossm/01-istiocni.yaml

oc wait --for=condition=Ready istios/default --timeout=5m
oc wait --for=condition=Ready IstioCNI/default --timeout=5m

Step 2 — Create and label namespaces

oc create namespace guardrails
oc create namespace app-ns

for ns in istio-system istio-cni guardrails app-ns; do
  oc label namespace $ns istio-discovery=enabled --overwrite
done

# Sidecar injection for app-ns only (NeMo in guardrails does NOT get a sidecar)
oc label namespace app-ns istio-injection=enabled --overwrite

If IstioRevision name differs from default (check with oc get istiorevisions), use istio.io/rev=<revision-name> instead of istio-injection=enabled.

Step 3 — Deploy NeMo Guardrails

# Apply all ConfigMaps (one per policy)
oc apply -f deploy/nemo/01-configmap-pii.yaml
oc apply -f deploy/nemo/01-configmap-ssn-only.yaml
oc apply -f deploy/nemo/01-configmap-credentials.yaml

# Deploy the single NeMo CR (hosts all three configs in one pod)
oc apply -f deploy/nemo/02-nemoguardrails-cr.yaml

oc wait nemoguardrails/nemo-pii -n guardrails \
  --for=jsonpath='{.status.phase}'=Ready --timeout=5m

oc apply -f deploy/nemo/03-authorizationpolicy.yaml

Step 4 — Validate NeMo endpoint and config routing

oc -n guardrails port-forward svc/nemo-pii 18000:80 &

# List available configs
curl -s http://localhost:18000/v1/rails/configs
# Expected: [{"id":"pii"},{"id":"ssn-only"},{"id":"credentials"}]

# Test config routing (config_id must be inside guardrails object)
curl -s http://localhost:18000/v1/guardrail/checks \
  -H "Content-Type: application/json" \
  -d '{"guardrails":{"config_id":"ssn-only"},"model":"","messages":[{"role":"user","content":"email bob@test.com"}]}'
# Expected: {"status":"success",...}  (ssn-only does not block email)

curl -s http://localhost:18000/v1/guardrail/checks \
  -H "Content-Type: application/json" \
  -d '{"guardrails":{"config_id":"ssn-only"},"model":"","messages":[{"role":"user","content":"SSN 123-45-6789"}]}'
# Expected: {"status":"blocked",...}

kill %1

Step 5 — Confirm Envoy cluster name

oc apply -f deploy/app-ns/mock-model.yaml
MOCK_POD=$(oc get pod -n app-ns -l app=mock-model -o jsonpath='{.items[0].metadata.name}')

oc exec -n app-ns "$MOCK_POD" -c istio-proxy -- \
  pilot-agent request GET /clusters | grep nemo-pii | head -1
# Expected: outbound|80||nemo-pii.guardrails.svc.cluster.local

Step 6 — Build and deploy the Wasm TrafficExtension

# Build and push Wasm image to internal OpenShift registry (default)
oc new-build --strategy=docker --binary --name=nemo-wasm-guard -n app-ns
cd wasm && oc start-build nemo-wasm-guard --from-dir=. --follow -n app-ns && cd ..

# Deploy wasm-server (HTTP server for binary delivery — internal OCI TLS workaround)
oc apply -f deploy/app-ns/wasm-server.yaml

# Get cluster-built SHA256 and update TrafficExtension
SHA=$(oc exec -n app-ns \
  $(oc get pod -n app-ns -l app=wasm-server -o jsonpath='{.items[0].metadata.name}') \
  -- sh -c 'sha256sum /serve/plugin.wasm' | awk '{print $1}')

for f in deploy/ossm/wasm/traffic-extension-*.yaml; do
  sed -i "s|sha256: .*|sha256: \"$SHA\"|" "$f"
done

# Apply all TrafficExtensions
oc apply -f deploy/ossm/wasm/

Step 7 — Test

oc apply -f deploy/app-ns/test-app.yaml
PROBE=$(oc get pod -n app-ns -l app=test-app -o jsonpath='{.items[0].metadata.name}')
MODEL=http://mock-model.app-ns.svc.cluster.local:8080/v1/chat/completions

# Clean — expect 200
oc exec -n app-ns "$PROBE" -- curl -s -w "\nHTTP:%{http_code}" -X POST "$MODEL" \
  -H "Content-Type: application/json" \
  -d '{"model":"test","messages":[{"role":"user","content":"Explain Kubernetes"}]}'

# Email PII — expect 403
oc exec -n app-ns "$PROBE" -- curl -s -w "\nHTTP:%{http_code}" -X POST "$MODEL" \
  -H "Content-Type: application/json" \
  -d '{"model":"test","messages":[{"role":"user","content":"My email is test@example.com"}]}'

---

Multiple pods with distinct policies

Multiple pods can each enforce a different guardrail policy simultaneously, all routed to a single NeMo instance via guardrails.config_id in the Wasm payload.

Architecture — single NeMo pod, multiple policies

Pod A  (label: config=pii)
  └── TrafficExtension: nemo-full-guard
       └── pluginConfig.nemoConfigId: "pii"
            └── POST {"guardrails":{"config_id":"pii"}, ...} → nemo-pii:80

Pod B  (label: config=ssn-only)
  └── TrafficExtension: nemo-ssn-only-guard
       └── pluginConfig.nemoConfigId: "ssn-only"
            └── POST {"guardrails":{"config_id":"ssn-only"}, ...} → nemo-pii:80

Pod C  (label: config=credentials)
  └── TrafficExtension: nemo-credentials-guard
       └── pluginConfig.nemoConfigId: "credentials"
            └── POST {"guardrails":{"config_id":"credentials"}, ...} → nemo-pii:80

Pod D  (label: config=ssn-credentials)
  └── TrafficExtension: nemo-ssn-credentials-guard
       └── pluginConfig.nemoConfigIds: ["ssn-only","credentials"]
            └── POST {"guardrails":{"config_ids":["ssn-only","credentials"]}, ...} → nemo-pii:80

                                               │
                                               ▼
                                    Single nemo-pii pod
                                    /app/config/pii/
                                    /app/config/ssn-only/
                                    /app/config/credentials/

Defined policies

Label	pluginConfig field	Blocks	Passes
config=pii	nemoConfigId: "pii"	Email, phone, SSN, credentials	clean text
config=ssn-only	nemoConfigId: "ssn-only"	SSN + US_ITIN	email, phone, credentials
config=credentials	nemoConfigId: "credentials"	Credential/token patterns	email, SSN, phone
config=ssn-credentials	nemoConfigIds: ["ssn-only","credentials"]	SSN + credentials (union)	email, phone

Merged policies via guardrails.config_ids (plural)

config_ids merges multiple configs into a single check. NeMo evaluates a union of all rails — if any rail from any config fires, the request is blocked.

# TrafficExtension pluginConfig
nemoConfigIds: ["ssn-only", "credentials"]

Input	ssn-only	credentials	ssn-only + credentials
email bob@test.com	200	200	200 (neither catches it)
SSN 123-45-6789	403	200	403 (ssn-only rail fires)
api_key abc123	200	403	403 (credentials rail fires)
SSN + api_key	403	403	403 (first rail wins)

NeMo API constraint: config_id and config_ids are mutually exclusive — sending both returns HTTP 422. config_ids takes priority if both are set in pluginConfig (the Wasm checks NemoConfigIds first).

Adding a new policy

# 1. Create a ConfigMap with config.yaml + rails.co
oc apply -f deploy/nemo/01-configmap-<policy>.yaml

# 2. Add the new config to the NemoGuardrails CR's nemoConfigs array and apply
oc apply -f deploy/nemo/02-nemoguardrails-cr.yaml

# 3. Restart NeMo to mount the new config directory
oc rollout restart deployment/nemo-pii -n guardrails

# 4. Create a TrafficExtension with nemoConfigId pointing to the new config
oc apply -f deploy/ossm/wasm/traffic-extension-<policy>.yaml

# 5. Label the target pod
oc label pod <pod-name> -n <namespace> guardrails.trustyai.io/config=<policy>

NeMo API — config_id routing rules

Rule	Detail
Field nesting	config_id MUST be inside {"guardrails":{"config_id":"name"}, ...}
Top-level ignored	{"config_id":"name", ...} is silently ignored — no error, no routing
URL routing absent	No query params (?config_id=) or path routing (/checks/ssn-only)
List configs	GET /v1/rails/configs returns all available config IDs
Latency	Presidio NER adds ~38ms; regex-only configs are ~39ms faster

---

Applying to a real InferenceService (Option B)

The mock-model stands in for any RHOAI InferenceService pod. For a real model:

# 1. Enable mesh on the model namespace
oc label namespace <model-ns> istio-discovery=enabled istio-injection=enabled

# 2. Restart predictor pod to inject sidecar
#    On GPU clusters: delete the pod (rolling restart fails — insufficient GPU for two pods)
oc delete pod <predictor-pod> -n <model-ns>

# 3. Label the new running pod (KServe owns the Deployment — patching it is reverted)
oc label pod <new-predictor-pod> -n <model-ns> guardrails.trustyai.io/config=pii

# 4. Create a TrafficExtension in the model namespace
#    Copy and adapt one of the wasm/ TEs, set the namespace, apply
oc apply -f deploy/ossm/glm4-baseline/traffic-extension-wasm.yaml  # working example

See deploy/ossm/glm4-baseline/ for a validated example targeting glm4-9b-chat.

---

Wasm binary delivery — internal registry workaround

OpenShift's internal registry uses a self-signed certificate that istiod cannot trust for OCI pulls. The workaround: serve plugin.wasm over plain HTTP from a sidecar-free pod (wasm-server). istiod fetches via HTTP and delivers the binary to proxies via xDS.

Cache invalidation after rebuild:

1. Restart wasm-server to load the new image
2. Get the cluster-built SHA256: oc exec -n app-ns <wasm-server-pod> -- sha256sum /serve/plugin.wasm
3. Bump ?v=N in all TrafficExtension YAML files and update sha256:
4. Delete + reapply the TrafficExtensions to force istiod to re-fetch

SHA256 must be from the cluster-built binary — local macOS/Linux builds use a different Go toolchain and produce a different binary with a different hash.

---

Configuration reference

NeMo config.yaml (key requirements)

models: []         # prevents NeMo attempting LLM generation — required without this causes 504
rails:
  config:
    sensitive_data_detection:
      input:
        entities: [EMAIL_ADDRESS, PHONE_NUMBER, US_SSN, CREDIT_CARD, PERSON]
        score_threshold: 0.5
      output:
        entities: [EMAIL_ADDRESS, PHONE_NUMBER, US_SSN, CREDIT_CARD, PERSON]
        score_threshold: 0.5
    regex_detection:
      input:
        patterns: ["\\b(password|secret|api[_-]?key|token)\\b", "\\d{3}-\\d{2}-\\d{4}"]
      output:
        patterns: ["\\b(password|secret|api[_-]?key|token)\\b", "\\d{3}-\\d{2}-\\d{4}"]
  input:
    flows:
      - detect sensitive data on input
      - regex check input
  output:
    flows:
      - detect sensitive data on output
      - regex check output

TrafficExtension pluginConfig fields

Field	Type	Purpose
nemoCluster	string	Envoy cluster name for NeMo service
nemoHost	string	NeMo service hostname (:authority header)
nemoPath	string	/v1/guardrail/checks (RHOAI 3.5); /v1/checks (RHOAI 3.6+)
nemoConfigId	string	Route to a single named config
nemoConfigIds	list	Merge multiple configs (union of rails); takes priority over nemoConfigId
timeoutMs	int	NeMo callout timeout in ms (minimum 3000 — cold-start ~1.2s)
failMode	string	closed (default, recommended) or open

---

Troubleshooting

Symptom	Cause	Fix
ReconcileError: namespace does not exist	Istio CR applied before istio-system created	Create namespace first
504 on clean requests	NeMo trying LLM generation	Add models: [] to config.yaml; restart NeMo
All requests → 403 RBAC: access denied	Wasm SHA256 mismatch — istiod applied deny-all	Recompute SHA256 from cluster-built binary and reapply
All requests → timeout	OnHttpResponseHeaders double-pause (old binary)	Redeploy with current wasm/main.go
TrafficExtension has no effect	Wrong istiod revision on namespace	Verify istio-injection=enabled or istio.io/rev=default
Wrong Envoy cluster name	Using container port 8000 instead of service port 80	Confirm with pilot-agent request GET /clusters | grep nemo-pii
Wasm not re-fetched after rebuild	istiod cache hit on same URL	Bump ?v=N param, update sha256, delete + reapply TE
config_id routing ignored	config_id at request body root instead of inside guardrails	Use {"guardrails":{"config_id":"name"}, ...}
KServe reverts deployment label	KServe operator owns the Deployment	Label the running pod directly, not the Deployment

---

Teardown

oc delete trafficextension -n app-ns --all --ignore-not-found
oc delete trafficextension nemo-full-guard -n glm4-baseline --ignore-not-found
oc delete -f deploy/app-ns/ --ignore-not-found
oc delete -f deploy/nemo/ --ignore-not-found
oc delete -f deploy/ossm/01-istiocni.yaml --ignore-not-found
oc delete -f deploy/ossm/00-istio.yaml --ignore-not-found
oc delete namespace guardrails app-ns istio-system istio-cni --ignore-not-found

---

Known limitations

Limitation	Detail
Response status on fixed Content-Length	When a server sends Content-Length, 200 status reaches the client before OnHttpResponseBody fires. Blocking causes connection reset (200 + empty body). vLLM uses chunked encoding → clean 403.
Phone number detection	Presidio requires international format (+1 xxx) — domestic 555-xxx-xxxx scores below the 0.5 confidence threshold
Streaming (SSE)	Body buffering (endOfStream=true) is incompatible with SSE streaming responses
KServe label persistence	Pod label does not survive pod restarts; persistence requires InferenceService CR predictor.labels (verify per RHOAI version)
Wasm binary cache	istiod caches by URL — bump ?v=N and sha256 on every rebuild
config_id nesting	Must be {"guardrails":{"config_id":"name"}, ...} — top-level field is silently ignored
config_id vs config_ids	Mutually exclusive — sending both returns HTTP 422

---

Ambient mode (future)

Requires routingViaHost: true in OVN-Kubernetes (currently false). Once enabled, ZTunnel + Waypoint proxies handle L4/L7 without per-pod sidecar injection.