𝗔𝗜 𝗚𝗮𝘁𝗲𝘄𝗮𝘆 𝗚𝘂𝗮𝗿𝗱𝗿𝗮𝗶𝗹𝘀 𝘄𝗶𝘁𝗵 𝗔𝗪𝗦 𝗕𝗲𝗱𝗿𝗼𝗰𝗸 𝗮𝗻𝗱 𝗞𝗼𝗻𝗴
You deployed an AI Gateway. Traffic flows. The LLM responds. Everything looks good.
Then a user sends a prompt to ignore all previous instructions. Or a user pastes a credit card number into your chatbot. Or a user asks your bot for financial advice in a regulated industry.
Sending traffic to an LLM is easy. Controlling that traffic is hard. You need safety policies.
You can use AWS Bedrock Guardrails with a Kong AI Gateway on Kubernetes. This setup uses the ai-aws-guardrails plugin. Every request and response passes through a policy layer. The gateway blocks violations before they reach your application code.
You can configure four guardrail types:
- Content Filters: Block hate, violence, or insults.
- Prompt Attack Protection: Block jailbreaks and injection attempts.
- PII Protection: Block emails, credit cards, and passwords.
- Denied Topics: Block specific subjects like investment advice.
For PII, you have two choices. Use BLOCK to stop the request. Use ANONYMIZE to replace sensitive data with placeholders. This helps in healthcare when you need context without raw data.
In your Kong configuration, set guarding_mode to BOTH. The default setting only checks inputs. Setting it to BOTH checks both the user prompt and the model response. This prevents a hijacked model from sending harmful data back to your user.
Safety is only one part of a good gateway. You should also use:
- Rate limiting to control costs.
- Semantic caching to reduce repeated queries.
- JWT auth to secure your routes.
Read the full guide on Hashnode for the complete JSON definitions and setup steps.
Optional learning community: https://t.me/GyaanSetuAi