Vulnerable AI Systems: Real Data, Responsible Design

29% of attacks bypass the security filters of the most widely used LLMs in production. It's not a bug. It's the nature of the system. LLMs are stochastic processes trained on human language—the most flexible, ambiguous, and manipulable medium that exists. This talk presents the results of llm-break-bench: 3,360 adversarial tests on GPT-4o, Claude, Gemini, Grok, and DeepSeek using MLCommons AI Safety v0.5 and OWASP LLM Top 10 as standards. The smartest model in the benchmark is 5 times more vulnerable than the cheapest. The data connects to real use cases where LLMs are in production: RAGs, chatbots, agents, code assistants. The closing is actionable: 5 design pillars for AI systems that don't depend on the model for their own security, with real code from NVIDIA NeMo Guardrails and Meta LlamaFirewall.

Want to know more?

Join PyCon Colombia newsletter and get a complete overview of our events, speakers and community participation.