Roberto Bedoya García

Your LLM Is Bleeding Money and Python Can Stop It

You put your LLM in production. It responds, it works, users use it. But there are questions you cannot answer: how much did it cost you exactly last week? Which prompt consumes the most tokens? Why was that response terrible on Tuesday at 3pm? How do you show the business that the system is improving? Most teams working with LLMs today operate blind. They deploy, receive complaints, adjust prompts without data, and get API bills that no one can explain. That is not engineering — it is expensive intuition. In this workshop you will build, line by line in Python, a complete observability system for LLM applications using OpenTelemetry, Langfuse, and Elasticsearch. No theory, no endless slides — real code you can take to your project the following Monday. What you will build: We start from a document Q&A agent that appears functional but is completely blind. During the workshop we instrument it step by step: first with Langfuse @observe decorators to capture every prompt, response, and token count; then we export those traces via OpenTelemetry to Elasticsearch, where we build a live Kibana dashboard showing cost per user, p95 latency per model, and low-quality response rate. Finally we implement an automatic evaluation layer with LLM-as-a-judge so the system audits itself. At the end of the workshop you have a functional, open source, self-hostable stack — without depending on closed SaaS platforms — that any team can adopt regardless of whether they use OpenAI, Anthropic, or local models via Ollama. What you take away: Complete GitHub repository with Docker Compose ready to spin up the environment, notebooks with each workshop step, and an exportable Kibana dashboard template. Everything in Spanish, documented so you can share it with your team. Who is this workshop for? For Python developers who already have basic experience with LLMs and APIs, and who are ready to take the step from "my POC works" to "my production system is reliable, measurable, and defensible to the business." You do not need prior experience with OpenTelemetry or Elasticsearch — we explain them from scratch in the context of LLMs. Why does this matter now? Global spending on LLM APIs doubled in less than six months. 47% of companies are running observability pilots for AI but only 7% have it in production. The gap between making an LLM work and operating it responsibly is the most urgent technical problem the industry has today — and Python has all the tools to close it.