How I Built a Production Observability Stack — And Broke It Twice Before It Worked

I used to dismiss monitoring as something you bolt on after the real engineering is done. Logs were noise. Metrics were "a later problem." Alerts were for teams with dedicated SREs, not a small startup running three service types on Render. I was wrong. Badly wrong. And it took a self-inflicted incident — where my own monitoring system became the thing that needed monitoring — to understand why observability is engineering, not afterthought. This is a detailed account of building our observability stack from scratch: what we built, what broke, why it broke, and what the architecture looks like today. The starting point We run three service types on Render: A web service — the main API and frontend server Background workers — async job processors (queuing, retries, scheduled tasks) Key-value stores — Render's managed Redis-compatible service Before this project, each service logged to Render's default log drain. If something broke in production, you'd open the Render dashboard, pick a s