Skip to content
devopsbymuh_

$ cat case-studies/

Production systems, taken apart in public

Not a projects page. Five systems I build, run and deliberately break — each one written up with the architecture, the failure drills, the numbers I measured myself, and the bill. Including the guardrails that did not hold.

I build and operate production AI systems on the cloud — and I run the infrastructure itself with AI agents in the loop, safely.

20 case studies planned0 published1 in progress

The five systems

Each one is a system I operate, dissected across four case studies. They are built in order, and nothing here is listed before it exists — a row without a link is a row I have not earned yet.

01

AI agents in the engineering workflow

I can put an AI agent inside real infrastructure work without handing it the keys to the business.

Why it is on the list · Two separate clients with $1M+ of hiring history now require it outright — one asks for LLM development companions integrated into infrastructure workflows, the other makes Claude Code experience a condition of the role.

17Giving an AI agent production access, safelyCan I let an AI agent touch production without losing the account?Building now
18AI-assisted Terraform: what it got right, and what it got dangerously wrongThe agent wrote infrastructure code. Do I trust the plan?Planned
19An agent that reviews infrastructure pull requestsEvery infra PR needs a senior reviewer, and we do not have one spare.Planned
20Rescuing a vibe-coded SaaS: an audit checklist for AI-generated codebasesNon-developers built our product with AI. Is it safe to run?Planned
02

Durable AI job platform

Long-running AI work that survives timeouts, crashes and provider outages — and can be resumed, not restarted.

Why it is on the list · The most repeated technical complaint in the research: workloads exceeding serverless execution limits, jobs that cannot be recovered, ingestion that stores half-parsed files as if they were valid.

01Why your AI feature dies at 30 secondsOur document processing times out and the user just sees a spinner.Planned
02Idempotent document ingestion: dedup, OCR fallback and partial failuresFailed pages are being stored as if they were complete records.Planned
03Job state you can trust: retries, leases and poison queuesA job is stuck and nobody can tell whether it is safe to run again.Planned
04Chaos day: I killed the worker, the database and the LLM provider mid-jobWhat actually happens to in-flight work when something goes down?Planned
03

LLM reliability lab

I can measure whether an AI feature is getting better, and enforce what it is not allowed to do in code rather than in a prompt.

Why it is on the list · Buyers now ask for evals by name — frozen datasets, pass rates, release-blocking safety invariants — and almost nobody can show one running.

05An eval harness from scratch: frozen dataset, rubric, one commandWe changed the prompt. Did anything actually improve?Planned
06Guardrails that are code, not promptsThe assistant told a user it moved something. It had not.Planned
07Prompt injection through user content: attacking my own RAG pipelineOur users upload the documents the model reads. What could they hide in one?Planned
08A failure taxonomy, and which fixes actually moved the numberHalf our AI outputs are wrong and we cannot say why.Planned
04

Multi-tenant AI knowledge system

Retrieval that respects who is asking — tenant isolation, entity resolution, and an audit trail for every action taken.

Why it is on the list · What every "AI brain for our company" posting is really asking for: hundreds of clients' data in one system, and answers that must never cross between them.

09RAG that knows who is asking: tenant isolation in a vector storeOne customer's data must never surface in another's answer.Planned
10Beyond the chatbot: entity resolution, routing, and refusing to answerIt answers about the wrong property, confidently.Planned
11Knowledge freshness without re-embedding the worldOur SOPs changed last week and the assistant is still quoting the old ones.Planned
12Agent tool-use with an audit log: traced, attributable, reversibleThe agent did something in our CRM and we cannot reconstruct what.Planned
05

The production platform

Reproducible infrastructure, a deploy path that can be reversed, zero-trust in front of it, and a bill I can explain line by line.

Why it is on the list · The highest-paying roles in the research, and the least glamorous: Terraform and Terragrunt, Kubernetes, Cloudflare Zero Trust, staging that is genuinely isolated from production.

13One codebase, two targets: cloud deploy and an on-prem installerHalf our customers will not put their data in someone else's cloud.Planned
14Promoting one feature — code and migration — without dragging the restOur staging environment is not really separate from production.Planned
15Zero-trust in front of everything, and what each layer actually blocksWe had an incident. We are not sure what else is exposed.Planned
16The bill: cost per month, cost per request, and the changes that cut itOur cloud spend went up again and nobody can explain why.Planned

$ ./audit.sh

Production Readiness Audit

Five days, fixed price, and a written report that tells you exactly what will break in production and in what order to fix it.