Running a service means failures will happen. Reading Google's SRE book made me realize that operations is a high-level engineering problem, not just toil. I walk through how the concepts of SLI, SLO, and Error Budget shift your mindset from firefighter to architect.
How to deploy without shutting down servers. Differences between Rolling, Canary, and Blue-Green. Deep dive into Database Rollback strategies, Online Schema Changes, AWS CodeDeploy integration, and Feature Toggles.
Solving server waste at dawn and crashes at lunch. Understanding Auto Scaling vs Serverless through 'Taxi Dispatch' and 'Pizza Delivery' analogies. Plus, cost-saving tips using Spot Instances.
Why your server isn't hacked. From 'Packet Filtering' checking ports/IPs to AWS Security Groups. Evolution of Firewalls.
Why would Netflix intentionally shut down its own production servers? Explore the philosophy of Chaos Engineering, the Simian Army, and detailed strategies like GameDays and Automating Chaos to build resilient distributed systems.
When you run a service, failures are inevitable. Traffic spikes fill up DB Connection Pools, Redis runs out of memory, a deployment breaks something unexpected.
When I started studying how to handle these situations, I kept running into the same question.
"Critical Alert: API Response Time > 5000ms"
When an alert like this fires, what do you actually do? Restart the server? Scale out? And if that keeps happening, where do you start fixing it at the root?
There had to be a way to build systems that fail less often—not just put out fires. Then I read Google's SRE (Site Reliability Engineering) book, and it hit me like a truck.
Google defines it this way: "SRE is what happens when you ask a software engineer to design an operations function."
Repetitive incident response and manual deployments are called Toil in SRE terms. A great SRE team keeps Toil under 50% and spends the remaining 50% on 'coding to improve systems'.
"Operations is not about cleaning up after developers. It's about ensuring system reliability through code." This sentence reframes what operations is actually for.
To understand SRE, you only need these three.
If someone asks "Is the server healthy?", it's hard to answer. You need numbers.
These metrics are SLIs. Putting these three front and center on a monitoring dashboard (Grafana) is where you start.
Once you measure SLI, you need a target. "Let's aim for 100% availability!" ...That sounds right, but from an SRE perspective it's the wrong goal.
100% availability is impossible and exponentially expensive. 99.9% is enough. (Allows 43 minutes downtime/month) 99.99% is excellent. (Allows 4 minutes downtime/month)
The key is defining a concrete, service-appropriate SLO—something like "Response Time P95 < 500ms".
This is the most interesting concept. If SLO is 99.9%, the remaining 0.1% is "allowance for failure". This is the Error Budget.
This rule eliminates the endless debate between Dev and Ops—Google's SRE teams use exactly this mechanism. "We have budget, let's ship." "We burned the budget, no deploys this week. Let's refactor."
Once you internalize SRE philosophy, how you approach operations changes fundamentally.
The goal shifts from firefighter running around putting out fires to architect designing fire-resistant buildings where automated sprinklers kick in. That's the role SRE aims for.
Taking it a step further, you reach the stage of "Setting fires on purpose". Netflix's 'Chaos Monkey' is famous for this. It randomly shuts down working servers.
You might think "Are they crazy?", but the logic is sound. "Failures will happen anyway. So let's trigger them intentionally when we are in control (office hours) and verify if the system self-heals."
For example, forcibly killing one web server during business hours lets you verify that the Load Balancer automatically excludes the dead node and redistributes traffic. Running this test in a controlled setting—before production forces the issue—is the core idea behind Chaos Engineering.
For those who want to start SRE but don't know what tools to use.
If you are new to SRE, these terms might be confusing.
Q. How is it different from DevOps? A. Google says: "class SRE implements interface DevOps". If DevOps is a cultural movement saying "Dev and Ops should collaborate", SRE provides concrete methodologies (SLO, Error Budget, etc.) on "How to actually do that".
Q. Do small startups need this? A. You don't need a dedicated team, but you need the 'Mindset'. Even a 3-person team needs to decide "Where do we store logs?" and "Who gets alerted if the server dies?". That is the start of SRE.
Q. Do I need to be good at Math? A. Basic statistics help. We use Percentiles (P95, P99) much more than Average. Averages are easily skewed by outliers.
A famous quote from the SRE book. "Hope is not a strategy."
Praying "Please let there be no errors this deployment" is not engineering. Engineering is assuming failure, measuring it, and agreeing on acceptable limits.
How about your service? When a user complains "It's slow", are you answering "It's just your feeling"? Open Grafana right now and check your SLIs.
