Stop fearing Friday deploys. Feature Flags (or Toggles) allow you to push code to production while keeping it dormant. We explore the 4 categories of flags, how they enable Trunk-Based Development, the power of Canary Releases, and the critical importance of cleaning up stale flags to prevent technical debt.
How to deploy without shutting down servers. Differences between Rolling, Canary, and Blue-Green. Deep dive into Database Rollback strategies, Online Schema Changes, AWS CodeDeploy integration, and Feature Toggles.
Solving server waste at dawn and crashes at lunch. Understanding Auto Scaling vs Serverless through 'Taxi Dispatch' and 'Pizza Delivery' analogies. Plus, cost-saving tips using Spot Instances.
Why your server isn't hacked. From 'Packet Filtering' checking ports/IPs to AWS Security Groups. Evolution of Firewalls.
Why would Netflix intentionally shut down its own production servers? Explore the philosophy of Chaos Engineering, the Simian Army, and detailed strategies like GameDays and Automating Chaos to build resilient distributed systems.
In my early days as a founder, Friday at 5 PM was "Fear Time." It was when my development team would push the features they had built all week to the live server.
"Alright, deploying now."
After pressing Enter, my heart would start racing. What if the login button doesn't work? What if there's an error on the payment page? Sure enough, the Slack notification would sound.
"Boss, sign-up isn't working."
From that moment, hell would break loose. We'd dig through logs to find the cause, frantically patch the code, rebuild, and redeploy. 10 minutes, 20 minutes... while the service was down, I'd break out in a cold sweat. Finally, deciding it wasn't going to work, I'd yell, "Rollback!" By the time we reverted to the previous version, it was already 8 PM. My team's weekend was gone.
I thought this was just a developer's fate. That deployment was inherently scary and risky. But it turned out, this was due to my ignorance in not distinguishing between "Deployment" and "Release."
When I first encountered this concept, it felt like I had been hit over the head.
I had always done these two things simultaneously. The moment code went up on the server (deployment), users could see the feature immediately (release). So, deployment equaled release, and a deployment failure equaled a service outage.
But after I discovered Feature Flags, this equation broke.
Even if the code is up on the server (deployment complete), if I turn the switch off, users can't see the feature (pre-release). Now, I smile even when deploying at 5 PM on a Friday. The feature is off anyway. I come in on Monday morning, have my morning coffee, and 'click'—turn the switch on. If a problem arises? 'Click'—turn it off again.
No need to take down the server to rollback, no need to rush a hotfix. This is the secret to my liberation from deployment fear.
At first, I thought it was just about turning features on and off with an if (true) statement. But as I studied more, I realized that feature flags have distinct purposes and types. The four categories summarized in Martin Fowler's blog resonated with me the most.
This is the most basic form. It's for hiding unfinished features.
Previously, when developing a large feature, I would create a branch like feature/big-update and work on it for 2 or 3 weeks. Then, when trying to merge it into the main branch (main), massive conflicts would occur. This is known as 'Merge Hell.'
But with Release Toggles, I can wrap even unfinished code in a flag and merge it into the main branch every day.
if (featureFlags.isOn('new-billing-system')) {
useNewBilling(); // Still WIP
} else {
useOldBilling(); // Currently working fine
}
This way, my code mixes with my colleagues' code daily, so there are no big conflicts later. This was the core of Trunk-Based Development.
Marketers and PMs love this one. It's usually called A/B Testing.
"Will users click more if the buy button is red or blue?"
Instead of spending hours in meetings debating this, we just build both and split them with a flag.
Then we look at the data. If the red button has a 5% higher click-through rate, we switch the flag to ON for all users. We created a culture of fighting with data, not gut feelings.
This is the system's 'seatbelt.' For example, let's say our main page has a 'Real-time Trends' widget. It's great normally, but if traffic spikes, this feature might overload the DB and slow down the whole site.
In this case, we plant an Ops Toggle in advance. If the server monitoring alarm goes off, a developer can turn off the show-realtime-search flag from the dashboard without modifying code. The widget disappears, but the rest of the site runs normally. This is also called a Kill Switch.
This opens features only to specific users.
My team uses this for 'Dogfooding' (using our own product). We open new features only to employees for a week (Permission Toggle), and if there are no bugs, we release them to all users (Release Toggle).
Hiding code with if statements is easy, but what about database (DB) changes?
For instance, say we're splitting the address column in the users table into address_city and address_detail. If we change the DB before deploying the code, the old version of the code will error out before the deployment finishes.
Here, I learned the Expand-Contract pattern.
address_city, address_detail). Leave the existing address column alone. Since we're only changing the DB schema, there's no service impact.address data of users joined in the past and fill the new columns.address column from the DB.This process is tedious and long. But it turned out to be a necessary step for Zero-Downtime Deployment. It's a hundred times better than trying to "deploy in one shot" and "failing in one shot."
At first, to save money, I managed flags in a config.json file.
{
"new-login": true,
"promo-banner": false
}
But this required a redeployment to change a flag. I had lost the core value of "changing flags without deployment."
So I made a feature_flags table in the DB and created an API. But querying the DB every time slowed down performance. So I attached Redis.
The work kept growing. I had to build a flag admin page, log who turned what off when (Audit Log)... Eventually, I concluded: "Don't reinvent the wheel."
Since we are in the startup phase, we are currently self-hosting the open-source Unleash.
Once we make some money, I plan to switch to a SaaS like LaunchDarkly. It requires no management and has incredibly powerful targeting features.
Feature Flags are not a 'free lunch.' There is a cost. That cost is code complexity.
if/else statements get plastered all over the code. Test codes have to be written twice (when ON, when OFF).
The most terrible thing is the "Zombie Flag." An experiment ends, option A is chosen, but the flag code isn't deleted and is just left there. A year later, someone accidentally switches that flag to OFF. Suddenly, a Paleolithic era UI from a year ago pops up.
After experiencing this mistake a few times, I established some principles.
dev_sj_login_refactor) or leave it in the metadata. There is nothing scarier than a flag left behind by someone who has resigned.Looking back, the biggest effect of introducing Feature Flags wasn't technical, but psychological.
"It's okay to make a mistake. We can just turn it off."
Once this belief was established, team members started deploying boldly and frequently. Deployments went from once a day to 10 times a day. Smaller deployment units made it easier to find bugs, and bug fixes became faster. A virtuous cycle began.
If you are still afraid of Friday deployments, if you are placing your hand on the rollback button and praying, try introducing Feature Flags right now. Your weekends will change.