I Regretted Using Kubernetes for a Service with 100 Users

1. "Hey, Aren't You Using Kubernetes Yet?"

Go to any developer community, and they say: "Who deploys directly to EC2 these days? You gotta use Kubernetes." "If you want microservices (MSA), k8s is mandatory."

I didn't want to fall behind the 'Hip' trend. So I decided to adopt Kubernetes (EKS) for my precious toy project (10 daily visitors, 100 total users). Without knowing it would eat away my bank account and sanity.

1.5. Kubernetes Explained with Shipping Analogy

If Docker is the "Shipping Container", Kubernetes is the "Automated Harbors".

• Container: Your application (The Box).
• Pod: A small barge carrying the box. (Can carry one or multiple).
• Node: The Dock where barges attach (The Server).
• Control Plane: The Control Tower. Orders "Dock A is full, move to Dock B!"
• Service: A "Radio Channel" that lets barges talk to each other without knowing exact coordinates.

---

2. The First Bill: $144 Shock

A month later, I doubted my eyes looking at the AWS bill. I only used 2 of the cheapest t3.small instances, why is the bill so high?

The culprit was the EKS Control Plane. Amazon charges $0.10 per hour for managing the master node, which is about $72 per month. Add worker node costs, Load Balancer (ALB) costs, NAT Gateway costs... the overhead was bigger than the actual service.

"Paying $150/month just to run a static website and one API server?"

One EC2 instance with Docker Compose would have cost $5. Kubernetes has a steep learning curve—and when adopted prematurely for a small service, costs can balloon dramatically. That's the trap of chasing the illusion of Scalability.

---

3. Falling into YAML Hell

A bigger problem than money was Productivity Drop. To change one line of code and deploy, I had to edit too many files.

Just look at the amount of code I had to write to simply "launch a web server".

1. Deployment.yaml (App Definition)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: my-app:latest
        ports:
        - containerPort: 3000
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"

2. Service.yaml (Internal Networking)

apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
  type: NodePort

3. Ingress.yaml (External Exposure & LB)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app-service
                port:
                  number: 80

"I just want to open port 3000, why do I need 3 files and 50+ lines of code?" In Docker Compose, it would be just one line: ports: - "3000:3000".

If kubectl apply failed, checking logs required 5 steps.

1. kubectl get pods (Find name)
2. kubectl logs my-app-xz8j9 (Check logs)
3. "Wait, wrong pod?"
4. kubectl describe pod my-app-xz8j9 (Check events: ImagePullBackOff? OOMKilled?)
5. Infinite loop...

I was spending 80% of my time on Infrastructure Config (YAML) and only 20% on Business Logic (Feature Dev). It was putting the cart before the horse.

3.5. Pod vs Node: The Most Confusing Part

People ask: "Why do we need Pod? Isn't Container enough?" Docker manages Containers, but Kubernetes manages Pods. Why? Because of "Inseparable Friends". For example, a Web Server container and a Log Collector container must always stick together. If you put them in one Pod, they share localhost and can talk to each other easily. Like roommates sharing a single room (Pod). On the other hand, Node is the Building (Server Computer) where these rooms exist.

---

4. Scenario: The Traffic Spike That Never Came

I even set up HPA (Horizontal Pod Autoscaler), dreaming, "What if a famous influencer mentions my app?" If CPU usage went over 50%, it would automatically spawn more pods.

But the reality was cruel.

• Concurrent Users: Max 3 (Me, my friend, Google Bot)
• CPU Usage: 0.1%

The HPA sat there doing nothing, while I paid for the Metrics Server to monitor it. It was like hiring 10 waiters for a restaurant with zero customers, just "in case".

To make matters worse, I used RDS (Relational Database). What good is scaling the web server if the DB bottlenecks? (And scaling DBs is much harder.)

4.5. The Hidden Cost: NAT Gateway

Many beginners miss the NAT Gateway trap. EKS Worker Nodes are usually placed in a Private Subnet for security. To talk to the internet (e.g., Pulling Docker Images), they must go through a NAT Gateway. AWS charges $0.045/hour + $0.045/GB. Just keeping it on costs ~$32/month. Every time you pull an image or call an external API, the meter spins. If you used a simple EC2 in a Public Subnet, this would be free.

---

5. Hidden Complexities: Networking & Storage

It wasn't just YAML. The learning curve was a cliff.

1. Networking Complexity: "Pods need a Service to talk, an Ingress to be exposed, and an ALB Controller for that Ingress..." I spent 3 days just connecting a domain.
2. Storage Nightmare (PVC/PV): In Docker, -v /data:/data is enough. Here, you need PersistentVolumeClaims, StorageClasses... one wrong move and data is gone.
3. Debugging Difficulty: Works on my machine, fails on the cluster. Why? NetworkPolicy? Image Tag? OOMKilled? Too many variables.
4. The Security Sinkhole: "Is your cluster open to the public?" "Are your secrets encrypted?" "Did you set up RBAC correctly?" Kubernetes is secure by default only if you are a security expert. I spent weeks just figuring out how to rotate certificates.

I was becoming an "Infrastructure Engineer", not a "Product Developer".

5.2. The Betrayal of Ingress

"Just create an Ingress, right?" Not so fast. In AWS, you need to install the AWS Load Balancer Controller separately. This controller creates an ALB (Application Load Balancer) for you. Guess what? Each ALB costs ~$20/month. If you have 5 apps and create 5 Ingresses, that's $100/month just for LBs. You eventually learn to set up a single Nginx Ingress Controller to route all traffic through one ALB, but configuring that is a whole new level of pain.

5.5. The Zombie Pods (Self-Healing)

Of course, there are good parts. The greatest charm is "Self-Healing". Since K8s is "Declarative", if you say "Keep 3 replicas," it will maintain 3 replicas no matter what. Even if I accidentally (or intentionally) kill a pod with kubectl delete, it spawns a new one in 0.1s. This is why developers can sleep when a server crashes at night due to OOM. K8s quietly resurrects it. (Unless the DB crashes. Then you wake up. K8s can't fix a corrupt DB.)

---

6. When Do You Actually Need Kubernetes?

Of course, Kubernetes is a great tool. Google didn't make it for nothing. But it wasn't for "My Service".

Before you blindly helm install everything, ask yourself these brutal questions:

1. Do you have more than 5 backend developers? (If it's just you, k8s is overkill.)
2. Do you deploy more than 10 times a day? (If you deploy once a week, simple scripts are fine.)
3. Do you really need auto-scaling? (Or can you just resize the instance from t3.small to t3.medium?)
4. Do you have a dedicated, full-time DevOps engineer? (If you are also the frontend dev, stay away.)

If you answered NO to any of these, stick to Docker Compose or a PaaS like Vercel/Railway.

Here is my realization of "When to use Kubernetes":

1. When you satisfy >100 servers: Humans cannot manage them manually.
2. When doing Real MSA: Complicated inter-service communication, independent deployment/scaling needed.
3. When you have a dedicated DevOps Team: The learning curve is too high for developers to handle infra too.

My service?

• 2 Servers
• Monolithic Architecture
• Developer: Just Me

Docker Compose was more than enough.

6.5. If You MUST Use It (Pro Tip: k9s)

If your company grows and you are forced to use Kubernetes, please don't just rely on raw kubectl. Install k9s. It gives you a text-based UI in the terminal to view pods, logs, and shell into containers. Without this tool, I would have probably quit my job.

---

7. Back to Docker Compose

Eventually, I swallowed my pride and deleted the EKS cluster. (Running terraform destroy felt cathartic.) And I bought a single AWS Lightsail server ($5/mo).

# docker-compose.yml
version: '3'
services:
  app:
    image: my-app:latest
    ports:
      - "3000:3000"
    restart: always
  db:
    image: postgres:13
    volumes:
      - ./data:/var/lib/postgresql/data

This 10-line file solved everything. Deploy? Just docker-compose up -d. Done.

Why Docker Compose Was Enough

Many people think they need Kubernetes for "Service Discovery". But Docker Compose has built-in DNS. In the app container, I can simply connect to postgres://db:5432. The hostname db automatically resolves to the database container's IP. I don't need CoreDNS, I don't need Kube-Proxy, I don't need Sidecars.

Also, for the "Zero Downtime Deployment" that Kubernetes boasts about? A simple docker-compose up -d --no-deps --scale app=2 --no-recreate app script can mimic a rolling update. For a service with 100 users, a 1-second downtime during deployment is absolutely acceptable. Nobody dies if my blog is down for 2 seconds at 3 AM.

Cost? $5/month. Stress? 0.

---

8. Conclusion: Stop Resume Driven Development (RDD)

We often get drunk on "Cool Tech" and overlook "Cost of Tech". This is called Resume Driven Development (RDD). Adding unnecessary complexity just to write "I used Kubernetes" on your resume.

The core of engineering is finding "Appropriate Technology". Trying to kill a chicken with a butcher's cleaver will only cut your own hand.

If your service is still a 'chicken', please just use a fruit knife (Docker). You can take out the cleaver (Kubernetes) later when your service becomes a 'bull'.

P.S. If you really want to dive deep into Kubernetes, I highly recommend the book "Kubernetes Up & Running". It explains the core concepts much better than random Medium articles or Chat GPT.