It’s Just the Way I Know How to Do It

“It’s just the way I know how to do it.”

That was the whole answer. No tradeoff. No constraint. No number.

A couple of days ago I had some free time, and I read a blog post about a cloud-native app on AWS. It teaches neurodivergent people the basics of handyman work. Good cause, good product. The architecture behind it was heavy: far more services than the job needed. I spent a while trying to reverse-engineer the design. I wanted to find the constraint that forced each choice. I couldn’t find one.

So I asked the lead developer directly.

Me: What tradeoff pushed you toward this stack? I’m trying to understand the reasoning.

Him: Honestly? It’s just the way I know how to do it.

I respect the honesty. But you should notice what that sentence is. It is not a design decision. It is a habit wearing the costume of one.

That conversation sent me back through every AWS project I designed during my freelancing years and inside a startup that eventually failed. One of those projects taught me more than all the others combined. This post is about that one.

A free consultation with a real bill attached

A close friend called me. A possible investor was interested in our startup, but we had to prove we could deliver. The test: give the investor a free cost-optimization consultation for his own product, a cross-platform delivery and dispatch app.

The app was running on AWS for roughly $3,500 a month. I can’t share the details of the old design because they sit under the customer’s NDA. What I can tell you is that it was complex.

My first move was the obvious one. I contacted the agency that built the system and asked for their architecture documentation and the reasoning behind it. What came back was AI slop. Lots of words with zero decisions and justifications.

If nobody can explain why a component exists, nobody can tell you what happens when you remove it. You are flying blind… So I stopped trying to understand the old design and started over from the requirements.

The constraints came first, the services came last

The product was really two platforms:

1. A customer platform for placing orders.
2. An operations platform for drivers, dispatchers, and supervisors.

The owner had one hard rule: do not touch the customer app. Fine. That rule did not freeze the backend. It froze the API contract. The customer app keeps calling the same endpoints with the same payloads. Everything behind those endpoints was mine to redesign…

So I went all in on the operations side, and I picked my battle using the oldest model in project management. The classic project triangle (SCT Triad), puts scope, time, and cost on the corners, with quality as the thing that bends when you squeeze. My version for this project was scope, cost, and quality. I squeezed scope (one platform, one region, nothing speculative) and cost (every service had to earn its line on the bill), and I accepted a controlled drop in quality.

Controlled is the key word. Quality was not allowed to drop wherever it liked. I decided in advance where it would drop and by how much.

The architecture

Note: The diagram below is an illustrative example based on my first draft. It is not the production design, which is the customer’s business secret (so to speak 😅).

Here’s the shape of it, request by request.

Edge. Cloudflare sits in front of everything for DNS, caching, and DDoS absorption. Behind it is one Application Load Balancer. Not two. An ALB has to span subnets in at least two Availability Zones, so it shows up in each AZ on a diagram, but it is one resource with one bill. Listener rules split the traffic: /order goes to the Order service, /dispatch goes to the Dispatch service. You only pay for a second load balancer when the two sides have genuinely different compliance needs, and these didn’t.

Compute. Two separate Auto Scaling Groups on small Graviton instances (t4g), both in private subnets across me-south-1a and me-south-1b. The Order service does one job: validate the order, write it, confirm it. The Dispatch service does everything operational: match drivers, push routes, and feed the supervisors’ live dashboard over WebSockets.

Why split them? Because their failures should never meet. A flood of customer orders must not starve dispatch. A dispatch restart must not stop the business from taking money. Two groups cost a little more than one. Isolated failure domains > a slightly smaller bill.

Events. When an order lands, the Order service publishes an OrderCreated event to Kafka, and Dispatch consumes it. The two services never call each other directly.

State. RDS PostgreSQL holds the truth: accounts, orders, billing. ElastiCache for Redis holds the volatile stuff: live driver positions and dashboard sessions. Redis has native geospatial commands, so “find the nearest free driver” is a single GEOSEARCH call, not a hand-written distance query against your database.

Why keep Kafka, if it’s the heaviest piece?

Fair question. I asked it myself. For a system this size, a managed queue such as SQS would be cheaper and simpler to run.

But Kafka was already a dependency, and the old system’s services already spoke it. My goal was to move to the new architecture with minimal downtime. Ripping out the event backbone in the middle of a migration would have turned one risky change into two. So Kafka stayed, on a single broker, on purpose.

This is the part juniors usually miss. The best component in a vacuum is not always the best component for this migration, this week, with this team. Architecture has a time dimension.

Why not go fully serverless?

Also fair. At low traffic, Lambda can be cheaper than idle EC2. But this system holds long-lived WebSocket connections for the dashboard and runs continuous Kafka consumers. Both are awkward fits for short-lived functions. Moving to serverless would also have meant rewriting most of the backend. A rewrite was the one thing I could not afford on a free consultation that had to prove itself.

What I gave up, and what it cost

Every cut on the bill was a risk I accepted. Here they are, written down, because a tradeoff you haven’t written down is just a surprise waiting for a date.

That last row deserves a closer look, because it is a classic.

The Order service commits the order to PostgreSQL. Then it publishes the event to Kafka. Now ask yourself: what if the commit succeeds and the publish fails? The customer gets a confirmation. The order exists in the database. Dispatch never hears about it. Nobody delivers the food.

This is called the dual-write problem. The standard fix is the transactional outbox pattern, documented by Chris Richardson on microservices.io. Instead of publishing directly, you write the event into an outbox table in the same database transaction as the order. A separate relay reads that table and publishes to Kafka, retrying until it succeeds. Either both writes happen or neither does. Example from phase two:

BEGIN;
INSERT INTO orders (id, customer_id, total) VALUES ($1, $2, $3);
INSERT INTO outbox (event_type, payload) VALUES ('OrderCreated', $4);
COMMIT;
-- A separate relay publishes rows from outbox to Kafka, then marks them sent.

We knew about the gap at launch, and we accepted it for phase one. Phase two closed it. Knowing your gaps is the job. Pretending you have none is how you end up debugging a missing order at 2 a.m.

The database row has a second lesson. Multi-AZ RDS is not instant either. AWS documents failover times of typically 60 to 120 seconds. During that window, new writes fail. Multi-AZ shrinks your outage. It does not erase it.

Scaling “just in case”

The small instances were a bet that normal traffic would stay normal. I don’t like bets without a hedge. So I wrote the whole thing in Terraform, including the Auto Scaling Groups and the CloudWatch alarms that drive them. Sample from the first version:

resource "aws_autoscaling_policy" "dispatch_scale_out" {
  name                   = "dispatch-scale-out"
  autoscaling_group_name = aws_autoscaling_group.dispatch.name
  adjustment_type        = "ChangeInCapacity"
  scaling_adjustment     = 1
  cooldown               = 180
}

resource "aws_cloudwatch_metric_alarm" "dispatch_cpu_high" {
  alarm_name          = "dispatch-cpu-high"
  namespace           = "AWS/EC2"
  metric_name         = "CPUUtilization"
  statistic           = "Average"
  period              = 60
  evaluation_periods  = 3
  threshold           = 70
  comparison_operator = "GreaterThanThreshold"

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.dispatch.name
  }

  alarm_actions = [aws_autoscaling_policy.dispatch_scale_out.arn]
}

One warning if you copy this idea. Burstable instances like t4g run on CPU credits. Watch CPUCreditBalance too, not just CPU. An instance that has burned its credits looks healthy right up until it doesn’t.

Deployments follow the same cost logic. GitHub triggers AWS CodeDeploy, which runs a blue/green deployment: a temporary green fleet receives the new build, gets verified, takes the traffic, and the old fleet is terminated. Nothing stays on just to wait for the next release.

Security didn’t get cheaper

Cutting cost is not a license to cut corners on exposure. The Order and Dispatch instances, Kafka, RDS, and Redis all live in private subnets. Nothing in that list has a public IP. The NAT Gateway allows outbound updates without allowing inbound connections.

The quiet risk is the load balancer. Cloudflare only protects you if traffic actually goes through Cloudflare. If your ALB accepts connections from anywhere, an attacker who finds it can skip the WAF and the DDoS protection entirely. Cloudflare’s own documentation recommends blocking everything at the origin that doesn’t come from its published IP ranges, or removing the public origin entirely with Cloudflare Tunnel. Lock the ALB’s security group to those ranges. It costs nothing.

The result

The bill went from ~$3,500 a month to $200 to $600 a month, depending on activity and events. At the busy end that is 5.8 times cheaper. At the quiet end it is 17.5 times cheaper. Over a year, the owner keeps somewhere between $34,800 and $39,600 that used to go to AWS.

The company has been earning on it ever since, with zero technical issues of its own. The only outages came from AWS.

And one of those deserves an honest paragraph. In March 2026, AWS’s Bahrain region, me-south-1, suffered a major disruption, with two of its three data centers losing power. This platform runs in that region, and it felt it.

Here’s the lesson, and I want you to take it seriously. Multi-AZ protects you from losing a building. It does not protect you from losing a region. Werner Vogels has spent years repeating that “everything fails all the time.” He means it literally. Region > AZ, and only you can decide whether your business can afford to plan for the bigger one.

Technician or engineer?

Yes, I’m yapping now. I’ll keep going anyway.

A technician knows how to build the thing. An engineer knows why this thing, and not the other one. The technician asks which service does the job. The engineer asks what the job costs, what it risks, and what happens when it breaks. The technician gives you an architecture. The engineer gives you an architecture and the list of things it will not do.

The agency had technicians. They could wire services together all day. Nobody had asked what each service was for, so nobody could tell me what removing it would cost. That’s not a skill gap. It’s a thinking gap.

This project was one of the most important steps for me before I moved into a one-year SysAdmin role at a private B2B game server hosting provider. It taught me to think architecturally before thinking technically. Start with constraints. Price every component. Write down every risk you accept.

So, the next time someone asks you why your system looks the way it does, listen to your own answer. If what comes out is “it’s just the way I know how to do it,” stop there. That’s not the end of the conversation. That’s where the engineering starts.

Takeaways

1. Freeze the API contract, not the backend behind it.
2. Make every service on your bill justify its line.
3. Decide in advance where quality is allowed to drop, and by how much.
4. Write down every accepted risk, including the ones you plan to fix later.
5. Use Terraform so your safety net is code, not memory.
6. Plan for the failure scope your business can’t survive: AZ, or region.

October 19, 2026