The Perfect Pipeline: How to Ship Code Without Crashing Production

Rantideb Howlader••16 min read

Introduction: The Friday Fear

We've all been there. You worked on a feature for two weeks. It works on your laptop. It works in the test environment. Now, it's time to push it to the real world. To the users.

Your finger hovers over the "Deploy" button. Your heart rate goes up. You wonder: "What if I break it? What if the site goes down?"

If this sounds familiar, your pipeline is broken. Not the code—the process.

Deploying code should be boring. It should be as exciting as taking out the trash. If deploying code feels like defusing a bomb, you are doing it wrong.

In this guide, we are going to fix that. We are going to look at how the big companies (Netflix, Amazon, Google) ship code thousands of times a day without breaking a sweat. And no, you don't need a team of 100 people to do it. You just need a better strategy.

We are going to talk about Blue/Green Deployment. It sounds fancy, but it's actually a very simple idea: Instead of trying to fix the car while driving it, just buy a second car, get it running perfectly, and then jump into it.

Part 1: The Old Way (Rolling Updates)

Most people start with what we call a "Rolling Update."

Imagine you have 10 servers running your website (Version 1). When you want to update to Version 2, you:

  1. Pick Server #1.
  2. Turn it off.
  3. Install Version 2.
  4. Turn it back on.
  5. Move to Server #2.

The Problem: While you are doing this, you have a weird mix. Some users see the old site, some see the new site. If Version 2 has a bug (like a bad database schema), you break the site slowly, server by server. By the time you realize it, half your users are angry. And fixing it? You have to do the whole slow process in reverse.

It works, but it's stressful.

Part 2: The Better Way (Blue/Green)

Blue/Green is different.

  1. You have your current environment (Let's call it Blue). It has 10 servers running Version 1. All your users are here.
  2. You build a completely new environment (Let's call it Green). It has 10 new servers running Version 2.
  3. No users are on Green yet. It's empty.

This is the magic part. Because no users are on Green, you can test it. You can log in, click around, run automation scripts, and break things. It doesn't matter. The real users on Blue are happy and safe.

The Switch: Once you are 100% sure Green is perfect, you flip a switch. Usually, this switch is a Load Balancer. You tell the Load Balancer: "Stop sending traffic to Blue. Send it to Green."

Boom. In one second, all users are on the new version.

The Safety Net: What if you missed a bug? What if Green crashes immediately? You flip the switch back. "Send traffic back to Blue." Since you didn't destroy Blue, it's still sitting there, waiting for you. The rollback is instant.

Part 3: How to Build This Usage AWS

You might be thinking: "Double the servers? That sounds expensive." It can be. But remember, you only need the second set of servers for a few hours while you deploy. Once the switch is done and stable, you can turn off the old ones (Blue) to save money.

Here is how we build this in AWS.

The Ingredients

  1. EC2 Instances: Your servers.
  2. Target Groups: These are logical groups of servers. We will make two:
    • TargetGroup-Blue (Old)
    • TargetGroup-Green (New)
  3. Application Load Balancer (ALB): The traffic cop. It listens on Port 80 (HTTP) and forwards traffic to one of the Target Groups.

The Deployment Scripts

We don't do this by hand. We use a tool like AWS CodeDeploy.

CodeDeploy is smart. It installs an agent on your servers. When you trigger a deployment:

  1. It sees you have an Auto Scaling Group.
  2. It spins up new instances (Green) with your new code.
  3. It runs your "Health Check" scripts (e.g., curl localhost/health).
  4. If the health check passes, it tells the Load Balancer to switch traffic.
  5. It waits (e.g., 1 hour) to make sure everything is stable.
  6. It terminates the old instances.

Part 4: Canary Deployments (Dipping a Toe in the Water)

Blue/Green is great, but it's an "All or Nothing" switch. Everyone moves at once. Sometimes, you want to be even more careful. This is where Canary Deployments come in.

The name comes from coal miners. They used to carry a canary bird into the mine. If there were toxic gases, the canary would get sick before the humans did, giving them time to escape.

In software, a "Canary" is a small group of users.

How it works:

  1. Deploy Version 2 to just one server.
  2. Send 10% of your traffic to that server. Keep 90% on the old version.
  3. Watch the logs.
    • Are errors spiking?
    • Is the CPU usage high?
    • Are users complaining?
  4. If it looks bad? Kill the Canary. Only 10% of users were annoyed.
  5. If it looks good? Increase to 20%. Then 50%. Then 100%.

CodeDeploy settings:

  • Canary10Percent5Minutes: Shift 10% traffic. Wait 5 minutes. If safe, shift the rest.
  • Linear10PercentEvery1Minute: Shift 10% every minute until done.

Part 5: The "Traffic Shaping" Secret (Weighted Routing)

Most people think Blue/Green is a binary switch. 100% Blue -> 100% Green. But what if you aren't sure? What if you want to test the water with just 1% of users? "Canary" deployments usually do this, but you can do it manually with an Application Load Balancer (ALB).

The Listener Rule Hack

Inside your ALB, you have "Listeners" (e.g., Port 443). Inside the Listener, you have "Rules". Standard Rule: Forward to TargetGroup-Blue (100%).

You can edit this rule to be:

  • Forward to TargetGroup-Blue: 95%
  • Forward to TargetGroup-Green: 5%

Why is this better than CodeDeploy? Because you control the speed. CodeDeploy runs on a timer. The ALB approach runs on your courage. You can leave it at 5% for a whole day while you analyze logs. If you see errors, you edit the rule back to 0%. Instant fix.

Sticky Sessions (The Silent Killer)

Imagine a user logs in. Their session ID is stored in the memory of Server A (Blue). You switch traffic to Green. The user's next click goes to Server B (Green). Server B doesn't know who they are. Result: The user is logged out.

The Fix:

  1. Don't store sessions on the server: Store them in Redis (ElastiCache) or a Database. This is "Stateless Architecture."
  2. Session Stickiness (Cookie): Tell the ALB "If a user Start on Blue, keep them on Blue for 1 hour."
    • Trade-off: This makes Blue/Green slower because users "linger" on the old version.

Part 6: Database Migrations (The 'Expand-Contract' Pattern)

This is the hardest part of DevOps. Period. Using 2 versions of code (Blue and Green) with 1 Database.

If Green requires a new column, but Blue crashes if it sees unexpected columns... you have a problem.

We use the Expand-Contract Pattern (also called Parallel Change). It takes 4 deployments to make 1 database change.

The Scenario: Rename user.fullname to user.display_name.

Phase 1: Expand (Add)

  • Database: Add the new column display_name. (Allow NULLs).
  • Code: Deploy version 1.1. It writes to BOTH fullname and display_name. It reads from fullname.
  • State: Blue and Green are compatible.

Phase 2: Backfill (Copy)

  • Script: Run a background job to copy all old data from fullname to display_name.
  • State: Data is now synced.

Phase 3: Switch (Read)

  • Code: Deploy version 1.2. It reads from display_name. (It still writes to both, just in case).
  • State: Now the app is effectively using the new column.

Phase 4: Contract (Delete)

  • Code: Deploy version 1.3. stop writing to fullname.
  • Database: Delete the fullname column.
  • State: Migration complete.

Why go through this pain? Because at no point did we stop the database. At no point did we lock the table. At no point did a user get a 500 error. This is how Amazon changes database constraints while you are shopping.

Part 7: Mobile Apps (The Nightmare Scenario)

Blue/Green works for Web. It does NOT work for Mobile Apps (iOS/Android).

Why? because you cannot control when the user updates the app. You might deploy the "Green" API backend. But User Dave hasn't updated his iPhone app since 2023. Dave is sending "Blue" requests to your "Green" server.

The Strategy: API Versioning You must never change an API endpoint. You only create new ones.

  • Old: POST /api/v1/login
  • New: POST /api/v2/login

Your server must support both v1 and v2 at the same time. Maybe for years. Eventually, you look at your logs. When v1 traffic drops to zero, then you can delete the code. This is why your backend code is always larger than your frontend code. It's a museum of old versions.

Part 8: Feature Flags (The Ultimate Cheat Code)

Sometimes, you don't need to deploy new servers to release a feature. You can put the features inside "Feature Flags."

if feature_flags.is_enabled("new_checkout_page", user_id):
    show_new_page()
else:
    show_old_page()

This effectively separates "Deploying" (moving code to servers) from "Releasing" (showing features to users).

You can deploy the code on Tuesday. It sits there, dormant, hidden behind the False flag. On Friday, you log into your dashboard (like LaunchDarkly or a simple database table) and toggle the flag to True. The feature appears instantly. If it breaks? Toggle it back to False.

This is faster than rolling back servers. It takes milliseconds.

Part 9: Serverless Blue/Green (Lambda Aliases)

"I use Lambda, so I don't need this, right?" Wrong. You need it more. If you overwrite your Lambda code, and it has a bug, every request fails instantly.

The Lambda Solution: Aliases.

  1. Version $LATEST: This is your draft.
  2. Version 1: An immutable snapshot of your code.
  3. Alias "PROD": Points to Version 1.

Your API Gateway points to the "PROD" alias, not $LATEST.

The Deployment:

  1. Upload new code to $LATEST.
  2. Publish Version 2.
  3. Update Alias "PROD" to point to Version 2.
    • Advanced: AWS CodeDeploy can create a "Traffic Shift". It points 10% of "PROD" traffic to Version 2, and 90% to Version 1.

It is the exact same concept as the Load Balancer, just purely logical.

Part 10: Handling Secrets (Passwords and Keys)

In your pipeline, you need passwords (database URLs, API keys). Never put these in your Git code.

Use a dedicated storage like AWS Parameter Store or HashiCorp Vault.

The Workflow:

  1. Your code says: db_password = get_secret("/app/prod/db_password").
  2. Your server has an IAM Role (a badge) that says "I am allowed to ask for secrets."
  3. When the app starts, it fetches the password from AWS.

This means you can rotate (change) your database password without changing your code. You just update the Parameter Store and restart the servers.

Part 11: The "Break Glass" Protocol

Even with Blue/Green, Canaries, and Feature Flags, things will break. You need a "Break Glass" plan.

What to do when everything fails:

  1. Don't Fix Forward: Do not try to write a patch to fix the bug in production. You are stressed. You will make mistakes. You will make it worse.
  2. Rollback First: Your priority is to restore service. Go back to the last known good version (git revert).
  3. Investigate Later: Once the fire is out, download the logs to a safe place (staging) and debug there.

The "Red Button": You should have a script or a button in Jenkins/GitHub Actions that performs an immediate, forced rollback to the previous "Good" commit. Every engineer on the team should know where this button is.

Part 12: Glossary for the Manager

When you are explaining this to your boss, use these words:

  • Artifact: The zip file or container image that contains your code. It should be "Immutable" (never changed once built).
  • Idempotency: The property that running a script twice produces the same result. (e.g., mkdir -p is idempotent. mkdir is not).
  • Rollback: Reverting to the previous safe state.
  • Blue/Green: Two environments (Active/Idle).
  • Canary: Gradual traffic shifting.
  • A/B Testing: Different from Canary. A/B is for marketing (Which color button converts better?). Canary is for engineering (Does this code crash?).
  • Dark Launching: Releasing a feature to production but hiding it from menu bars/links so only users with the direct URL can find it.
  • Mean Time To Recovery (MTTR): The only metric that matters. How fast can you fix it when it breaks?

Part 13: DevSecOps (Scanning for Bombs)

A pipeline that deploys fast is great. A pipeline that deploys a virus is bad. Security cannot be an afterthought. It must be a "Gate" in your Blue/Green flow.

The "Left Shift" Strategy

Don't wait for the Pentest report next month. Scan the code now.

  1. SAST (Static Application Security Testing): Tools like SonarQube or Semgrep look at your source code.

    • Check: "Is there a hardcoded password?" "Is there SQL Injection?"
    • Action: If found, fail the build. Do not deploy to Blue.

  2. SCA (Software Composition Analysis): Tools like Snyk or Dependabot.

    • Check: "Is log4j outdated?" "Does this NPM package have a CVE?"
    • Action: Fail the build.

  3. DAST (Dynamic Analysis): Once Green is running (but before the switch), run OWASP ZAP. It attacks your running Green server. It tries to hack it.

    • Check: "Can I steal a cookie?" "Can I crash the API?"

The Rule: If the Security Gate fails, the Blue/Green switch never happens.

Part 14: Infrastructure as Code (Terraform Blue/Green)

How do you model this in Terraform? It's tricky because Terraform wants 1 state. Blue/Green implies 2 states.

The Swap Method (DNS)

resource "aws_route53_record" "www" {
  zone_id = var.zone_id
  name    = "www.example.com"
  type    = "CNAME"
  ttl     = "300"
  # Variable defines which environment is live
  records = [var.live_environment == "blue" ? aws_lb.blue.dns_name : aws_lb.green.dns_name]
}

Workflow:

  1. Run terraform apply -var="live_environment=blue". (Traffic goes to Blue).
  2. Deploy new code to Green infrastructure.
  3. Test Green using its direct URL green.example.com.
  4. Run terraform apply -var="live_environment=green".
  5. Route53 updates. Traffic shifts.

Pros: Simple. Cons: DNS caching. Some ISP routers ignore TTL. The switch might take 30 minutes for some users. (Use ALB Weighted routing for instant switching).

Part 15: Chaos Engineering (Testing the Pipeline)

How do you know your Rollback works? Have you ever tested it? Or are you just hoping it works?

The Fire Drill: Once a month, do a "Game Day".

  1. Deploy a "Bad Version" of the app to Green. (e.g., A version that returns 500 Errors on purpose).
  2. Trigger the Blue/Green switch.
  3. Watch your automated alarms.
    • Did CloudWatch detect the 500s?
    • Did CodeDeploy trigger the "Stop"?
    • Did it automatically roll back to Blue?

If you have to manually click "Stop", you failed the test. The pipeline should be smarter than you.

Part 16: Multi-Region Active/Active (The Unicorn)

Blue/Green is typically single-region (deploying new servers in Virginia). What if Virginia disappears? (It happens).

True resilience is Global Blue/Green.

  • Blue: The entire US-EAST-1 Region.
  • Green: The entire US-WEST-2 Region.

Global Application Load Balancer (AWS Global Accelerator): It gives you 2 static IP addresses. It routes users to the closest healthy region. If you deploy bad code to East, and East dies, Global Accelerator sends everyone to West.

Complexity Warning: Database replication (DynamoDB Global Tables or Aurora Global Database) is required. Latency constraints apply (Speed of light is slow). Cost is Double. Only do this if you are a bank or Netflix.

Part 17: Containerization (ECS/EKS Patterns)

If you use Kubernetes (EKS) or ECS, you don't swap Servers. You swap Tasks/Pods.

ECS Rolling Update:

  1. Target: maintain 10 tasks.
  2. Start 1 New Task (11 running).
  3. Drain 1 Old Task (10 running).
  4. Repeat.
  • Pros: Cheap. You only need capacity for 1 extra task.
  • Cons: Slow.

ECS Blue/Green (CodeDeploy):

  1. Target: 10 tasks in TaskSet-Blue.
  2. Start 10 tasks in TaskSet-Green. (20 running).
  3. Swap ALB Target Group.
  4. Kill TaskSet-Blue.
  • Pros: Instant. Safe.
  • Cons: You need double cluster capacity for 10 minutes. (Use Fargate so you don't pay for idle EC2s).

Part 18: Observability (Eyes on the Switch)

When you flip the switch, you are flying blind unless you have the right dashboards. Not just "CPU Usage". That's a vanity metric. You need Business Metrics.

The Dashboard:

  1. Order Volume: Did sales drop to zero when we switched?
  2. Login Success Rate: Can people get in?
  3. Latency (p99): Did the site get slower?
  4. Error Rate (5xx): Are we crashing?

Anomaly Detection: Use CloudWatch Anomaly Detection (Machine Learning). It knows that "Traffic usually drops at 3 AM". If traffic drops at 10 AM, it triggers an alarm. Hook this alarm into CodeDeploy. If it goes off -> Auto Rollback.

Part 19: The Culture of "Shipping"

Technology is easy. People are hard. If your team is afraid to deploy, they will hoard changes. They will wait 2 weeks, bundle 50 features into one "Big Bang" release on Friday. This guarantees failure.

The Golden Rule: Deploy Small. Deploy Often. If you deploy 1 line of code, and it breaks, you know exactly where the bug is. If you deploy 10,000 lines, good luck finding the needle.

Make deployment boring. Make it automatic. Make it reversible. And then, go home on time.

Conclusion: Boring is Good

The best compliment a DevOps engineer can get is silence. If nobody notices you, you are doing a great job. If nobody notices that you deployed 50 times today, you have built a Perfect Pipeline.

Start small. implementing Blue/Green takes time. Start by adding a simple Health Check endpoint. Then try a manual Blue/Green switch (changing DNS manually). Then automate it with CodeDeploy.

You will sleep better. Your users will be happier. And you will never have to fear the "Deploy" button again.

Further Reading

Ranti

Rantideb Howlader

Author

Connect
LinkedIn