Shipping reliable AWS services the boring way

Most outages I've dealt with weren't caused by exotic bugs. They were caused by the boring things nobody set up: no health check, no alarm, a deploy that changed three things at once. Reliability is mostly a discipline, not a tool.

Here's the short version of how I keep services on AWS ECS quietly running.

Make the service tell you it's healthy

Every container gets a real health endpoint — not just "the process is up", but "the process can reach its database and its dependencies". ECS uses it to decide whether to keep routing traffic.

// A health check that actually checks something
app.get("/healthz", async (_req, res) => {
  try {
    await db.query("SELECT 1");
    res.status(200).json({ status: "ok" });
  } catch {
    res.status(503).json({ status: "degraded" });
  }
});

If this endpoint lies, every layer above it lies too.

Alarm on symptoms, not causes

I wire CloudWatch alarms to the things a user actually feels — error rate, p99 latency, queue depth — and route them to SNS so they reach a human (or a Slack channel) within seconds. You don't need a hundred alarms. You need three that you trust enough to wake up for.

An alarm you ignore is worse than no alarm. It trains you to ignore alarms.

Deploy small, deploy often

The single biggest reliability win is shrinking the blast radius of each change:

• One logical change per deploy, behind a flag where it's risky.
• Roll forward, don't scramble to roll back, when health checks catch a bad task.
• Keep the pipeline fast enough that small deploys feel cheap.

None of this is clever. That's the point — reliability comes from removing surprises, not adding cleverness. Do the boring things consistently and your pager stays quiet.

Working on something on AWS and want a second pair of hands? Get in touch.