Immediate Incident Checklist: What to Do When Cloudflare or AWS Causes an Email Outage

Immediate Incident Checklist: What to Do When Cloudflare or AWS Causes an Email Outage

Hook: When an upstream outage from Cloudflare or AWS knocks mail delivery offline, every minute costs business continuity, SLAs and customer trust. This prioritized runbook gives sysadmins exact triage steps, CLI commands and DNS/MX/SMTP checks you can run now to mitigate, restore, and safely fail back email services.

Why this matters in 2026

Late 2025 and early 2026 saw multiple high‑profile outages that demonstrated a persistent risk: centralized internet services (CDNs, DNS providers, cloud DNS like AWS Route 53) can create single points of failure for email flow. Organizations have since adopted multi‑provider redundancy and automated DNS failover — but during a live incident you still need a concise, prioritized runbook you can act on immediately.

Principle: act in priority order — detect, contain, restore, communicate, and prepare failback — no improvisation.

At-a-glance prioritized runbook (one-line checklist)

1. Detect & scope: Confirm outage source (Cloudflare or AWS) and scope impact on DNS, MX, HTTP(S), SMTP.
2. Notify: Publish initial incident on Statuspage and notify on‑call/PagerDuty.
3. Contain/mitigate: Route inbound MX to backup, enable alternate SMTP relay for outbound.
4. Stabilize queues: Stop mail loss, hold retry loops, monitor MTA queues.
5. Monitor & validate: Verify DNS propagation, SMTP handshake, TLS and authentication (DKIM/SPF).
6. Failback: Revert DNS/MX when upstream restored; watch queues as messages flow back.

1) Detect & scope — confirm the upstream provider is the cause

Quickly determine if Cloudflare or AWS is the likely root cause. Use provider status pages and independent checks.

• Check official status pages: Cloudflare (https://www.cloudflarestatus.com) and AWS (https://status.aws.amazon.com). Also check your provider consoles.
• Query public outage aggregators and social reports to triangulate (DownDetector, community Slack, Mastodon/X threads).

Quick CLI checks

Run these from an external network (not behind the potentially-impacted provider) and from multiple public resolvers — consider a trusted portable setup or remote workstation outside the affected region using a portable network kit.

DNS / MX

dig +short example.com MX                # Gets MX records
dig @8.8.8.8 example.com MX +noall +answer  # Check Google resolver
host -t mx example.com                      # host binary check

Look for missing MX records, or MX pointing to hosts that resolve to Cloudflare/AWS IPs that are down.

DNS TXT (SPF/DKIM/DMARC)

dig @8.8.8.8 example.com TXT +short    # SPF
dig selector._domainkey.example.com TXT +short  # DKIM
dig _dmarc.example.com TXT +short          # DMARC

SMTP reachability & TLS

telnet mail.example.com 25
# or using openssl to check STARTTLS and certs
openssl s_client -starttls smtp -connect mail.example.com:587 -crlf

Provider-specific checks

• Cloudflare: curl the API zone/dns endpoints to confirm API reachability.
• AWS Route 53: aws route53 list-hosted-zones & aws health events (if configured) from a workstation not in affected region — also consider integrating health checks with your operational playbooks (see observability & health playbooks).

2) Notify — publish a clear incident and assemble your response team

Why: Early, accurate updates reduce support load and prevent duplicate remediation steps by multiple teams.

• Publish a “We are investigating an email delivery issue” on your Statuspage with current impact and mitigation ETA (use incident templates).
• Notify on‑call/SRE via PagerDuty/opsgenie and channel in Slack/Teams with runbook link.
• Assign roles: Incident Lead, DNS Lead, MTA Lead, Communications Lead.

3) Contain & mitigate — immediate technical steps

Containment focuses on keeping mail flowing or preserving it for later delivery. Choose the path that matches your risk tolerance and existing setup.

Option A — Use an existing secondary MX (best if already configured)

If you have a configured secondary MX hosted with a different provider, verify it's accepting mail and ensure priorities are correct.

dig +short example.com MX
# If secondary exists, temporarily change priority if necessary via DNS provider

Note: MX priority values are preference weights — lower = higher priority. To force immediate inbound to a backup MX you can lower its preference value or temporarily remove the primary MX.

Option B — Add a temporary MX pointing to a known-good external relay

If no secondary exists, add one immediately with a short TTL and point to a hosted SMTP relay (SendGrid, Mailgun, Postmark, or a spare server outside Cloudflare/AWS).

Route53 example — change MX via AWS CLI

cat > /tmp/route53-change.json <<'JSON'
{
  "Comment": "Temporary MX for outage",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "example.com.",
        "Type": "MX",
        "TTL": 60,
        "ResourceRecords": [ { "Value": "10 altmail.example.net." } ]
      }
    }
  ]
}
JSON

aws route53 change-resource-record-sets --hosted-zone-id Z12345ABCDEF --change-batch file:///tmp/route53-change.json

Cloudflare API example — create MX record

curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"type":"MX","name":"example.com","content":"altmail.example.net","priority":10,"ttl":60}'

Verify quickly with dig from multiple resolvers:

dig @8.8.8.8 example.com MX +short
dig @1.1.1.1 example.com MX +short

Option C — For outbound mail: change relayhost to alternative SMTP provider

When your outbound smart host (e.g., SES or a CDN-protected host) is down, configure your MTA to use a secondary relay.

Postfix example: set relayhost and restart

# /etc/postfix/main.cf
relayhost = [smtp-alt.relay.example.net]:587

# add SASL creds (if required)
postmap /etc/postfix/sasl_passwd
systemctl restart postfix
# force queue run
postqueue -f

Exim example: set a secondary_smarthost in config or use manual route

exim -qff   # force queue run after config change
exim -bp    # show queue

Hold ambiguous outgoing retries to avoid bounce storms

If you're uncertain whether messages will be delivered, you can hold outgoing queues temporarily to prevent consumer-facing errors and retry storms:

# Postfix - hold all messages
postsuper -h ALL
# To list and hold individual queue IDs
postqueue -p
postsuper -h 

# Exim - set queue timeouts or use hold flags
exim -bp
exim -Mrm    # remove if necessary

4) Stabilize queues & prevent mail loss

Inspect MTA queues and determine how many messages are pending delivery. Prioritize business-critical senders (billing, security alerts) to be routed first.

Check MTA queues

# Postfix
postqueue -p
# Show number of messages
postqueue -p | tail -n1

# Exim
exim -bp

# Sendmail
mailq

To selectively requeue or prioritize messages, filter by sender/recipient and re-inject via sendmail or postsuper.

Force a retry after establishing alternate route

# Postfix: flush queue
postqueue -f

# Exim
exim -qff

5) Validate inbound and outbound flows

Once DNS changes are live and your relay is configured, perform these validation steps.

DNS propagation checks

# confirm MX seen from multiple public resolvers
dig @8.8.8.8 example.com MX +short
dig @1.1.1.1 example.com MX +short

# check TTL and A/AAAA for MX targets
dig +nocmd altmail.example.net A +noall +answer

SMTP end-to-end tests

# Use swaks (recommended) to test SMTP send and auth
swaks --to ops+test@example.com --server altmail.example.net:587 --auth LOGIN \
  --auth-user smtpuser --auth-password 'smtppass' --tls --from postmaster@example.com

# Alternatively raw telnet test
telnet altmail.example.net 25
EHLO test.example.com
MAIL FROM:
RCPT TO:
DATA
Subject: Test

Test message
.
QUIT

Check TLS and DKIM alignment

openssl s_client -starttls smtp -connect altmail.example.net:587 -crlf
# Check DKIM headers on test message delivered to external mailbox

6) Communicate — status updates and stakeholder messaging

Keep customers and internal teams informed with clear, time‑stamped updates on Statuspage, ticketing systems and support channels. If you use Statuspage/Atlassian, use the API to programmatically update the incident state as you progress — treat your incident updates like a documented publishing workflow (see modular publishing workflows).

7) Failback — safely return to normal when upstream is healthy

Don’t rush failback. Confirm stability and plan a well-orchestrated switch back to the primary provider.

Failback checklist

1. Confirm upstream status green on provider status pages for at least one full TTL interval.
2. Reduce MX priority back to original or re-add original primary MX via DNS API/console.
3. Monitor both MX endpoints for acceptance of queued mail and for errors.
4. Drain secondary queues and ensure DKIM signatures remain valid.
5. Revert MTA relayhost to original if you changed it; rotate credentials if necessary.
6. Document timeline and post‑incident analysis.

Example: Revert MX in Route53 to primary

aws route53 change-resource-record-sets --hosted-zone-id Z12345ABCDEF --change-batch file:///tmp/revert-mx.json
# revert-mx.json contains the original MX set with TTL restored

Validate with dig from global resolvers and watch Mail Transfer Agent logs for incoming connections.

8) Post-incident: root cause, improvements and automation

Once the incident is closed, conduct a blameless postmortem and prioritize improvements.

• Short-term fixes: Add secondary MX records, preconfigure alternate SMTP relays, document exact change commands and stored API tokens in your secrets manager.
• Medium-term: Implement automated DNS failover with health checks (Route53 health checks or third-party DNS failover) and verify TTL strategy (pre‑lower TTL before maintenance windows).
• Long-term: Adopt multi‑provider architecture: independent DNS providers, multiple SMTP vendors, and cross‑region redundancy for SES or SMTP stacks. Balance resilience with your cloud cost optimization strategy when adding providers.

Recommended configuration checklist for resilience

• At least one independent secondary MX hosted outside your primary DNS/CDN provider.
• Documented DNS change commands for each DNS provider (Cloudflare API, Route53 CLI, other panels).
• Pre-authorized alternate SMTP relays with credentials stored in HashiCorp Vault / AWS Secrets Manager.
• Automated Statuspage updates and alerting runbooks integrated with incident tooling.

Advanced strategies and 2026 trends

As of 2026, several trends shape how you should prepare and respond:

• Increased adoption of automated DNS failover products and BGP-based rerouting. These reduce manual changes but require careful testing to avoid split-brain scenarios.
• Wider use of “MTA-as-a-Service” (SendGrid, Postmark, Mailgun) as deliberate secondary relays to ensure outbound continuity.
• Cloudflare and AWS now offer more advanced connectivity products (Cloudflare Spectrum, AWS Global Accelerator) that can protect SMTP in enterprise plans — but these also make dependency analysis crucial.
• Regulatory and compliance pressure (privacy and cross-border data flows) through 2025–2026 requires that secondary providers meet data residency and logging requirements.

Risk tradeoffs

Every mitigation increases complexity: more providers mean more configuration to secure and audit. Maintain runbooks and test failovers quarterly — for example combine runbook docs with your observability practices described in observability for workflow microservices — to keep confidence high.

Quick reference: Common commands

• dig MX:

  dig @8.8.8.8 example.com MX +short

• Check SPF/DKIM/DMARC:

  dig example.com TXT +short
  dig selector._domainkey.example.com TXT +short
  dig _dmarc.example.com TXT +short

• SMTP TLS test:

  openssl s_client -starttls smtp -connect mail.example.com:587 -crlf

• Postfix queue:

  postqueue -p
  postsuper -h ALL  # hold all
  postsuper -r ALL  # release all
  postqueue -f      # force flush

• Route53 MX change:

  aws route53 change-resource-record-sets --hosted-zone-id Z... --change-batch file:///tmp/change.json

• Cloudflare create MX (API):

  curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" -H "Authorization: Bearer $CF_API_TOKEN" -H "Content-Type: application/json" --data '{"type":"MX","name":"example.com","content":"altmail.example.net","priority":10,"ttl":60}'

• Test SMTP send:

  swaks --to ops+test@example.com --server altmail.example.net:587 --auth LOGIN --auth-user user --auth-password pass --tls --from postmaster@example.com

Real‑world example (short case study)

During a January 2026 Cloudflare control‑plane outage, one mid‑market SaaS company saw inbound MX records unresolvable in some regions. They executed a preplanned runbook:

1. Incident Lead published Statuspage and spun up incident Slack channel.
2. DNS Lead used stored Cloudflare API token to add a temporary MX record pointing to a Mailgun relay and set TTL to 60s.
3. MTA Lead changed Postfix relayhost for outbound to SendGrid and forced queue flush for priority messages.
4. They validated with swaks and watched queue counts drop as remote MTAs connected to the alternate MX.
5. After Cloudflare reported full recovery and MX responses were stable for 10 minutes at multiple resolvers, they reverted DNS and monitored logs for 24 hours.

The keys to success were preprovisioned alternate relays, stored API tokens, and test‑drilled runbooks — many teams now keep those runbooks in visual doc tools like Compose.page so changes are repeatable and auditable.

Final takeaways — act decisively, automate diligently

• Prepare now: Preconfigure secondary MX, alternate relays, and store DNS API commands in a secured runbook.
• During incident: Prioritize inbound continuity first (MX failover), then outbound (relayhost), then queue handling and monitoring.
• After incident: Conduct a blameless postmortem and invest in automation and cross‑provider redundancy.

In 2026, incidents will keep happening — the differentiator is how fast and confidently your team executes a tested runbook.

Call to action

If you don't have a drilled MX failover playbook yet, download our ready-to-use Incident Runbook template and DNS change snippets for Cloudflare and Route53, and sign up for a quarterly failover test alert. Want help tailoring it to your environment? Contact our engineering team for a runbook review and a live tabletop exercise.

Related Reading

• Building a Resilient Freelance Ops Stack in 2026
• Advanced Strategy: Channel Failover, Edge Routing and Winter Grid Resilience
• Observability for Workflow Microservices — 2026 Playbook
• Compose.page for Cloud Docs — Visual Editing Meets Infrastructure Diagrams
• Make Your Smartphone a Film Festival: Sound Packs from EO Media’s Cannes-Worthy Titles
• How to Audit Torrents for Licensed IP Before Publishing: A Practical Workflow
• Value Arbitrage: When Bookmaker Odds Diverge from Model Probabilities
• Affordable In-Store Upgrades: How a Smart Lamp and a Mac mini Can Transform a Small Jewelry Shop
• Protecting Email-to-Link Attribution When Gmail AI Summarizes Threads