Custom Email Alerts and Escalations: Advanced Incident Routing

The Escalation Problem#

It's 3 AM. Your website goes down. You need someone to respond immediately.

Without escalation:

• Alert fires to on-call Slack channel
• Nobody sees it (everyone's asleep, Slack notifications off)
• 45 minutes later, customer complaints wake someone up
• MTTR: 45 minutes

With smart escalation:

• Alert fires to Slack (low priority)
• If no acknowledgment within 2 minutes → SMS to on-call primary
• If no response within 5 minutes → Call on-call backup
• MTTR: 5 minutes

This guide shows you how to build intelligent escalation workflows.

Understanding Incident Severity#

First, classify incidents by severity:

Tier 1: Critical (Page Immediately)#

• Production website down (revenue-impacting)
• Payment gateway failure
• API returning 5xx errors
• Database replication down

Actions:

• Slack #critical-incidents (always monitored)
• SMS to primary on-call
• Phone call if no SMS response
• Automatic Jira ticket creation
• Status page update

Tier 2: Warning (Page During Business Hours)#

• Response time degradation
• Non-critical service errors
• Email deliverability issues
• Database query slowness

Actions:

• Slack #alerts (checked during business hours)
• Jira ticket creation
• Email digest at end of day

Tier 3: Info (Log Only)#

• Domain expiry in 30 days
• SSL expiry in 90 days
• Weekly trend report
• Non-critical metric threshold

Actions:

• Weekly email digest
• Dashboard notification (no page)

Building Your Escalation Policy#

Step 1: Define On-Call Rotations#

Create a rotation showing who's on-call when:

Monday-Friday 9 AM - 5 PM: Alice (primary), Bob (backup)
Monday-Friday 5 PM - 9 AM: Charlie (primary), Diana (backup)
Saturday-Sunday 24 hrs: Eve (primary), Frank (backup)
Holidays: George (on-call all day)

Step 2: Define Escalation Times#

T+0:    Alert fires to Slack
T+2min: No acknowledgment → SMS to primary
T+5min: No response → Call primary
T+10min: No response → SMS to backup
T+15min: Still no response → Page full team

Step 3: Implement Escalation Logic#

In Nova Uptime (if supported):

1. Domain settings → Alerting
2. Set severity: Critical
3. Configure escalation:
   • Step 1: Slack #critical-incidents
   • Step 2 (2 min): SMS to on-call
   • Step 3 (5 min): Phone call
   • Step 4 (10 min): All team

Via Webhook + Custom System:

async function handleCriticalIncident({ domain, detectedAt }) {
  const oncall = await getOnCallEngineer(new Date());

  // Step 1: Immediate Slack alert
  const slackMessage = await postToSlack({
    channel: '#critical-incidents',
    text: `🚨 CRITICAL: ${domain} is down`,
    blocks: [
      {
        type: 'section',
        text: {
          type: 'mrkdwn',
          text: `@${oncall.slackHandle}: ${domain} is down. Acknowledge with reaction.`
        }
      }
    ]
  });

  // Step 2: Wait 2 minutes for acknowledgment
  const acknowledged = await waitForAcknowledgment(slackMessage, 2 * 60 * 1000);

  if (!acknowledged) {
    // Step 2: Send SMS
    await sendSMS({
      to: oncall.phone,
      message: `CRITICAL: ${domain} down. Reply OK to acknowledge.`
    });
  }

  // Step 3: Wait 5 minutes total
  const smsAcknowledged = await waitForSMS(oncall.phone, 5 * 60 * 1000);

  if (!smsAcknowledged) {
    // Step 3: Phone call
    await makePhoneCall({
      to: oncall.phone,
      message: `Critical incident. Website ${domain} is down. Press 1 to acknowledge.`
    });
  }

  // ... Continue escalation chain
}

Advanced: Smart Alert Routing#

Pattern 1: Route by Time of Day#

Different people on-call at different times.

async function getOnCallEngineer(timestamp) {
  const hour = new Date(timestamp).getHours();
  const dayOfWeek = new Date(timestamp).getDay();

  // Business hours (9 AM - 5 PM weekdays)
  if (dayOfWeek >= 1 && dayOfWeek <= 5 && hour >= 9 && hour < 17) {
    return {
      name: 'Alice',
      slackHandle: 'alice',
      phone: '+1-555-0100',
      email: 'alice@company.com'
    };
  }

  // After hours (weekdays)
  if (dayOfWeek >= 1 && dayOfWeek <= 5 && (hour < 9 || hour >= 17)) {
    return {
      name: 'Charlie',
      slackHandle: 'charlie',
      phone: '+1-555-0102',
      email: 'charlie@company.com'
    };
  }

  // Weekend
  if (dayOfWeek === 0 || dayOfWeek === 6) {
    return {
      name: 'Eve',
      slackHandle: 'eve',
      phone: '+1-555-0104',
      email: 'eve@company.com'
    };
  }
}

Pattern 2: Route by Domain#

Different teams own different domains.

async function getTeamForDomain(domain) {
  // Engineering team owns api.*, backend.*
  if (domain.startsWith('api.') || domain.startsWith('backend.')) {
    return 'engineering';
  }

  // Infrastructure team owns server, monitoring, infra
  if (domain.includes('server') || domain.includes('monitor')) {
    return 'infrastructure';
  }

  // Support team owns customer-facing domains
  if (domain.startsWith('support.') || domain.startsWith('customer.')) {
    return 'support';
  }

  // Default: DevOps
  return 'devops';
}

async function handleIncident({ domain, severity }) {
  const team = await getTeamForDomain(domain);
  const oncall = await getOnCallEngineer(team, new Date());

  if (severity === 'critical') {
    await escalate(oncall);
  }
}

Pattern 3: Route by Incident Type#

Different expertise for different failures.

async function getExpertForIncident(domain, incidentType) {
  if (incidentType === 'database_down') {
    // Page database expert
    return await getOnCallExpert('database');
  } else if (incidentType === 'api_errors') {
    // Page API lead
    return await getOnCallExpert('backend');
  } else if (incidentType === 'email_delivery_failing') {
    // Page email ops
    return await getOnCallExpert('email');
  } else if (incidentType === 'ssl_expired') {
    // Page security
    return await getOnCallExpert('security');
  }

  // Default: on-call engineer
  return await getOnCallEngineer(new Date());
}

Pattern 4: Conditional Escalation#

Different escalation paths based on incident properties.

async function escalateIncident({ domain, severity, duration }) {
  const oncall = await getOnCallEngineer(new Date());

  if (severity === 'critical' && duration > 5 * 60 * 1000) {
    // Critical for >5 minutes: Aggressive escalation
    await Promise.all([
      postSlack({ channel: '#critical-incidents', text: 'CRITICAL ESCALATION' }),
      sendSMS(oncall.phone),
      makePhoneCall(oncall.phone),
      pageBackup(oncall.backup)
    ]);
  } else if (severity === 'critical') {
    // Critical but recent: Gentle escalation
    await Promise.all([
      postSlack({ channel: '#critical-incidents' }),
      sendSMS(oncall.phone)
    ]);
  } else if (severity === 'warning') {
    // Just log
    await postSlack({ channel: '#alerts' });
  }
}

Integration with PagerDuty#

For sophisticated on-call management, integrate with PagerDuty:

const PagerDutyClient = require('pagerduty');

async function pageOnCallVia PagerDuty(domain, severity) {
  const client = new PagerDutyClient({
    token: process.env.PAGERDUTY_TOKEN
  });

  // Create incident
  const incident = await client.incidents.create({
    type: 'incident_reference',
    incident: {
      type: 'incident',
      title: `${domain} is down`,
      body: {
        type: 'incident_body',
        description: `Website ${domain} is down. Response: Down. Severity: ${severity}`
      },
      urgency: severity === 'critical' ? 'high' : 'low',
      service: {
        id: process.env.PAGERDUTY_SERVICE_ID,
        type: 'service_reference'
      }
    }
  });

  console.log(`Created PagerDuty incident: ${incident.id}`);
}

Acknowledgment and Handoff#

Acknowledgment Patterns#

On-call engineer must acknowledge incident:

// Via Slack reaction
async function waitForSlackAcknowledgment(messageId, maxWait) {
  const startTime = Date.now();

  while (Date.now() - startTime < maxWait) {
    // Poll for reactions
    const reactions = await getSlackMessageReactions(messageId);

    if (reactions.includes('white_check_mark')) {
      return true; // Acknowledged
    }

    await sleep(10 * 1000); // Check every 10 seconds
  }

  return false; // Not acknowledged within max wait
}

// Via SMS
async function waitForSmsAcknowledgment(phone, maxWait) {
  const startTime = Date.now();

  while (Date.now() - startTime < maxWait) {
    // Poll SMS responses
    const responses = await getSmsResponses(phone);

    if (responses.some(m => m.text.toUpperCase().includes('OK'))) {
      return true; // Acknowledged
    }

    await sleep(5 * 1000); // Check every 5 seconds
  }

  return false; // Not acknowledged
}

Handoff Between Teams#

When one on-call engineer needs to handoff to another:

async function handoffIncident(incident, fromEngineer, toEngineer) {
  // Update incident
  incident.assignedTo = toEngineer;
  incident.handoffAt = new Date();
  await incident.save();

  // Notify both
  await sendMessage({
    to: fromEngineer.slack,
    text: `Handing off ${incident.domain} to ${toEngineer.name}`
  });

  await sendMessage({
    to: toEngineer.slack,
    text: `Taking over incident: ${incident.domain}. See: ${incident.dashboard}`
  });

  // Update status page
  await updateStatusPage({
    message: `Working with ${toEngineer.name}'s team on investigation`
  });
}

Measuring Escalation Effectiveness#

Track escalation metrics:

async function analyzeEscalationMetrics() {
  const incidents = await Incident.find({
    createdAfter: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000) // Last 30 days
  });

  const metrics = {
    totalIncidents: incidents.length,
    avgTimeToAcknowledgment: calculateAvg(
      incidents.map(i => i.acknowledgedAt - i.alertedAt)
    ),
    avgTimeToEscalation: calculateAvg(
      incidents.map(i => i.escalatedAt - i.alertedAt)
    ),
    escalationRate: incidents.filter(i => i.escalatedAt).length / incidents.length,
    avgMTTR: calculateAvg(
      incidents.map(i => i.resolvedAt - i.alertedAt)
    )
  };

  console.log('Escalation Metrics (Last 30 days):');
  console.log(`Total Incidents: ${metrics.totalIncidents}`);
  console.log(`Avg Time to Acknowledgment: ${metrics.avgTimeToAcknowledgment / 60}m`);
  console.log(`Escalation Rate: ${(metrics.escalationRate * 100).toFixed(1)}%`);
  console.log(`Avg MTTR: ${metrics.avgMTTR / 60}m`);
}

Common Escalation Mistakes#

Mistake 1: Alert Fatigue Leading to Dismissal#

Problem: Too many alerts → team stops responding → real issues missed

Solution: Use strict thresholds. Only page for truly critical issues.

Mistake 2: Escalating Too Aggressively#

Problem: Page entire team on every incident → burnout → people leave

Solution: Escalate gradually. Give primary 5 minutes before paging backup.

Mistake 3: Escalating Too Slowly#

Problem: Critical incident goes unnoticed for 30 minutes → massive impact

Solution: For critical incidents, escalate within 2-5 minutes.

Mistake 4: No Handoff Process#

Problem: Primary on-call didn't know they were being handed off → chaos

Solution: Explicitly communicate handoffs via Slack/email.

Summary: Escalation Setup Checklist#

• ✅ Define incident severity tiers (Critical/Warning/Info)
• ✅ Create on-call rotation schedule
• ✅ Define escalation timing (2 min → 5 min → 10 min → all)
• ✅ Set up escalation channels (Slack → SMS → Phone → all-hands)
• ✅ Configure routing by time of day
• ✅ Route by domain/team ownership
• ✅ Route by incident type (database vs API vs email)
• ✅ Integrate with PagerDuty (if applicable)
• ✅ Set up acknowledgment mechanisms (Slack reaction, SMS response)
• ✅ Define handoff procedures
• ✅ Track escalation metrics
• ✅ Monthly review of escalation effectiveness

Get Started Today#

Start simple: Just Slack + SMS. Add complexity (PagerDuty, conditional routing) as your incident volume grows.

Document your escalation policy in your team wiki. Share with all team members. Test quarterly to ensure everything still works.

Your escalation policy is the difference between "incident resolved in 5 minutes" and "outage lasted 3 hours." Invest in getting this right.