We shipped multi-channel alerting, then realized we couldn't verify it without waiting for a real outage. Here's the endpoint that fixed it.

A few weeks ago we shipped multi-channel SRE notifications — when something fails in production, the system fans out to: (1) an in-app inbox, (2) admin emails, (3) a Teams channel. Standard severity-routing: info goes inbox-only; warning and above fan out to all three.

Spec 039 was clean. The fan-out worked in dev. We shipped it.

Then a week later, ops set the real TEAMS_WEBHOOK_URL in production via the admin panel. Question: does the production Teams integration actually work?

There was no way to verify without waiting for a real failure. We knew — from past experience — that "we'll find out when something breaks" is a lie. Integrations go dark silently. You discover the breakage at the worst possible moment, which is when you're already in incident response.

Spec 039 had explicitly de-scoped a heartbeat / test-fire endpoint, on the theory that real failures would surface the integration. In practice, the integration goes dark until something genuinely breaks — by which time it's too late to confirm whether ops will actually receive the alert.

This is the endpoint we added to fix that, the reasoning behind every design choice, and why a "simulate" endpoint beats a "mock" approach.

The endpoint

POST /api/admin/sre/simulate. Admin-only. Fires a synthetic event through the real fan-out path with a [SIMULATION] title prefix.

# inbox-only ping (info severity)
curl -X POST https://api.example.com/api/admin/sre/simulate \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "severity": "info",
    "type": "system_config_changed",
    "title": "Heartbeat ping",
    "body": "verifying alerting integration"
  }'

# full fan-out (inbox + email + Teams)
curl -X POST https://api.example.com/api/admin/sre/simulate \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "severity": "warning",
    "type": "system_queue_backlog_high",
    "title": "Pipeline test",
    "body": "Verifying Teams + email fan-out"
  }'

Response:

{
  "ok": true,
  "type": "system_queue_backlog_high",
  "severity": "warning",
  "title_sent": "[SIMULATION] Pipeline test",
  "channels": ["inbox", "email", "teams"]
}

Expected effect of the second call: an Adaptive Card lands in #your-alerts-channel titled [SIMULATION] Pipeline test. An email reaches every admin user with subject [WARNING] [SIMULATION] Pipeline test. A row appears in each admin's notifications inbox.

If any of those don't happen, you've found the breakage now, not during an incident.

Three design choices worth naming

1. Same fan-out path as a real event. No mocks. No separate transport.

The temptation is to build a "test mode" that mocks the Teams API call. Don't. A mock proves the mock works, not that the integration works.

The simulate endpoint calls the same notify.SendSRE() function the rest of the codebase calls when something breaks. If the Teams URL is misconfigured, the simulation will surface it. If the email service is down, the simulation will surface it. A successful simulation proves the production wiring is intact — that's only true because the path is identical to a real event.

The only difference is the [SIMULATION] title prefix, which lets receivers tell apart "we're testing" from "something is on fire."

2. Whitelist severities and types. Don't accept arbitrary strings.

var allowedSeverities = map[string]notify.Severity{
    "info":     notify.Info,
    "warning":  notify.Warning,
    "error":    notify.Error,
    "critical": notify.Critical,
}

var allowedTypes = map[string]notify.Type{
    string(notify.SystemProviderKeyInvalid):  notify.SystemProviderKeyInvalid,
    string(notify.SystemQueueBacklogHigh):    notify.SystemQueueBacklogHigh,
    string(notify.SystemPaymentFailureRate): notify.SystemPaymentFailureRate,
    // ... ~10 known types
}

A typo in the type field shouldn't pollute the inbox with non-canonical type values. The whitelist surfaces typos as 400s. Worth the 20 lines of map-init.

3. No rate limiting. Admin auth IS the rate limit.

Spec 039 follow-up briefly considered adding per-type cooldowns. We dropped it. Admin auth is the gate. If an admin is rage-clicking the simulate button during an actual incident, they have bigger problems than a slightly noisy #alerts channel. Don't optimize for adversarial use of the simulate endpoint by your own team — there's no threat there.

What we explicitly chose NOT to build

• No UI button on the admin page. curl-only for v1. UI is a follow-up if curl turns out to be annoying. Usually it doesn't.
• No "simulate everything" sweep mode that fires one of every severity in sequence. That's easy to script externally with curl.
• No Severity: Info follow-up auto-emission (acknowledging recovery). The caller fires what they want.
• No storing simulated events separately from real ones. The [SIMULATION] prefix is the only marker, on purpose, so the fan-out path is bit-for-bit identical to a real event.

The runbook this enables

# every Monday, ops runs:
curl ... '{"severity":"info","type":"system_config_changed","title":"Weekly heartbeat","body":"..."}'

# expect: row in each admin's inbox within 5 seconds.
# if not → integration broken, investigate before anything actually fails.

This is the "synthetic transaction" pattern from observability practice, applied to the alerting layer specifically. Most teams have synthetic transactions for their core product flows (login → dashboard → action). Almost no teams have synthetic transactions for their alerting infrastructure itself. The alerting layer is the one you find broken precisely when you need it most.

Catch it on a Monday morning, not on a Saturday at 2 AM.

The bigger principle

Anything you only exercise during failure will eventually fail to exercise during failure.

The simulate endpoint is one application. Others worth considering:

• A /api/admin/migration/dry-run against a recent migration set — catches schema drift before deploy.
• A /api/admin/credentials/probe that loads every secret your app needs and reports which ones are missing (without exposing them).
• A /api/admin/external/healthcheck that pings every external dependency and surfaces the matrix.

These are five-minute endpoints. They prevent the class of bug where you discover the breakage during an incident.

Where the rest lives

More notes on alerting hygiene, SRE patterns, and engineering clarity at: 🔗 deemwar.com

— deemwar

References

• Google SRE — Site Reliability Engineering, Chapter 6: Monitoring Distributed Systems (2016). sre.google/sre-book
• Google SRE — The Site Reliability Workbook, Chapter 5: Alerting on SLOs (2018). sre.google/workbook
• Honeycomb — Engineering blog: testing, observability, and production debugging (ongoing). honeycomb.io/blog
• Netflix — Chaos Monkey (2012–). Netflix/chaosmonkey on GitHub
• Casey Rosenthal & Nora Jones — Chaos Engineering: System Resiliency in Practice (O'Reilly, 2020). oreilly.com
• Forsgren, Humble & Kim — Accelerate: The Science of Lean Software and DevOps (IT Revolution, 2018). itrevolution.com