Database Monitoring Playbook: Smart Alerts from Queries to Triage

Published Feb 28, 2026

Build low-noise database monitoring with query-based alerts, secure read-only access, and clear triage steps for SQL and NoSQL teams.

Database Monitoring Playbook: Smart Alerts from Queries to Triage

Database monitoring is often treated as “check CPU, watch disk, hope for the best.” That approach catches outages late and creates noisy alerts that teams learn to ignore. A more reliable pattern is to monitor what your database means to the business: failed payments, stuck jobs, rising error rates, replication lag, or rapidly growing tables—using queries that express those conditions.

This playbook shows how to design query-driven monitoring for SQL and NoSQL databases, reduce alert fatigue, and build a consistent triage workflow. You’ll leave with practical alert patterns, example queries, and a checklist you can adapt to production.

1) Start with outcomes, not metrics

Infrastructure metrics (CPU, memory, I/O) matter, but they are rarely the earliest signal of a user-visible problem. Begin with outcomes that map to incidents:

  • Availability: Can the app read/write critical data?
  • Freshness: Is data up to date (ETL, sync, queues, replicas)?
  • Correctness: Are “impossible” states appearing (negative inventory, duplicate orders)?
  • Performance: Are key queries breaching latency/timeout budgets?
  • Growth: Are tables/collections growing beyond expectations?

“Monitor what users feel, then prove it with data.”

Outcome-first monitoring keeps your alerts actionable because every alert answers: what’s broken, who is impacted, and what to do next.

2) Choose the right alert type

Most query-based alerts fall into four categories. Mixing them intentionally prevents both blind spots and noise.

Threshold alerts

Trigger when a value crosses a boundary (e.g., error rate > 2%, queue depth > 10,000). Use when there’s a clear “too high/too low.”

Change alerts

Trigger when something changes state (e.g., an order becomes FAILED, a feature flag flips). Use when you care about discrete events.

Absence alerts

Trigger when expected data does not appear (e.g., no successful payments in 15 minutes, no new events in an ingestion table). These catch silent failures.

Anomaly alerts (lightweight)

Trigger on deviation from a baseline (e.g., today’s signups are 60% below last week’s weekday average). Use sparingly; they’re powerful but can be noisy without context.

3) Design alerts to be low-noise and high-action

Every alert should include:

  1. Signal query: a read-only query that returns a small result (count, max age, rate, status list).
  2. Condition: the rule (e.g., count > 0, max_age_minutes > 10).
  3. Context payload: enough detail to triage (IDs, timestamps, top offenders).
  4. Runbook link: one page with “check this, then that.”
  5. Routing: who gets paged vs emailed, based on severity.

Two practical tips:

  • Add a debounce: alert only after N consecutive failures or for a duration (e.g., “breached for 10 minutes”).
  • Prefer counts over full rows, but include a small sample for debugging (e.g., 5 IDs).

4) Practical query patterns (SQL and NoSQL)

The following examples assume you run them on a schedule (every 1–15 minutes depending on criticality) and notify when the condition is met.

Pattern A: “Stuck jobs” (PostgreSQL / MySQL)

Detect background work that has not progressed.

-- Jobs running too long (Postgres-compatible SQL)
SELECT
  COUNT(*) AS stuck_jobs,
  MAX(EXTRACT(EPOCH FROM (NOW() - started_at)) / 60) AS max_age_minutes
FROM jobs
WHERE status = 'RUNNING'
  AND started_at < NOW() - INTERVAL '15 minutes';

Alert condition: stuck_jobs > 0. Include a follow-up query for top 5 job IDs to speed triage.

Pattern B: “No recent successes” (absence alert)

Catch silent failures where nothing is happening (often more dangerous than explicit errors).

-- No successful checkouts in the last 10 minutes
SELECT
  COUNT(*) AS successful_checkouts
FROM orders
WHERE status = 'PAID'
  AND paid_at > NOW() - INTERVAL '10 minutes';

Alert condition: successful_checkouts = 0 AND traffic is expected (optionally add a second query to confirm inbound requests are non-zero).

Pattern C: “Error-rate spike” (SQL)

Monitor an application events table or audit log.

-- Error rate in last 5 minutes
WITH recent AS (
  SELECT level
  FROM app_events
  WHERE created_at > NOW() - INTERVAL '5 minutes'
)
SELECT
  SUM(CASE WHEN level = 'ERROR' THEN 1 ELSE 0 END) AS errors,
  COUNT(*) AS total,
  ROUND(100.0 * SUM(CASE WHEN level = 'ERROR' THEN 1 ELSE 0 END) / NULLIF(COUNT(*),0), 2) AS error_pct
FROM recent;

Alert condition: error_pct > 2 and total > 200 (minimum volume prevents tiny-sample noise).

Pattern D: “Replication lag” (PostgreSQL)

Replication lag can look like “everything is fine” until reads are stale.

-- Approximate replay lag on replicas (run on primary)
SELECT
  application_name,
  EXTRACT(EPOCH FROM (NOW() - pg_last_xact_replay_timestamp())) AS replay_lag_seconds
FROM pg_stat_replication;

Alert condition: any replica replay_lag_seconds > 60 for 5+ minutes.

Pattern E: “Suspicious documents” (MongoDB)

In NoSQL, you often alert on malformed states or missing fields that should exist after a workflow step.

// Orders stuck without a shipment after 2 hours
db.orders.countDocuments({
  status: "PAID",
  shippedAt: { $exists: false },
  paidAt: { $lt: new Date(Date.now() - 2*60*60*1000) }
});

Alert condition: count > 0, and optionally sample a few IDs for context.

5) Scheduling: frequency, cost, and freshness

Scheduling is part of alert design. Too frequent wastes resources; too slow delays response. Use tiers:

  • Tier 0 (paging): every 1–2 minutes (payments, auth, writes failing).
  • Tier 1 (urgent): every 5 minutes (queue backlogs, replication lag).
  • Tier 2 (email/slack): every 15–60 minutes (growth, anomalies, data quality drift).

Also consider query cost:

  • Prefer indexed filters on time ranges (e.g., created_at).
  • Aggregate (COUNT/MAX) rather than scanning full records.
  • Keep results small; return IDs only for the “top N offenders.”

6) A simple alert catalog (use as a template)

Alert Type Example signal Notify when Channel
Failed payments detected Change/Threshold COUNT(*) FROM payments WHERE status='FAILED' AND created_at > now()-5m > 0 for 2 checks Push/Page
Queue backlog Threshold COUNT(*) FROM jobs WHERE status='QUEUED' > 10k for 10 min Push/Page
No new events ingested Absence MAX(created_at) FROM ingestion_events Age > 15 min Push
Table growth spike Anomaly pg_total_relation_size('events') +30% day-over-day Email

7) Security and safety for query-based monitoring

Monitoring should not increase your attack surface. Adopt these defaults:

  • Read-only credentials with least privilege (ideally restricted schemas/views).
  • Encrypted connections (TLS) and certificate verification where supported.
  • Parameterize queries in any tooling that accepts input, to avoid injection risks.
  • Avoid sensitive payloads in notifications: send IDs, not PII; link to internal dashboards for details.
  • Audit the monitoring user: log connections and query execution if feasible.

If you must monitor multiple environments, keep credentials and routing isolated (prod alerts should never be triggered by staging data).

8) Triage: turn alerts into a repeatable workflow

A good alert wakes you up with a clear next step. A minimal runbook section for each alert:

  1. Confirm: rerun the signal query; check if it’s sustained.
  2. Scope: identify affected tenants/regions/services.
  3. Correlate: recent deploys, schema changes, job runners, network events.
  4. Mitigate: rollback, scale workers, pause a queue, fail over, or apply a safe index.
  5. Resolve: fix root cause, then add/adjust an alert to catch it earlier next time.

Common causes to check quickly

  • Connection pool exhaustion (app-side)
  • Lock contention / long transactions (DB-side)
  • Missing or unused indexes after query changes
  • Replica lag after spikes or maintenance
  • Retention jobs failing, causing unbounded growth

9) Avoid these database monitoring anti-patterns

  • Alerting on raw CPU alone: use it as context, not a primary page.
  • No minimum volume: percentages without denominators create noise.
  • Paging for “might be a problem”: route low-confidence signals to email.
  • One mega-query: break monitoring into small, targeted checks to simplify debugging.
  • Ignoring absence alerts: “nothing happened” is often the earliest incident indicator.

10) Quick checklist to implement this week

  • Pick 5 critical outcomes (payments, sign-in, job processing, replication freshness, data ingestion).
  • Write one read-only query per outcome that returns a small numeric signal.
  • Add thresholds with minimum volume + debounce.
  • Route: page for Tier 0–1; email for Tier 2.
  • Create a one-page runbook for each alert.
  • Review weekly: drop noisy alerts, tighten conditions, add missing context.

When you treat queries as first-class monitoring signals, database monitoring becomes clearer and more actionable: you’re no longer watching systems indirectly—you’re watching the truth your application depends on.

If you like managing these kinds of scheduled, read-only checks from an iPhone with push or email notifications, Query Alert is one option to explore.