Overview

Diagnostics lets an administrator measure, from the Web UI, where a slow ToBeAlert deployment is losing time. One click runs a set of timed probes on the server — the real alert queries with their execution plans, table and index health, PostgreSQL settings, connections, cache and transactions, Redis, the Celery workers, the alert ingest pipeline, recent notifications, disk space, clock skew and the backend process — and returns a report with a verdict per probe, the thresholds it was judged against, the raw details, and a one-line summary that names the likely bottleneck.

Every run is stored (the last 50) so you can compare a deployment before and after a change, and any report can be downloaded as JSON to send to support.

The same probes also power two unattended checks: Self-monitoring runs the light probes on a schedule and raises ToBeAlert alerts when one fails, and the external health endpoint (GET /api/health) lets an outside monitor poll the backend without a login.

Menu	Management → Diagnostics
Who	Administrators (API scope admin:management)
Typical use	Customers running ToBeAlert against their own PostgreSQL server, where the database host, sizing and tuning are outside the ToBeAlert containers
Cost	A run takes a few seconds and reads the real tables once (plus a second pass for the execution plans). It does not change data or settings

Also on the Status page

Administrators (admin:management) also see a Performance diagnostics section at the bottom of the public Status page (/status), with the same controls, history and download. Visitors without that permission see nothing there and no diagnostics request is made.

When to use it

• The alert dashboard or the history page loads slowly and you want to know whether the time goes to the database, to the PostgreSQL server itself, to Redis / the workers, or to the network between the browser and ToBeAlert.
• After a change — a PostgreSQL upgrade, a postgresql.conf edit, a VACUUM, a migration to another host — to confirm with numbers that it helped.
• Before contacting support: attach the downloaded JSON so support sees the execution plans, settings and sizes without asking for database access.

How to run it

1. Open Management → Diagnostics.
2. Click Run diagnostics. The page shows the report when the server finishes; there is nothing to configure.
3. Read the summary card first, then expand the probes marked slow, critical or error in the table.
4. Use Download JSON to save the full report for support.

Each run is written to the ToBeAlert database and listed in the history at the bottom of the page, and to the audit trail as diagnostics-run (the message is the bottleneck line of the summary).

Reading the summary

The summary card shows:

Element	Meaning
Verdict	The worst verdict of all probes: ok, slow, critical or error (see the scale below)
Likely bottleneck: …	One line naming the area of the worst probe — database query time, PostgreSQL server configuration or load, infrastructure (Redis, Celery, ingest or notifications) or host resources (disk, clock, process) — followed by that probe's own message. With every probe within its threshold it reads No bottleneck found: every probe is within its threshold.
Group chips	One chip per group (database, postgresql, infrastructure, host, plus browser) with the worst verdict of that group
Environment	Release, active hotfix (if any), and the database host taken from DATABASE_URL with the credentials masked
Counts	How many probes ended ok / info / slow / critical / error

Verdict scale

Verdict	Meaning
ok	Within the probe's ok threshold, or no rule triggered
info	Informational only: the probe has no pass/fail rule (API timers, backend process), could not measure (pg_stat_statements not installed) or has nothing to judge yet (too little cache traffic since the statistics reset, no alerts ingested by this process)
slow	Above the ok threshold but within the slow one, or a soft rule triggered (low setting, dead tuples ≥ 20 %, connections ≥ 60 %, cache hit < 95 %, disk < 15 % free…)
critical	Above the slow threshold, or a hard rule triggered (missing index, autovacuum off, no worker heartbeat, connections ≥ 80 %, cache hit < 85 %, disk < 5 % free, clock skew ≥ 30 s…)
error	The probe itself failed (the error text is in the row). The rest of the report is still produced

ok and info weigh the same; slow is worse; critical and error are the worst. The probe with the worst verdict — and, on a tie, the longest duration — is the one the bottleneck line describes.

Browser versus server

The browser group is not measured on the server: the page that runs the diagnostics times three requests from your browser — the lightweight management/gtg round trip, the dashboard list (alerts?page-size=50) and the alert count (alerts/count) — five times each and keeps the median. Next to each one the page shows the server time of the equivalent probe (db.connect, db.alerts_list, db.alerts_count).

• Browser time ≈ server time → the time is spent in ToBeAlert or the database.
• Browser time ≫ server time → the difference is network, TLS termination or a proxy between the browser and ToBeAlert, not the server. Run the diagnostics from another location or from inside the network to confirm.

Browser timings are measured by whoever clicks Run diagnostics, so they vary with the user's location; the server probes do not.

Probes and thresholds

A full run executes 22 probes in four server-side groups — database, postgresql, infrastructure and host — plus the browser group measured by the page. Timed probes get their verdict from two limits in milliseconds: up to the first value is ok, up to the second is slow, above it is critical. Rule-based probes still report their duration and a threshold, but the verdict comes from the rule in the last column.

database

Probe	Id	What it measures	ok / slow (ms)	Rule
Database round trip (SELECT 1)	db.connect	Latency to the database: SELECT 1, median of 5	5 / 50	Time
Dashboard alert list (50 open alerts)	db.alerts_list	The real dashboard query: 50 open alerts (not closed, expired or deleted) by last receive time	300 / 1500	Time
Alert count	db.alerts_count	Count of all alerts	300 / 1500	Time
Counts by severity, status and environment	db.alerts_count_groups	The grouped counts the dashboard header uses	300 / 1500	Time
History list (50 rows)	db.history_list	The real history query: 50 newest history rows	300 / 1500	Time
History count	db.history_count	Count of history rows	300 / 1500	Time
Resource graph ancestors lookup	db.resource_graph	Ancestors (depth 10) of the most recently updated relation in the resource graph	50 / 500	Time
Table sizes, scans, dead tuples, vacuum and indexes	db.stats	Statistics of alerts, alert_history, resource_relations, policies, notification_history and tool_run_history, plus the indexes the shipped schema expects	200 / 2000	slow if a table with more than 1 000 live rows has ≥ 20 % dead tuples, or a table with ≥ 10 000 live rows has more sequential than index scans; critical if dead tuples ≥ 50 % or an expected index is missing
Database size and server version	db.size	Size of the database in MB and the PostgreSQL version() string	200 / 2000	Time

The four list / count probes run the same SQL the Web UI runs, then run it again under EXPLAIN (ANALYZE, BUFFERS) to capture the plan. The probe duration therefore includes the explain pass; the executionMs inside the plan is the query alone. The explain is limited to 30 seconds and the plan text to 20 000 characters (truncated: true when cut).

postgresql

Probe	Id	What it measures	ok / slow (ms)	Rule
Server settings	pg.settings	shared_buffers, effective_cache_size, work_mem, maintenance_work_mem, max_connections, random_page_cost, jit, autovacuum, max_parallel_workers_per_gather, statement_timeout, wal_level, synchronous_commit	200 / 2000	slow if shared_buffers < 256 MB, effective_cache_size < 1 GB or work_mem < 16 MB; critical if autovacuum is off. random_page_cost = 4 adds a note (set 1.1 on SSD storage) without changing the verdict
Connections and locks	pg.activity	Connections to this database, active ones, sessions waiting on a lock, max_connections, and the longest running query in seconds	200 / 2000	slow at ≥ 60 % of max_connections or when any session waits on a lock; critical at ≥ 80 %
Slowest queries (pg_stat_statements)	pg.statements	The 10 queries with the highest total time, with calls, mean and total ms, and rows	500 / 5000	info when the pg_stat_statements extension is not installed; otherwise slow if the worst mean is above 300 ms, critical above 1 500 ms
Buffer cache hit ratio	pg.cache_hit	Share of block reads served from PostgreSQL's shared buffers for this database (pg_stat_database: blks_hit / (blks_hit + blks_read))	200 / 2000	info with fewer than 10 000 block reads since the statistics reset (not enough traffic to judge); slow below 95 %; critical below 85 %
Idle and long transactions	pg.transactions	Sessions idle in transaction, the age of the oldest open transaction (other than the probe's own), and sessions waiting on a lock	200 / 2000	slow if any session is idle in transaction or a transaction has been open for ≥ 60 s; critical if a transaction has been open for ≥ 600 s

infrastructure

Probe	Id	What it measures	ok / slow (ms)	Rule
Redis round trip	redis.ping	PING to Redis, median of 5, plus used and maximum memory	5 / 50	Time
Celery workers and queue	celery.heartbeat	The worker heartbeat stored in Redis (same check as the health endpoint) and the number of queued tasks	—	critical if there is no heartbeat or it is older than 60 s; slow if more than 100 tasks are queued
API timers since start	api.timers	The 20 endpoints with the highest average time since the backend started (count, avgMs, totalMs)	—	info by default; slow if an endpoint called at least 5 times averages more than 1 500 ms
Alert ingest pipeline (per stage)	ingest.pipeline	Average, p95 and maximum time per stage — pre-receive (plugins and policies before the alert is stored), persist (the database write) and post-receive (plugins, notifications and policies after it) — over the last 200 alerts received by this API process	300 / 1500	info when this process has not received any alert yet; otherwise slow if the average total is above 300 ms, critical above 1 500 ms
Notifications in the last hour	notifications.recent	Notification rule executions in the last hour: runs, messages sent, errors, average and maximum duration	200 / 2000	slow if ≥ 20 % of the attempted sends failed; critical if ≥ 50 % failed. ok when nothing was sent

host

Probe	Id	What it measures	ok / slow (ms)	Rule
Disk space	host.disk	Free space on the volumes the backend writes to (/tobealert, the configuration directory, /tmp, the working directory). The PostgreSQL data directory (SHOW data_directory) is measured only when it is reachable from the backend, that is, when the database runs on the same host; otherwise the row says to monitor it from the operating system	200 / 2000	slow if any measured volume has less than 15 % free; critical below 5 %
Clock skew backend vs database	host.clock_skew	Difference between the backend clock and the database clock (SELECT now() compared with the midpoint of the round trip)	200 / 2000	slow at ≥ 2 s of skew; critical at ≥ 30 s
Backend process and host load	host.process	Memory (RSS), CPU and thread count of the API process, plus the host's CPU count, load average and memory	200 / 2000	info by default; slow if the process uses more than 2 GB of RSS

Clock skew matters because relative times, timeouts, maintenance windows and the openFor / silentFor conditions compare backend timestamps with database now(): both hosts should sync with NTP.

Probe details

Expand a row in the probe table to see the details object of that probe. What you find depends on the probe:

Probe	Details
List / count probes	rows, the sql that ran, and explain with the plan text, seqScans (tables read with a sequential scan), planningMs, executionMs, truncated
db.resource_graph	rows returned by the ancestors lookup
db.stats	tables[] with liveRows, deadRows, deadRatio, seqScans, idxScans, totalMb, tableMb, lastVacuum, lastAnalyze and a per-table verdict; expectedIndexes (count), missingIndexes[], notes[], and statsResetAt — scan counters are cumulative since the last statistics reset
db.size	database, sizeMb, version
pg.settings	settings keyed by name with value, unit and, for memory settings, mb; notes[]
pg.activity	connections, active, waitingOnLocks, maxConnections, usageRatio, longestActiveSeconds
pg.statements	available and top[] with query (first 400 characters), calls, meanMs, totalMs, rows
redis.ping	usedMemoryMb, maxMemoryMb (null when Redis has no limit)
celery.heartbeat	workers (the heartbeat message) and queuedTasks
api.timers	timers[] with group, name, count, avgMs, totalMs
pg.cache_hit	hitRatio (0–1), blocksHit, blocksRead
pg.transactions	idleInTransaction, oldestTransactionSeconds, waitingOnLocks
ingest.pipeline	samples, slowestStage, and stages keyed preReceive, persist, postReceive, total, each with avgMs, p95Ms, maxMs
notifications.recent	runsLastHour, sentLastHour, errorsLastHour, errorRatio, avgMs, maxMs
host.disk	volumes[] with path, totalMb, freeMb, freeRatio; database with the same fields for the data directory, or a note when it is on another host or container
host.clock_skew	skewSeconds (positive when the database clock is ahead), backendTime, databaseTime
host.process	rssMb, cpuPercent, threads, hostCpus, hostLoad1, hostLoad5, hostMemoryTotalMb, hostMemoryAvailableMb

A probe that ended in error has an error field with the first line of the exception instead of meaningful details.

Sequential scans are not always a problem

PostgreSQL chooses a sequential scan on purpose for small tables or when a query touches most rows. Treat seqScans in a plan, or a db.stats row with more sequential than index scans, as a problem only on a large table that is also slow, or when missingIndexes is not empty.

Slowest queries: what counts for the verdict

The pg_stat_statements list shows the ten statements with the highest total time, each tagged Application or Maintenance. Only application queries (SELECT, INSERT, UPDATE, DELETE) called at least three times decide the verdict; a manual VACUUM, a CREATE INDEX or the EXPLAIN runs of the diagnostics themselves are listed for information but never flag the probe.

Why "last analyze" does not change when you re-run

The Last vacuum and Last analyze columns are PostgreSQL autovacuum timestamps, not something the diagnostics trigger. Autovacuum re-analyzes a table only after autovacuum_analyze_threshold + autovacuum_analyze_scale_factor × rows have changed (50 + 10 % by default), and vacuums it after a similar number of dead rows. The table shows Changed since analyze as changed / threshold and Next analyze in (changes still needed), so a history table of 14 000 rows that received only 100 new rows keeps its old timestamp, and that is expected. On large history tables lower autovacuum_analyze_scale_factor and autovacuum_vacuum_scale_factor for alert_history (for example 0.02 to 0.05) so statistics and dead rows stay current.

History and comparison

The history table lists the stored runs — date, user, duration, verdict and the bottleneck line — newest first. Only the last 50 runs are kept; older ones are deleted automatically when a new run is stored.

• Load a run to see its full report in the same summary card and probe table.
• Delete a run you no longer need.

To compare before and after a change, run the diagnostics, apply the change, run them again and load the earlier run from the history. Compare the same probe in both: db.alerts_list and db.history_list for query time, db.stats for dead tuples and scans, pg.settings for the server configuration. Keep in mind that db.stats scan counters accumulate since the last statistics reset, so they move only in one direction unless the counters are reset.

If the report could not be stored (for example a database error after the probes ran), the page still shows it and the report carries stored: false; download it before leaving the page.

Downloading for support

Download JSON saves the report currently shown as a .json file. It is the same document the API returns for that run — summary, environment, and every probe with its details, including the SQL, the execution plans and the PostgreSQL settings. Attach it to the support request; it contains the database host name (credentials masked) but no alert content beyond the queries' row counts.

What to do with each finding

Finding	Where it shows	What to do
missingIndexes not empty (critical)	db.stats	Indexes declared by the ToBeAlert schema are absent. Restart the backend so it re-applies the schema (CREATE INDEX IF NOT EXISTS is idempotent), or create the missing index by hand with the definition from support
Sequential scans on a large, slow table	explain.seqScans, db.stats	Confirm the expected indexes exist (above). If they do and the plan still avoids them, run ANALYZE so the planner has fresh statistics, and check random_page_cost
Dead tuples ≥ 20 % / ≥ 50 %	db.stats	Run VACUUM ANALYZE on the table (VACUUM ANALYZE alerts;). If it comes back often, make sure autovacuum is on and consider lowering the autovacuum thresholds for that table
autovacuum is off (critical)	pg.settings	Turn it on in postgresql.conf (autovacuum = on) and reload PostgreSQL
Memory settings below the recommendation	pg.settings	Raise shared_buffers (≥ 256 MB, typically 25 % of RAM on a dedicated server), effective_cache_size (≥ 1 GB, typically 50–75 % of RAM) and work_mem (≥ 16 MB) in postgresql.conf; shared_buffers needs a restart
random_page_cost = 4 on SSD	pg.settings note	Set random_page_cost = 1.1 so the planner prefers index access on SSD storage
Connections ≥ 60 % / ≥ 80 % of max_connections	pg.activity	Find what else uses the server; raise max_connections or put a pooler (for example PgBouncer) in front of PostgreSQL. Sessions waiting on locks point to long transactions — see longestActiveSeconds and the slowest queries
High db.connect with fast queries	db.connect versus the list probes	The database is remote or the network path adds latency: every request pays that round trip several times. Move the database closer to the backend or fix the network path
Slow Redis round trip, no worker heartbeat, queued tasks	redis.ping, celery.heartbeat	Check that the Redis and worker containers are running and reachable from the backend (REDIS_HOST, REDIS_PORT); a stale heartbeat means notifications and background jobs are not being processed
Browser time much higher than server time	browser group	The slowness is between the browser and ToBeAlert: reverse proxy, TLS, VPN or WAN. The server-side probes are fine
Slow endpoint in API timers with fast database probes	api.timers	Note the endpoint and the average and send the report to support; timers accumulate since the last backend start, so one slow period weighs on the average until a restart
Cache hit ratio below 95 % / 85 %	pg.cache_hit	PostgreSQL is reading blocks from disk: the working set does not fit in shared_buffers. Raise shared_buffers (and effective_cache_size) or give the database host more RAM; on a shared host, move other services away. Check pg.settings for the current values. Counters accumulate since the last statistics reset, so a one-off bulk load can weigh on the ratio for a while
Sessions idle in transaction, or a transaction open for minutes	pg.transactions	Another client (a reporting tool, a connection pool with transaction-level pooling, a script) is holding a transaction open. Long transactions block VACUUM and hold locks; find the session in pg_stat_activity and fix the client or set idle_in_transaction_session_timeout on the server. If the only client is ToBeAlert, send the report to support
A volume below 15 % / 5 % free	host.disk	Free space on the backend host (logs, backups, /tmp) or on the PostgreSQL data volume. A full data volume stops PostgreSQL from writing: grow the volume or move the database before it fills. When the database is on another host, the row only reminds you to monitor it from that host's OS
Clock skew ≥ 2 s / ≥ 30 s	host.clock_skew	Synchronise both the backend host and the database host with NTP (chrony, systemd-timesyncd or the cloud provider's time service). Containers use the host clock, so fix it on the hosts, not inside the containers
Ingest average above 300 ms / 1 500 ms	ingest.pipeline	Look at slowestStage. A slow pre-receive or post-receive points to policies, plugins or notification rules doing too much work per alert (external lookups, large condition sets); a slow persist points to the database (see the database probes). Samples are kept per API process, so run the diagnostics on the process that receives alerts
≥ 20 % / ≥ 50 % of notifications failed in the last hour	notifications.recent	Open Notifications → Rules → All executions to read the errors; most failures are an unreachable SMTP server or chat endpoint, expired credentials or a wrong recipient in the method. Fix the method and re-test it
Backend process above 2 GB RSS	host.process	Memory grows with the alert cache and very large API responses. Restart the backend if it keeps growing between restarts and send the report to support with the time since start (uptimeSeconds in GET /api/health)

For postgresql.conf locations and how settings are applied, follow the PostgreSQL documentation for your version; the PostgreSQL image used by the Docker stack reads the file in its data volume.

Self-monitoring

Self-monitoring runs the light probes on a schedule and turns every finding into a normal ToBeAlert alert, so the platform watches its own database, Redis, workers, disk, clock and notifications with the same policies, notification rules, dashboards and views you use for everything else. It is off by default.

Which probes run

Only the probes that are cheap enough to run every few minutes: db.connect, db.alerts_count, pg.activity, pg.cache_hit, pg.transactions, redis.ping, celery.heartbeat, host.disk, host.clock_skew and notifications.recent. The list, resource graph and history queries, the execution plans, pg_stat_statements and the process probe are left to manual runs.

Turning it on

1. Open Management → Backend Config, category Self-monitoring and health (see Configuration settings).
2. Switch Self-monitoring enabled on, set the Check interval (seconds, minimum 60, default 300) and, if you want, the Environment and Service the generated alerts carry (defaults ToBeAlert and tobealert).
3. Save.

The same keys can be written to tobealertd.conf (SELF_MONITORING_ENABLED, SELF_MONITORING_INTERVAL, SELF_MONITORING_ENVIRONMENT, SELF_MONITORING_SERVICE — see Settings).

Restart the Celery workers on a running installation

Saving applies the new values to the API process immediately, but the Celery workers read the configuration when they start. On an installation that is already running, restart the workers after turning self-monitoring on or changing the interval — with Docker Compose, restart the backend container (docker compose restart tobealert-backend), which hosts the API and the workers. A fresh deployment that starts with the keys already in the file needs nothing else.

A Celery task wakes up every minute and checks, through Redis, whether the configured interval has elapsed since the last run; when it has, it runs the light probes. These scheduled runs are not added to the diagnostics history on this page; their outcome is the alerts they raise.

The alerts it generates

Every probe becomes one event through the normal ingest pipeline, so plugins, policies and notification rules see it like any other alert:

Field	Value
resource	The component: database, postgresql, redis, celery, host, notifications or backend
event	The probe id, for example pg.cache_hit or host.disk
environment / service	The configured Environment and Service (ToBeAlert / tobealert by default)
severity	warning for a slow verdict, critical for critical, major when the probe itself failed (error), normal when it is back within its threshold
origin	self-monitoring
group	self-monitoring
type	selfMonitoring
tags	self-monitoring
text	The probe's message (for example Disk space: /tobealert: 4% free) or … took N ms (threshold M ms); on recovery … is back within its threshold (N ms)
value	The probe duration, for example 12.4 ms
attributes	probe, verdict, thresholdOkMs, thresholdSlowMs, component

Because resource + event + environment identify the alert, repeated findings de-duplicate into the same alert and bump its count instead of creating new rows. When a probe goes back to ok, the same event is sent once with severity normal — the alarm model closes the alert — and nothing more is sent while the probe stays healthy. A probe that is healthy from the start never generates an alert.

Notifying on self-monitoring alerts

Create a notification rule that matches the origin and the severity you care about, for example:

Setting	Value
Conditions	origin Equals → self-monitoring · AND severity Is in → critical, major, warning · All Day
Trigger	Post receive (every finding) or Status change (opening and closing only)
Method	Your email or chat method

The recovery event (severity normal) also passes through rules, so a rule with Status change sends one message when the alert opens and one when it closes. To keep self-monitoring out of the operators' dashboard, add a view or policy that filters origin = self-monitoring into its own tab.

External health endpoint

GET /api/health is an unauthenticated endpoint for external monitors (Uptime Kuma, Nagios, a load balancer, a Kubernetes probe). It answers 200 when the database, Redis and the Celery workers respond and 503 when any of them fails, so the HTTP status alone is enough for a monitor.

curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9000/api/health

{
  "status": "ok",
  "version": "1.4.0",
  "hotfix": null,
  "uptimeSeconds": 86112,
  "timestamp": "2026-10-12T09:15:30Z",
  "components": {
    "database": { "status": "ok", "latencyMs": 1.8, "message": null },
    "redis": { "status": "ok", "latencyMs": 0.6, "message": null },
    "celery_workers": { "status": "ok", "latencyMs": 0.9, "message": null }
  },
  "latencyMs": 3.7
}

Field	Meaning
status	ok, or degraded when any component failed (HTTP 503)
version / hotfix	Backend release and the active hotfix id (null without one)
uptimeSeconds	Seconds since the API process started
components.<name>	status ok or failed, the check's latencyMs, and message with the failure text (null when ok)
latencyMs	Time the whole check took

Deep check

GET /api/health?deep=1 additionally runs the light probes (the same list self-monitoring uses) and adds a diagnostics block. It returns 503 also when the probes' overall verdict is critical or error, so a monitor can alert on a full disk, a stale worker heartbeat or a broken cache ratio, not only on a dead component.

The deep mode is protected by a token: send it in the X-Health-Token header (or as ?token=) and set the same value as Deep health check token in Backend Config → Self-monitoring and health, or as HEALTHCHECK_TOKEN in tobealertd.conf. With no token configured, or a token that does not match, the endpoint answers 403 and runs nothing.

curl -s -H 'X-Health-Token: <your token>' 'http://localhost:9000/api/health?deep=1'

{
  "status": "degraded",
  "version": "1.4.0",
  "components": { "database": { "status": "ok", "latencyMs": 1.9, "message": null }, "redis": { "status": "ok", "latencyMs": 0.5, "message": null }, "celery_workers": { "status": "ok", "latencyMs": 0.8, "message": null } },
  "diagnostics": {
    "verdict": "critical",
    "bottleneck": "Likely bottleneck: host resources (disk, clock, process). /tobealert: 4% free.",
    "probes": [
      { "id": "db.connect", "group": "database", "verdict": "ok", "durationMs": 1.6, "message": null },
      { "id": "host.disk", "group": "host", "verdict": "critical", "durationMs": 3.2, "message": "/tobealert: 4% free" }
    ]
  },
  "latencyMs": 412.5
}

diagnostics.probes[] carries id, group, verdict, durationMs and the probe's message (or its error text). A slow verdict keeps the HTTP status at 200 — read diagnostics.verdict if you want to warn on it.

Suggested monitor setup

Check	URL	Frequency	Alert when
Shallow	GET /api/health	every 30 s	HTTP ≠ 200
Deep	GET /api/health?deep=1 with X-Health-Token	every 5 min	HTTP ≠ 200, or diagnostics.verdict is slow for a warning

Notes:

• The response exposes no secrets: no database host, no credentials, no alert content. The shallow mode needs no token and is safe to expose to a monitoring network; the deep mode runs real queries against the database, so keep its frequency in minutes, not seconds.
• /api/management/healthcheck still exists and feeds the public Status page; /api/health is the one to use for probes because of its 200 / 503 semantics and its JSON shape. Docker Compose and Kubernetes examples are in Docker and Kubernetes.
• Behind the frontend's nginx, the same endpoint is reachable as http://<frontend>/api/health (see HTTPS and TLS).

Permissions

Running, listing, loading and deleting diagnostics requires the admin:management scope; the global admin permission covers it. read:management is not enough and there is no read-only access: a user who cannot run diagnostics cannot see the history either. See the permissions reference.

API

All endpoints need admin:management and return status: "ok" on success.

Method	Endpoint	Purpose
GET	/api/diagnostics	runs[] — the stored runs (id, createTime, user, durationMs, verdict, summary), newest first (?limit=, default 50) — plus probes[], the catalogue of probe ids, groups and labels
POST	/api/diagnostics/run	Run the probes now and store the result. Optional body {"probes": ["db.connect", "db.alerts_list"]} runs only those ids (unknown ids are ignored; an empty list runs everything). Returns report
GET	/api/diagnostics/<id>	report — the full stored report (what Download JSON saves). 404 if the id does not exist
DELETE	/api/diagnostics/<id>	Delete a stored run. 404 if the id does not exist

Report shape:

{
  "id": "4b4c6d0e-…",
  "createTime": "2026-10-09T10:15:30Z",
  "user": "tbadmin",
  "durationMs": 1834.2,
  "stored": true,
  "environment": { "release": "1.4.0", "hotfix": null, "timezone": "Europe/Madrid", "databaseHost": "postgres://***@tobealert-bbdd:5432/tobealert" },
  "verdict": "slow",
  "summary": {
    "verdict": "slow",
    "groups": { "database": "slow", "postgresql": "ok", "infrastructure": "info", "host": "ok" },
    "bottleneck": "Likely bottleneck: database query time. alerts: 23% dead tuples.",
    "counts": { "ok": 18, "info": 3, "slow": 1, "critical": 0, "error": 0 }
  },
  "probes": [
    {
      "id": "db.alerts_list",
      "group": "database",
      "label": "Dashboard alert list (50 open alerts)",
      "durationMs": 41.7,
      "threshold": { "ok": 300, "slow": 1500 },
      "verdict": "ok",
      "details": { "rows": 50, "sql": "SELECT …", "explain": { "plan": "…", "seqScans": [], "planningMs": 0.4, "executionMs": 18.2, "truncated": false } }
    }
  ]
}

message (the rule's note) and error (the failure text) appear on a probe only when present. The run is written to the audit trail as diagnostics-run with the report id as resource and the bottleneck line as message.

Related modules

Module	Relationship
Settings & environment variables	DATABASE_URL, REDIS_HOST and the PostgreSQL container variables the probes depend on
Backend Config	Alert retention and timeouts that change how much data the probed tables hold; the Self-monitoring and health category that enables self-monitoring and sets the deep health check token
Notification rules	Where to notify on self-monitoring alerts (origin = self-monitoring)
Hotfixes	The active hotfix id is part of the report's environment block
Resource graph API	The relations the db.resource_graph probe walks