Checkmk notification script
Checkmk sends its host and service notifications to ToBeAlert through a custom notification script installed in the Checkmk site. Each notification becomes an alert with the host as resource, the service as event, the Checkmk state mapped to a severity and the attributes checkmk_host, checkmk_service, checkmk_site and checkmk_type. With a Checkmk integration, those attributes let ToBeAlert show the Checkmk graphs of the service in the alert details.
The script and its parameter form ship with the ToBeAlert backend:
| File | Purpose | Install in the Checkmk site as |
|---|---|---|
tobealert.py | The notification script (Python 3, standard library only) | ~/local/share/check_mk/notifications/tobealert.py |
tobealert_params.py | The ToBeAlert parameters form shown in the notification rule (Setup) | ~/local/share/check_mk/web/plugins/wato/tobealert_params.py |
Both files are in the sql/ folder of the backend package (tobealert/sql/ in the source tree and in the backend image).
Install
-
Create an API key in ToBeAlert with
write:alerts. Alerts from Checkmk are created with that key. -
Create the alert source the script will use. Open Alerts → Alert Sources, click Create alert source and name it
checkmk(one source per Checkmk site, for examplecheckmk-prod, if you have several). The name may contain only letters, numbers,-and_. You can also let the integration drawer create it, see Integrations — Alert source created on save. -
Copy the two files into the Checkmk site as the site user:
# as the site user (omd su mysite)
install -m 755 tobealert.py ~/local/share/check_mk/notifications/tobealert.py
install -m 644 tobealert_params.py ~/local/share/check_mk/web/plugins/wato/tobealert_params.py
omd restart apache -
In Checkmk open Setup → Events → Notifications, add a rule and choose the notification method ToBeAlert custom notification. Fill in the ToBeAlert parameters (below), save and activate the changes.
-
Send a test notification from Checkmk (for example a fake check result on a service) and check that the alert appears on the ToBeAlert dashboard with the icon of the source.
Parameters
| Parameter | Default | Description |
|---|---|---|
| ToBeAlert endpoint URL | — | Required. Base address of the ToBeAlert API, for example https://your-tobealert.com:9000 (no path) |
| API path | /api/alert | Path appended to the endpoint |
| Disable SSL certificate verification | off | Accept a self-signed ToBeAlert certificate |
| HTTP proxy | — | Proxy for the request, if Checkmk needs one |
| Authentication | Bearer token | Bearer token with the ToBeAlert API key, Basic auth, or No authentication |
| Source | checkmk | The alert source of the alerts (sourceId). It must be the name of the alert source linked to the integration; otherwise the alert details show no graphs. Letters, numbers, - and _ only |
| Source icon / Auto create source | checkmk / off | Not used by the current script: the alert source is created in ToBeAlert (step 2), not by Checkmk |
| Default environment | Production | environment of the alerts |
| Default service | checkmk | Single entry of the service list |
| Default severity | major | Severity when the Checkmk state is not in Severity mapping |
| Resource template | $HOSTNAME$ | resource of the alert |
| Event template | $SERVICEDESC$ | event of the alert (empty for host notifications; the script then uses the host name) |
| Text template | $HOSTNAME$/$SERVICEDESC$: $SERVICEOUTPUT$ | text of the alert |
| Origin template | checkmk/$OMD_SITE$ | origin of the alert |
| Static tags (comma separated) | — | Tags added to every alert |
| Labels mapping | — | target=source lines; the rendered values are stored under attributes.labels |
| Attributes mapping | — | target=source lines added to attributes (they can overwrite the checkmk_* attributes) |
| Extra HTTP headers | — | Header=Value lines sent with the request |
| Severity mapping | CRIT=critical, WARN=warning, UNKNOWN=minor, OK=normal | Checkmk short state (CRIT, WARN, UNKNOWN, OK; DOWN, UP, UNREACH for hosts) → severity. Add the host states if you want them different from Default severity |
| Status mapping | PROBLEM=open, RECOVERY=closed, ACKNOWLEDGEMENT=ack | Checkmk notification type → alert status |
| HTTP timeout | 10 seconds | Request timeout |
| Dry run | off | Only print the JSON payload in the notification log, without sending |
Templates accept any Checkmk notification variable as $NAME$ ($HOSTNAME$, $SERVICEDESC$, $SERVICEOUTPUT$, $HOSTALIAS$, $OMD_SITE$, …).
What the alert contains
Besides the fields above, every alert carries:
| Field | Value |
|---|---|
group | The host groups of the host, or checkmk |
value | The service output (host output for host notifications) |
type | exceptionAlert |
attributes.checkmk_host | Checkmk host name |
attributes.checkmk_service | Service description; empty for host notifications |
attributes.checkmk_site | Site name |
attributes.checkmk_type | PROBLEM, RECOVERY, ACKNOWLEDGEMENT, … |
The Checkmk integration reads checkmk_host, checkmk_service and checkmk_site by default. Keep them if you add an Attributes mapping.
Troubleshooting
| Symptom | Cause | What to do |
|---|---|---|
Notification log: ERROR ToBeAlert HTTP 401 | Wrong or missing API key | Check Authentication |
ERROR ToBeAlert HTTP 403 | The key lacks write:alerts | Create a key with that scope |
ERROR ToBeAlert request failed: … | Network, TLS or proxy between Checkmk and ToBeAlert | Check ToBeAlert endpoint URL, HTTP proxy, and the certificate (Disable SSL certificate verification for self-signed) |
| Alert arrives without the source icon | Source does not match an existing alert source | Create the alert source with exactly that name |
| Alert arrives but the details show no Integration section | Source differs from the alert source linked to the integration, or the integration is disabled | Align the two names; check the Enabled switch in Management → Integrations |
Related
- Checkmk integration — graphs in the alert details
- Alert sources
- Sending alerts — alert fields