Skip to main content
Version: 1.4 (Current)

Checkmk notification script

Checkmk sends its host and service notifications to ToBeAlert through a custom notification script installed in the Checkmk site. Each notification becomes an alert with the host as resource, the service as event, the Checkmk state mapped to a severity and the attributes checkmk_host, checkmk_service, checkmk_site and checkmk_type. With a Checkmk integration, those attributes let ToBeAlert show the Checkmk graphs of the service in the alert details.

The script and its parameter form ship with the ToBeAlert backend:

FilePurposeInstall in the Checkmk site as
tobealert.pyThe notification script (Python 3, standard library only)~/local/share/check_mk/notifications/tobealert.py
tobealert_params.pyThe ToBeAlert parameters form shown in the notification rule (Setup)~/local/share/check_mk/web/plugins/wato/tobealert_params.py

Both files are in the sql/ folder of the backend package (tobealert/sql/ in the source tree and in the backend image).

Install​

  1. Create an API key in ToBeAlert with write:alerts. Alerts from Checkmk are created with that key.

  2. Create the alert source the script will use. Open Alerts → Alert Sources, click Create alert source and name it checkmk (one source per Checkmk site, for example checkmk-prod, if you have several). The name may contain only letters, numbers, - and _. You can also let the integration drawer create it, see Integrations — Alert source created on save.

  3. Copy the two files into the Checkmk site as the site user:

    # as the site user (omd su mysite)
    install -m 755 tobealert.py ~/local/share/check_mk/notifications/tobealert.py
    install -m 644 tobealert_params.py ~/local/share/check_mk/web/plugins/wato/tobealert_params.py
    omd restart apache
  4. In Checkmk open Setup → Events → Notifications, add a rule and choose the notification method ToBeAlert custom notification. Fill in the ToBeAlert parameters (below), save and activate the changes.

  5. Send a test notification from Checkmk (for example a fake check result on a service) and check that the alert appears on the ToBeAlert dashboard with the icon of the source.

Parameters​

ParameterDefaultDescription
ToBeAlert endpoint URL—Required. Base address of the ToBeAlert API, for example https://your-tobealert.com:9000 (no path)
API path/api/alertPath appended to the endpoint
Disable SSL certificate verificationoffAccept a self-signed ToBeAlert certificate
HTTP proxy—Proxy for the request, if Checkmk needs one
AuthenticationBearer tokenBearer token with the ToBeAlert API key, Basic auth, or No authentication
SourcecheckmkThe alert source of the alerts (sourceId). It must be the name of the alert source linked to the integration; otherwise the alert details show no graphs. Letters, numbers, - and _ only
Source icon / Auto create sourcecheckmk / offNot used by the current script: the alert source is created in ToBeAlert (step 2), not by Checkmk
Default environmentProductionenvironment of the alerts
Default servicecheckmkSingle entry of the service list
Default severitymajorSeverity when the Checkmk state is not in Severity mapping
Resource template$HOSTNAME$resource of the alert
Event template$SERVICEDESC$event of the alert (empty for host notifications; the script then uses the host name)
Text template$HOSTNAME$/$SERVICEDESC$: $SERVICEOUTPUT$text of the alert
Origin templatecheckmk/$OMD_SITE$origin of the alert
Static tags (comma separated)—Tags added to every alert
Labels mapping—target=source lines; the rendered values are stored under attributes.labels
Attributes mapping—target=source lines added to attributes (they can overwrite the checkmk_* attributes)
Extra HTTP headers—Header=Value lines sent with the request
Severity mappingCRIT=critical, WARN=warning, UNKNOWN=minor, OK=normalCheckmk short state (CRIT, WARN, UNKNOWN, OK; DOWN, UP, UNREACH for hosts) → severity. Add the host states if you want them different from Default severity
Status mappingPROBLEM=open, RECOVERY=closed, ACKNOWLEDGEMENT=ackCheckmk notification type → alert status
HTTP timeout10 secondsRequest timeout
Dry runoffOnly print the JSON payload in the notification log, without sending

Templates accept any Checkmk notification variable as $NAME$ ($HOSTNAME$, $SERVICEDESC$, $SERVICEOUTPUT$, $HOSTALIAS$, $OMD_SITE$, …).

What the alert contains​

Besides the fields above, every alert carries:

FieldValue
groupThe host groups of the host, or checkmk
valueThe service output (host output for host notifications)
typeexceptionAlert
attributes.checkmk_hostCheckmk host name
attributes.checkmk_serviceService description; empty for host notifications
attributes.checkmk_siteSite name
attributes.checkmk_typePROBLEM, RECOVERY, ACKNOWLEDGEMENT, …

The Checkmk integration reads checkmk_host, checkmk_service and checkmk_site by default. Keep them if you add an Attributes mapping.

Troubleshooting​

SymptomCauseWhat to do
Notification log: ERROR ToBeAlert HTTP 401Wrong or missing API keyCheck Authentication
ERROR ToBeAlert HTTP 403The key lacks write:alertsCreate a key with that scope
ERROR ToBeAlert request failed: …Network, TLS or proxy between Checkmk and ToBeAlertCheck ToBeAlert endpoint URL, HTTP proxy, and the certificate (Disable SSL certificate verification for self-signed)
Alert arrives without the source iconSource does not match an existing alert sourceCreate the alert source with exactly that name
Alert arrives but the details show no Integration sectionSource differs from the alert source linked to the integration, or the integration is disabledAlign the two names; check the Enabled switch in Management → Integrations