Sending alerts to ToBeAlert

Overview

ToBeAlert provides a simple REST API for sending alerts from any monitoring system, application, or custom script. This guide covers everything you need to know about sending alerts, including required fields, optional attributes, and advanced features like links and tagged values.

Authentication

All API requests must include an API key in the Authorization header:

Authorization: Key YOUR_API_KEY_HERE

For detailed information on how to create, manage, and use API keys, see the API Keys documentation.

Endpoint

POST /api/alert
Content-Type: application/json
Authorization: Key YOUR_API_KEY

Required fields

These fields are mandatory for all alerts based on the Alert model validation:

Field	Type	Description	Example
resource	String	The resource affected by the alert	"web-01.example.com"
event	String	Type of event that triggered the alert	"ServerDown"

Note: The id field is automatically generated if not provided. Do not include it unless you need to specify a particular ID.

Optional fields

Field	Type	Default	Description	Example
environment	String	"" (empty string)	Environment where the alert originated	"production"
severity	String	"normal"	Alert severity level	"critical"
text	String	"" (empty string)	Human-readable description of the alert	"Server is not responding"
service	Array	[] (empty array)	List of related services	["web-service", "api"]
group	String	"Misc"	Organizational group	"infrastructure"
value	String	null	Metric value	"503"
tags	Array	[] (empty array)	Tags for categorization	["linux", "production"]
attributes	Object	{} (empty object)	Custom key-value pairs	See below
origin	String	Auto-generated	Server that sent the alert	"nagios"
source_id	String	null	Application that sent the alert	"app-monitor"
type	String	"Event"	Alert type	"exceptionAlert"
correlate	Array	[] (empty array)	Events to correlate with	["cpu-high", "memory-low"]
timeout	Integer	From config	Alert timeout in seconds	300
raw_data	String	null	Original raw data	"Raw alert data"
minutesDelayed	Integer	null	Minutes to hold the alert as delayed before open. An integer (including 0) overrides the delay-plugin setting. Omit or null to use Backend Config.	15
delayedUntil	String	null	ISO datetime when a delayed alert should become open (usually set by the plugin, not the client)	"2025-04-15T14:00:00Z"
resourceParent	String	null	Parent resource of resource, in the same environment. Stored on the alert and added to the resource graph	"web-cluster-01"
resourceChildren	Array	[] (empty array)	Resources that hang from resource, in the same environment. Not stored on the alert: it only feeds the resource graph	["vm-7", "vm-8"]
causeId	String	null	ID of an existing alert that explains this one. The new alert becomes its effect and is suppressed. See Cause and effect	"a1b2c3"
suppressed	Boolean	false (true when causeId is accepted)	Dim and group the alert on the dashboard. Can be sent without causeId	true

Database schema notes

From the database schema and model:

• id is auto-generated as UUID if not provided
• resource and event are the only truly required fields (validated in Alert.init)
• All other fields have sensible defaults as shown above
• severity defaults to "normal" (Severity.Normal)
• origin defaults to script name/hostname if not provided
• attributes is stored as JSONB for efficient querying
• minutesDelayed is an ingest-only field (not stored). After create, use status and delayedUntil
• resourceChildren is an ingest-only field (not stored). The relations it declares live in the resource graph
• Fields in invalidCreateColumns are automatically removed from requests

Cause and effect

An alert can declare another alert as its cause so that the dashboard groups it, dimmed, under that alert. See Cause and effect alerts for the concept and the dashboard behavior, and Modifiers and tools — Mark as effect when you prefer policies to build the links.

You send	Result
causeId of an existing alert that is not itself an effect	The alert is created as an effect of that alert, with suppressed: true
the same plus "suppressed": false	The alert is linked (causeId kept) but not suppressed — it is listed in its normal position
causeId that does not exist, or that points to an alert which is already an effect	The link is dropped: the alert is created without a cause and its history says so. The request still succeeds
"suppressed": true without causeId	The alert is dimmed on the dashboard but has no cause

Rules:

• Links have one level: an effect is never a cause, a cause is never an effect.
• The alert's status is not changed by being an effect.
• A repeat of an existing alert (same resource, event, and environment) never changes causeId or suppressed.
• An effect that already has a cause is never reassigned to another cause. To change it, release it first (close or delete the cause, or run a policy that sets suppressed to false and deletes causeId) and send the alert again.
• When the cause is deleted, or its status becomes closed / expired, or its severity goes back to normal / ok / cleared, every effect gets causeId: null and suppressed: false (with a history entry by system). Effects are not re-attached automatically.

Alert objects returned by the API carry the read-only fields effectCount (number of effects pointing to the alert), isEffect, and isCause next to causeId and suppressed — see Querying alerts.

Example: a database outage and its effects

Send these four requests in order. Replace DB_ALERT_ID with the id returned by the first one.

1. The cause:

Cause alert (will be linked by a policy or by causeId)

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "db-prod-01",
         "event": "DatabaseDown",
         "environment": "Production",
         "severity": "critical",
         "service": [
             "shop"
         ]
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "db-prod-01",
        "event": "DatabaseDown",
        "environment": "Production",
        "severity": "critical",
        "service": [
            "shop"
        ]
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

2. An effect linked by a policy — with a Mark as effect policy whose conditions are event = AppUnavailable and whose cause conditions are event = DatabaseDown (window 30 minutes, direction Before or after), this alert is attached to db-prod-01 automatically:

Effect linked by the policy

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "jboss-01",
         "event": "AppUnavailable",
         "environment": "Production",
         "severity": "major",
         "service": [
             "shop"
         ]
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "jboss-01",
        "event": "AppUnavailable",
        "environment": "Production",
        "severity": "major",
        "service": [
            "shop"
        ]
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

3. An effect linked by the sender (no policy needed, no policy history entry):

Effect linked by causeId in the request

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "jboss-02",
         "event": "AppUnavailable",
         "environment": "Production",
         "severity": "major",
         "service": [
             "shop"
         ],
         "causeId": "DB_ALERT_ID"
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "jboss-02",
        "event": "AppUnavailable",
        "environment": "Production",
        "severity": "major",
        "service": [
            "shop"
        ],
        "causeId": "DB_ALERT_ID"
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

4. A suppressed alert without a cause:

Suppressed alert with no cause

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "batch-07",
         "event": "JobSlow",
         "environment": "Production",
         "severity": "warning",
         "service": [
             "shop"
         ],
         "suppressed": true
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "batch-07",
        "event": "JobSlow",
        "environment": "Production",
        "severity": "warning",
        "service": [
            "shop"
        ],
        "suppressed": true
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

Resource relations

ToBeAlert keeps a resource graph: which resource is the parent of which, per environment. A resource is identified by the alert's environment and resource (an empty environment is allowed, and "" and "Production" are different environments). The graph is shared by alert events, policies, service topologies, and CMDB imports; each relation remembers who declared it (source). An alert can declare its own place in the graph with two fields:

Field	What it declares	Stored on the alert
resourceParent	The parent of resource (one string)	Yes — returned with the alert, kept in alert history, and filterable with resourceParent (see Querying alerts)
resourceChildren	Resources that hang from resource (array of strings) — for example a hypervisor alert listing its virtual machines	No — it only feeds the graph

Rules:

• Both fields create relations with source event at the moment the alert is stored. A rejected alert (validation error, blacklist, pre-receive plugin) creates nothing.
• A repeat of an existing alert (same resource, event, and environment) that omits resourceParent keeps the parent already stored. A repeat with a different resourceParent updates it.
• In resourceChildren, duplicates and the alert's own resource are ignored. Values are trimmed; empty strings are dropped.
• A resource can have several parents (for example a database shared by two applications): a new parent is added to the graph, it does not replace the others. To remove a relation, use the resource graph API.
• When the same relation is also declared by another source, the highest-ranked one wins: manual > event > cmdb > topology > policy. A relation added through the API (manual) is never overwritten by an alert payload.

Example: a VMware host and its virtual machines

Host alert that declares its parent cluster and its virtual machines

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "esx-01",
         "event": "HostNotResponding",
         "environment": "Production",
         "severity": "critical",
         "service": [
             "virtualization"
         ],
         "resourceParent": "cluster-a",
         "resourceChildren": [
             "vm-7",
             "vm-8",
             "esx-01"
         ]
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "esx-01",
        "event": "HostNotResponding",
        "environment": "Production",
        "severity": "critical",
        "service": [
            "virtualization"
        ],
        "resourceParent": "cluster-a",
        "resourceChildren": [
            "vm-7",
            "vm-8",
            "esx-01"
        ]
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

Once the alert is stored, the graph for environment Production contains three relations, all with source event:

Resource	Parent
esx-01	cluster-a
vm-7	esx-01
vm-8	esx-01

esx-01 was listed in its own resourceChildren and was ignored. The alert itself stores resourceParent: "cluster-a"; vm-7 and vm-8 do not need to have alerts of their own. A later alert for vm-7 will report cluster-a among its ancestors — see Querying alerts — Resource relation fields and the Resource graph API.

Delay plugin

The delay plugin can store a new Open alert as delayed and set delayedUntil (createTime + N minutes). Celery later moves it to open. Duplicates, correlations, and non-Open statuses skip the plugin.

Precedence:

1. If minutesDelayed is an integer (including 0), that value is used and NOTIFICATION_DELAY is ignored.
2. If the field is omitted or null (not an integer), the plugin uses Delay plugin delay from Backend Config (NOTIFICATION_DELAY).
3. 0 or a non-positive value means do not delay.

This does not hold outbound notifications. See Backend Config for the UI setting and Simulate alerts if you are testing from the dashboard.

Severity levels

Level	Text	Description
0	security	Security-related alerts
1	critical	Critical issues requiring immediate attention
2	major	Major problems affecting functionality
3	minor	Minor issues with limited impact
4	warning	Warnings about potential problems
5	indeterminate	Issues with undetermined severity
6	informational	Informational alerts, no action required
7	normal	Normal operation (ok/cleared)
8	debug	Debug-level information
9	trace	Detailed trace information
10	unknown	Unknown severity level

Alert attributes

The attributes field is a powerful way to add custom context to your alerts. It accepts any key-value pairs and supports special data types.

Basic attributes

"attributes": {
  "cpu_usage": "95.2%",
  "memory_free": "256MB",
  "error_code": "HTTP_503",
  "region": "us-west-2"
}

Links in attributes

ToBeAlert automatically detects and makes URLs clickable in alert attributes. The frontend supports three different link formats:

Format 1: Simple string URLs

"attributes": {
  "dashboard": "https://grafana.example.com/d/abc123",
  "runbook": "https://wiki.example.com/runbooks/server-down"
}

Format 2: Simple object with title and URL

"attributes": {
  "links": {
    "title": "Grafana Dashboard",
    "url": "https://grafana.example.com/d/abc123"
  }
}

Format 3: Array of objects with title and URL

"attributes": {
  "links": [
    {
      "title": "Grafana Dashboard",
      "url": "https://grafana.example.com/d/abc123"
    },
    {
      "title": "Runbook",
      "url": "https://wiki.example.com/runbooks/server-down"
    },
    {
      "title": "Logs",
      "url": "https://logs.example.com/search?q=server-down"
    }
  ]
}

Supported URL formats:

• HTTP/HTTPS URLs starting with http:// or https://
• URLs starting with www. are automatically converted to https://
• Links open in a new tab when clicked

Recomendations for links:

• Use descriptive titles in object format for better UX
• Include relevant monitoring dashboards
• Add runbook links for troubleshooting procedures
• Provide direct links to logs or traces

Array attributes

For multiple values:

"attributes": {
  "affected_servers": ["web-01", "web-02", "web-03"],
  "error_codes": ["HTTP_503", "TIMEOUT", "CONN_REFUSED"]
}

Tagged values

Add tags to values for enhanced filtering and visualization:

"attributes": {
  "status": "degraded#danger",
  "priority": "high#warning",
  "component": "database#danger"
}

Format: value#tag - the tag appears as a colored badge in the UI.

Array with tagged values

"attributes": {
  "components": ["database#danger", "cache#warning", "web#ok"],
  "services": ["auth#production", "payment#production", "analytics#staging"]
}

Complete examples

Basic server alert

Basic server alert example

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "web-01.example.com",
         "event": "ServerDown",
         "environment": "production",
         "severity": "critical",
         "text": "Server web-01 is not responding",
         "service": [
             "web-service"
         ],
         "group": "infrastructure",
         "origin": "nagios",
         "attributes": {
             "ip_address": "10.0.1.100",
             "last_check": "2025-04-15T10:30:00Z",
             "dashboard_url": "https://grafana.example.com/d/server"
         }
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "web-01.example.com",
        "event": "ServerDown",
        "environment": "production",
        "severity": "critical",
        "text": "Server web-01 is not responding",
        "service": [
            "web-service"
        ],
        "group": "infrastructure",
        "origin": "nagios",
        "attributes": {
            "ip_address": "10.0.1.100",
            "last_check": "2025-04-15T10:30:00Z",
            "dashboard_url": "https://grafana.example.com/d/server"
        }
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

Application error with links

Application error with links to logs and runbook

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "payment-service-02",
         "event": "ApplicationException",
         "environment": "production",
         "severity": "major",
         "text": "Database connection pool exhausted in payment service",
         "service": [
             "payment-service",
             "api-gateway"
         ],
         "group": "backend",
         "origin": "application",
         "attributes": {
             "error_count": "150",
             "pool_size": "20",
             "active_connections": "20",
             "logs_url": "https://logs.example.com/search?q=payment-service",
             "runbook": "https://wiki.example.com/runbooks/db-pool-exhausted",
             "component": "database#danger",
             "affected_endpoints": [
                 "/api/v1/pay",
                 "/api/v1/validate"
             ]
         }
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "payment-service-02",
        "event": "ApplicationException",
        "environment": "production",
        "severity": "major",
        "text": "Database connection pool exhausted in payment service",
        "service": [
            "payment-service",
            "api-gateway"
        ],
        "group": "backend",
        "origin": "application",
        "attributes": {
            "error_count": "150",
            "pool_size": "20",
            "active_connections": "20",
            "logs_url": "https://logs.example.com/search?q=payment-service",
            "runbook": "https://wiki.example.com/runbooks/db-pool-exhausted",
            "component": "database#danger",
            "affected_endpoints": [
                "/api/v1/pay",
                "/api/v1/validate"
            ]
        }
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

Cloud infrastructure alert

Cloud infrastructure alert with AWS monitoring links

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "i-1234567890abcdef0",
         "event": "HighCPUUtilization",
         "environment": "production",
         "severity": "warning",
         "text": "EC2 instance CPU usage above threshold",
         "service": [
             "web-tier"
         ],
         "group": "cloud-infrastructure",
         "origin": "cloudwatch",
         "tags": [
             "aws",
             "ec2",
             "cpu"
         ],
         "value": "87.5%",
         "attributes": {
             "instance_type": "t3.medium",
             "region": "us-west-2",
             "availability_zone": "usw2a",
             "cpu_threshold": "80%",
             "cloudwatch_url": "https://console.aws.amazon.com/cloudwatch/home?region=us-west-2",
             "instance_url": "https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Instances:instanceId=i-1234567890abcdef0",
             "metrics": [
                 "cpu#high",
                 "memory#normal",
                 "disk#normal"
             ]
         }
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "i-1234567890abcdef0",
        "event": "HighCPUUtilization",
        "environment": "production",
        "severity": "warning",
        "text": "EC2 instance CPU usage above threshold",
        "service": [
            "web-tier"
        ],
        "group": "cloud-infrastructure",
        "origin": "cloudwatch",
        "tags": [
            "aws",
            "ec2",
            "cpu"
        ],
        "value": "87.5%",
        "attributes": {
            "instance_type": "t3.medium",
            "region": "us-west-2",
            "availability_zone": "usw2a",
            "cpu_threshold": "80%",
            "cloudwatch_url": "https://console.aws.amazon.com/cloudwatch/home?region=us-west-2",
            "instance_url": "https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Instances:instanceId=i-1234567890abcdef0",
            "metrics": [
                "cpu#high",
                "memory#normal",
                "disk#normal"
            ]
        }
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

Response format

Success response

{
  "status": "ok",
  "id": "generated-alert-id",
  "alert": {
    "id": "generated-alert-id",
    "href": "http://your-tobealert.com:9000/alert/generated-alert-id"
  }
}

Error response

{
  "status": "error",
  "message": "Invalid severity level: 'invalid'",
  "errors": [
    {
      "field": "severity",
      "message": "Must be one of: critical, major, minor, warning, etc."
    }
  ]
}

Best practices

1. Use descriptive alert IDs - Include service/resource information for easier tracking
2. Provide context in attributes - Include URLs to dashboards, runbooks, and related systems
3. Use consistent naming - Standardize environment, service, and group names across alerts
4. Include relevant links - Add dashboard URLs, runbook links, and troubleshooting resources
5. Use tagged values - Leverage tagging for visual indicators in the UI
6. Set appropriate severity - Use severity levels consistently to avoid alert fatigue
7. Add meaningful tags - Use tags for easy filtering and categorization

Troubleshooting

Common issues

1. 401 Unauthorized - Check your API key is valid and properly formatted
2. 400 Bad Request - Verify required fields are present and correctly formatted
3. 429 Too Many Requests - You've hit the rate limit, implement backoff
4. Links not clickable - Ensure URLs start with http:// or https://

Validation rules

• Only resource and event are validated as mandatory
• timeout: Must be a non-negative integer
• attribute keys: Cannot contain . or $ characters (validated in Alert.init)