Sending alerts to ToBeAlert

Overview

ToBeAlert provides a simple REST API for sending alerts from any monitoring system, application, or custom script. This guide covers everything you need to know about sending alerts, including required fields, optional attributes, and advanced features like links and tagged values.

Authentication

All API requests must include an API key in the Authorization header:

Authorization: Key YOUR_API_KEY_HERE

For detailed information on how to create, manage, and use API keys, see the API Keys documentation.

Endpoint

POST /api/alert
Content-Type: application/json
Authorization: Key YOUR_API_KEY

Required fields

These fields are mandatory for all alerts based on the Alert model validation:

Field	Type	Description	Example
resource	String	The resource affected by the alert	"web-01.example.com"
event	String	Type of event that triggered the alert	"ServerDown"

Note: The id field is automatically generated if not provided. Do not include it unless you need to specify a particular ID.

Optional fields

Field	Type	Default	Description	Example
environment	String	"" (empty string)	Environment where the alert originated	"production"
severity	String	"normal"	Alert severity level	"critical"
text	String	"" (empty string)	Human-readable description of the alert	"Server is not responding"
service	Array	[] (empty array)	List of related services	["web-service", "api"]
group	String	"Misc"	Organizational group	"infrastructure"
value	String	null	Metric value	"503"
tags	Array	[] (empty array)	Tags for categorization	["linux", "production"]
attributes	Object	{} (empty object)	Custom key-value pairs	See below
origin	String	Auto-generated	Server that sent the alert	"nagios"
source_id	String	null	Application that sent the alert	"app-monitor"
type	String	"Event"	Alert type	"exceptionAlert"
correlate	Array	[] (empty array)	Events to correlate with	["cpu-high", "memory-low"]
timeout	Integer	From config	Alert timeout in seconds	300
raw_data	String	null	Original raw data	"Raw alert data"
minutes_delayed	Integer	null	Minutes to delay the alert	15
delayed_until	String	null	ISO datetime to delay until	"2025-04-15T14:00:00Z"
resource_parent	String	null	Parent resource	"web-cluster-01"

Database schema notes

From the database schema and model:

• id is auto-generated as UUID if not provided
• resource and event are the only truly required fields (validated in Alert.init)
• All other fields have sensible defaults as shown above
• severity defaults to "normal" (Severity.Normal)
• origin defaults to script name/hostname if not provided
• attributes is stored as JSONB for efficient querying
• minutes_delayed and delayed_until are used for alert scheduling
• Fields in invalidCreateColumns are automatically removed from requests

Severity levels

Level	Text	Description
0	security	Security-related alerts
1	critical	Critical issues requiring immediate attention
2	major	Major problems affecting functionality
3	minor	Minor issues with limited impact
4	warning	Warnings about potential problems
5	indeterminate	Issues with undetermined severity
6	informational	Informational alerts, no action required
7	normal	Normal operation (ok/cleared)
8	debug	Debug-level information
9	trace	Detailed trace information
10	unknown	Unknown severity level

Alert attributes

The attributes field is a powerful way to add custom context to your alerts. It accepts any key-value pairs and supports special data types.

Basic attributes

"attributes": {
  "cpu_usage": "95.2%",
  "memory_free": "256MB",
  "error_code": "HTTP_503",
  "region": "us-west-2"
}

Links in attributes

ToBeAlert automatically detects and makes URLs clickable in alert attributes. The frontend supports three different link formats:

Format 1: Simple string URLs

"attributes": {
  "dashboard": "https://grafana.example.com/d/abc123",
  "runbook": "https://wiki.example.com/runbooks/server-down"
}

Format 2: Simple object with title and URL

"attributes": {
  "links": {
    "title": "Grafana Dashboard",
    "url": "https://grafana.example.com/d/abc123"
  }
}

Format 3: Array of objects with title and URL

"attributes": {
  "links": [
    {
      "title": "Grafana Dashboard",
      "url": "https://grafana.example.com/d/abc123"
    },
    {
      "title": "Runbook",
      "url": "https://wiki.example.com/runbooks/server-down"
    },
    {
      "title": "Logs",
      "url": "https://logs.example.com/search?q=server-down"
    }
  ]
}

Supported URL formats:

• HTTP/HTTPS URLs starting with http:// or https://
• URLs starting with www. are automatically converted to https://
• Links open in a new tab when clicked

Recomendations for links:

• Use descriptive titles in object format for better UX
• Include relevant monitoring dashboards
• Add runbook links for troubleshooting procedures
• Provide direct links to logs or traces

Array attributes

For multiple values:

"attributes": {
  "affected_servers": ["web-01", "web-02", "web-03"],
  "error_codes": ["HTTP_503", "TIMEOUT", "CONN_REFUSED"]
}

Tagged values

Add tags to values for enhanced filtering and visualization:

"attributes": {
  "status": "degraded#danger",
  "priority": "high#warning",
  "component": "database#danger"
}

Format: value#tag - the tag appears as a colored badge in the UI.

Array with tagged values

"attributes": {
  "components": ["database#danger", "cache#warning", "web#ok"],
  "services": ["auth#production", "payment#production", "analytics#staging"]
}

Complete examples

Basic server alert

Basic server alert example

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "web-01.example.com",
         "event": "ServerDown",
         "environment": "production",
         "severity": "critical",
         "text": "Server web-01 is not responding",
         "service": [
             "web-service"
         ],
         "group": "infrastructure",
         "origin": "nagios",
         "attributes": {
             "ip_address": "10.0.1.100",
             "last_check": "2025-04-15T10:30:00Z",
             "dashboard_url": "https://grafana.example.com/d/server"
         }
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "web-01.example.com",
        "event": "ServerDown",
        "environment": "production",
        "severity": "critical",
        "text": "Server web-01 is not responding",
        "service": [
            "web-service"
        ],
        "group": "infrastructure",
        "origin": "nagios",
        "attributes": {
            "ip_address": "10.0.1.100",
            "last_check": "2025-04-15T10:30:00Z",
            "dashboard_url": "https://grafana.example.com/d/server"
        }
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

Application error with links

Application error with links to logs and runbook

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "payment-service-02",
         "event": "ApplicationException",
         "environment": "production",
         "severity": "major",
         "text": "Database connection pool exhausted in payment service",
         "service": [
             "payment-service",
             "api-gateway"
         ],
         "group": "backend",
         "origin": "application",
         "attributes": {
             "error_count": "150",
             "pool_size": "20",
             "active_connections": "20",
             "logs_url": "https://logs.example.com/search?q=payment-service",
             "runbook": "https://wiki.example.com/runbooks/db-pool-exhausted",
             "component": "database#danger",
             "affected_endpoints": [
                 "/api/v1/pay",
                 "/api/v1/validate"
             ]
         }
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "payment-service-02",
        "event": "ApplicationException",
        "environment": "production",
        "severity": "major",
        "text": "Database connection pool exhausted in payment service",
        "service": [
            "payment-service",
            "api-gateway"
        ],
        "group": "backend",
        "origin": "application",
        "attributes": {
            "error_count": "150",
            "pool_size": "20",
            "active_connections": "20",
            "logs_url": "https://logs.example.com/search?q=payment-service",
            "runbook": "https://wiki.example.com/runbooks/db-pool-exhausted",
            "component": "database#danger",
            "affected_endpoints": [
                "/api/v1/pay",
                "/api/v1/validate"
            ]
        }
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

Cloud infrastructure alert

Cloud infrastructure alert with AWS monitoring links

Bash

curl -X POST "http://your-tobealert.com:9000/api/alert" \
  -H "Content-Type: application/json" \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '{
         "resource": "i-1234567890abcdef0",
         "event": "HighCPUUtilization",
         "environment": "production",
         "severity": "warning",
         "text": "EC2 instance CPU usage above threshold",
         "service": [
             "web-tier"
         ],
         "group": "cloud-infrastructure",
         "origin": "cloudwatch",
         "tags": [
             "aws",
             "ec2",
             "cpu"
         ],
         "value": "87.5%",
         "attributes": {
             "instance_type": "t3.medium",
             "region": "us-west-2",
             "availability_zone": "usw2a",
             "cpu_threshold": "80%",
             "cloudwatch_url": "https://console.aws.amazon.com/cloudwatch/home?region=us-west-2",
             "instance_url": "https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Instances:instanceId=i-1234567890abcdef0",
             "metrics": [
                 "cpu#high",
                 "memory#normal",
                 "disk#normal"
             ]
         }
     }'

Python

import requests

response = requests.post(
    "http://your-tobealert.com:9000/api/alert",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Key YOUR_API_KEY"
    },
    json={
        "resource": "i-1234567890abcdef0",
        "event": "HighCPUUtilization",
        "environment": "production",
        "severity": "warning",
        "text": "EC2 instance CPU usage above threshold",
        "service": [
            "web-tier"
        ],
        "group": "cloud-infrastructure",
        "origin": "cloudwatch",
        "tags": [
            "aws",
            "ec2",
            "cpu"
        ],
        "value": "87.5%",
        "attributes": {
            "instance_type": "t3.medium",
            "region": "us-west-2",
            "availability_zone": "usw2a",
            "cpu_threshold": "80%",
            "cloudwatch_url": "https://console.aws.amazon.com/cloudwatch/home?region=us-west-2",
            "instance_url": "https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#Instances:instanceId=i-1234567890abcdef0",
            "metrics": [
                "cpu#high",
                "memory#normal",
                "disk#normal"
            ]
        }
    }
)

if response.status_code in [200, 201]:
    print("Request successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

Response format

Success response

{
  "status": "ok",
  "id": "generated-alert-id",
  "alert": {
    "id": "generated-alert-id",
    "href": "http://your-tobealert.com:9000/alert/generated-alert-id"
  }
}

Error response

{
  "status": "error",
  "message": "Invalid severity level: 'invalid'",
  "errors": [
    {
      "field": "severity",
      "message": "Must be one of: critical, major, minor, warning, etc."
    }
  ]
}

Best practices

1. Use descriptive alert IDs - Include service/resource information for easier tracking
2. Provide context in attributes - Include URLs to dashboards, runbooks, and related systems
3. Use consistent naming - Standardize environment, service, and group names across alerts
4. Include relevant links - Add dashboard URLs, runbook links, and troubleshooting resources
5. Use tagged values - Leverage tagging for visual indicators in the UI
6. Set appropriate severity - Use severity levels consistently to avoid alert fatigue
7. Add meaningful tags - Use tags for easy filtering and categorization

Troubleshooting

Common issues

1. 401 Unauthorized - Check your API key is valid and properly formatted
2. 400 Bad Request - Verify required fields are present and correctly formatted
3. 429 Too Many Requests - You've hit the rate limit, implement backoff
4. Links not clickable - Ensure URLs start with http:// or https://

Validation rules

• Only resource and event are validated as mandatory
• timeout: Must be a non-negative integer
• attribute keys: Cannot contain . or $ characters (validated in Alert.init)