Version: 1.4 (Current)

Custom Webhooks Troubleshooting

Common Issues

401 Unauthorized

Problem: Inbound auth failed for this webhook (authMode is header or api_key).

What happens:

No alerts are created
HTTP 401 with a short reason (never echoes the received secret or signature)
If the body was valid keepable JSON (non-empty object/array ≤ 256 KiB), it is stored in payloadHistory with the fixed message:
Authentication failed. This payload is kept for testing and field mapping, but it will not be converted into an alert.
errorHistory always records the failure (message = the short auth reason)

Typical messages: missing header, invalid header, invalid signature, invalid API key, API key does not match this webhook, API key lacks write:alerts.

Solution:

Confirm authMode on GET /api/webhook_config/{id}
For header / static or generic: send the configured header name and value / HMAC of the raw body
For github / slack: use the canonical headers and signing string documented in Inbound authentication
For api_key: send Authorization: Key <key> or X-API-Key (not ?api-key=). The key must match apiKeyId, not be expired, and include write:alerts
If the key was deleted or lost scope, the webhook stays in api_key mode — fix the key or change auth; it does not become open

# Example: static header
curl -X POST https://your-tobealert.com:9000/api/webhooks/custom/your-webhook-id \
  -H "Content-Type: application/json" \
  -H "cf-webhook-auth: your-shared-secret" \
  -d '{"hostname":"web-01","check":"cpu","status":"WARNING","message":"test"}'

400 Bad Request

Problem: Field mapping failed, invalid data, empty body, invalid JSON, or no alerts from the configured Array Path.

Common causes:

Invalid JSON in request
Empty request body (including null, {}, or [] treated as empty)
Array Path set, but the POST yields no alerts: the path is missing from the JSON, or the array at that path is empty
Every mapped alert was rejected: a required field ended up missing (An environment must be provided), service / tags is not a list, or severity is not a ToBeAlert severity (Severity (CRITICAL) is not one of …)

When Array Path is set and no alerts are generated, the HTTP response and errorHistory use messages such as:

No alerts generated from payload: array path "{path}" was not found. Top-level keys: … — the path is not present in the JSON
No alerts generated from payload: array path "{path}" is an empty list. — the array exists but has no items

This is not an auth failure: with authMode: none, auth has already passed. A disabled webhook would have returned 403 before mapping.

Example error responses:

{
  "status": "error",
  "message": "No alerts generated from payload: array path \"events\" is an empty list."
}

When the mapping ran but no alert could be created, each rejected alert is listed with the data the mapping produced:

{
  "status": "error",
  "webhook_id": "your-webhook-id",
  "webhook_name": "My Monitoring Integration",
  "alerts_created": 0,
  "alerts": [],
  "errors": [
    {
      "alert_data": { "resource": "web-01", "event": "cpu", "severity": "CRITICAL", "...": "..." },
      "error": "Severity (CRITICAL) is not one of security, critical, major, minor, warning, …"
    }
  ]
}

Invalid JSON and empty bodies add an errorHistory entry only (payload: null) with a message such as Invalid JSON payload: … or Empty payload. They do not appear in payloadHistory. Mapping failures after a valid body appear in both histories. Inspect via GET /api/webhook_config/{id} (scope read:webhook_configs).

Solution:

Read errors[].alert_data in the response (or the errorHistory message) to see what the mapping produced
Give every required field a default value, so a missing path does not drop it
Keep values that are not ToBeAlert severities in an attribute and translate them with a policy (see Transforming values)
Validate incoming JSON format (non-empty object or array)
Make Array Path match a key in the payload (for example events), or leave it empty to treat the whole body as one alert

413 Payload Too Large

Problem: The raw request body exceeds 256 KiB (262144 bytes).

What happens:

Auth is not checked (size is evaluated first)
The body is not stored in payloadHistory
No alerts are created
HTTP 413
errorHistory only: payload: null, oversized: true, and a message like Payload exceeded the size limit (received {n} bytes, limit 262144 bytes).

Solution:

Reduce the payload size (drop unused fields, truncate logs/stack traces)
Confirm the limit is measured on the raw body, not a pretty-printed copy
Check errorHistory on GET /api/webhook_config/{id} for the oversized entry

403 Forbidden (disabled)

Problem: The webhook configuration is disabled.

What happens (auth already succeeded):

Valid in-limit JSON is stored in payloadHistory with message: null (useful for mapping)
errorHistory gets 403 without a traceback
No alerts are created

Enable the config when you are ready to create alerts.

Missing Data After Mapping

Problem: Some fields are empty or missing in the created alert.

Causes:

The path does not exist in this payload (typo, different nesting, or the value is inside an array — use an index such as items[0].name or Array Path)
The mapping value is fixed text ("origin": "nagios") or a template ("${hostname}"): both are read as paths that do not exist, so the field is left out
With Array Path set, the path points outside the array element (for example commonLabels when arrayPath is alerts)

Solution:

Compare the paths with a stored payload (payloadHistory on GET /api/webhook_config/{id})
Move fixed text to defaultValues, and add defaults for fields that are not always present:

"defaultValues": {
  "environment": "Production",
  "group": "default",
  "origin": "nagios"
}

Templates or Expressions Have No Effect

Problem: A mapping such as "${hostname}-${check}", "${status == 'CRITICAL' ? 'critical' : 'warning'}" or "${message.trim()}" produces nothing.

Cause: Field mappings do not support templates, expressions, or functions. Each string is a path, and these strings are paths that do not exist.

Solution:

Map the plain value: "text": "message"
For a boolean source, use {"path": "is_critical", "if_true": "critical", "if_false": "warning"}
For case changes, trimming, value translation, or numeric thresholds, add a policy on receive — see Transforming values

Array Mapping Issues

Problem: Arrays not mapping correctly.

Solution:

service and tags need a list — map the array directly:

"fieldMappings": {
  "tags": "tags",
  "service": "services"
}

A text field needs a string — join the array:

"fieldMappings": {
  "text": { "path": "errors", "join": ", " }
}

One element by index, with a default when the array is empty:

"fieldMappings": { "resource": "servers[0]" },
"defaultValues": { "resource": "unknown" }

One alert per element: {"path": "servers[]", "expand": true}, or set Array Path when each element is a whole event.

Debugging Techniques

1. Test with Simple Data

Start with a simple test payload:

{
  "hostname": "test-server",
  "check": "cpu",
  "status": "WARNING",
  "message": "Test message"
}

2. Inspect recent payload and error history

curl -H "Authorization: Key YOUR_API_KEY" \
  http://your-tobealert.com:9000/api/webhook_config/your-webhook-id

payloadHistory (up to 5, newest first): valid JSON kept for mapping (including disabled and auth-failed keepable bodies). Does not include oversized, invalid JSON, or empty bodies.
errorHistory (up to 5): 400 / 401 / 403 / 413 / 500 / partial failures; oversized and parse errors live here with payload: null.
List endpoint: payloadHistoryCount, lastPayloadAt, errorHistoryCount, lastErrorAt only (no bodies).

Validation Tips

1. Validate Input Structure

Know what data your monitoring system sends:

// Example: Prometheus Alertmanager
{
  "alerts": [
    {
      "labels": {
        "alertname": "HighCPU",
        "instance": "server-01"
      },
      "annotations": {
        "description": "CPU is high"
      }
    }
  ]
}

2. Test Edge Cases

Test with:

Missing fields
Null values
Empty arrays
Special characters
Missing or wrong auth headers when auth is enabled

3. Check the JSON Syntax

fieldMappings and defaultValues must be valid JSON objects (no comments, no trailing commas). The API rejects invalid mappings on save with Invalid field mappings: ….

Common Mapping Patterns

Handling Different Severity Formats

Keep the source value in an attribute, give severity a default, and translate with policies (condition on the attribute → severity set):

"fieldMappings": {
  "attributes": { "source_severity": "severity" }
},
"defaultValues": {
  "severity": "warning"
}

When the source already sends ToBeAlert severities in lower case (critical, warning, …), map it directly: "severity": "severity".

Environment From a Tag

Mappings cannot search inside a list such as ["env:production", "team:backend"]. Map the list to tags, set environment in defaultValues, or send the environment as its own field from the source.

Alert Identity

There is no alert id template. Alerts are identified by environment, resource, and event: map resource and event to the fields that identify the problem on your side.

Getting Help

Check the error response - Look for detailed error messages
Inspect histories - GET /api/webhook_config/{id} → payloadHistory and errorHistory
Test with curl - Verify payload format and auth headers
Contact support - Include webhook configuration (without secrets) and a sample payload

Test Script

#!/bin/bash
# Test custom webhook

WEBHOOK_URL="https://your-tobealert.com:9000/api/webhooks/custom/your-webhook-id"

# Test payload
TEST_DATA='{
  "hostname": "test-server-01",
  "check": "cpu_usage",
  "severity": "warning",
  "message": "CPU usage is 85%",
  "timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%S.%3NZ)'"
}'

echo "Testing webhook: $WEBHOOK_URL"
echo "Payload: $TEST_DATA"
echo ""

response=$(curl -s -w "\n%{http_code}" \
  -X POST \
  -H "Content-Type: application/json" \
  -d "$TEST_DATA" \
  $WEBHOOK_URL)

http_code=$(echo "$response" | tail -n1)
body=$(echo "$response" | head -n -1)

echo "HTTP Code: $http_code"
echo "Response: $body"

if [ "$http_code" = "200" ]; then
  echo "✓ Webhook test successful!"
  alert_id=$(echo "$body" | jq -r '.alerts[0].id')
  echo "Alert ID: $alert_id"
else
  echo "✗ Webhook test failed"
fi

Common Issues​

401 Unauthorized​

400 Bad Request​

413 Payload Too Large​

403 Forbidden (disabled)​

Missing Data After Mapping​

Templates or Expressions Have No Effect​

Array Mapping Issues​

Debugging Techniques​

1. Test with Simple Data​

2. Inspect recent payload and error history​

Validation Tips​

1. Validate Input Structure​

2. Test Edge Cases​

3. Check the JSON Syntax​

Common Mapping Patterns​

Handling Different Severity Formats​

Environment From a Tag​

Alert Identity​

Getting Help​

Test Script​

Common Issues

401 Unauthorized

400 Bad Request

413 Payload Too Large

403 Forbidden (disabled)

Missing Data After Mapping

Templates or Expressions Have No Effect

Array Mapping Issues

Debugging Techniques

1. Test with Simple Data

2. Inspect recent payload and error history

Validation Tips

1. Validate Input Structure

2. Test Edge Cases

3. Check the JSON Syntax

Common Mapping Patterns

Handling Different Severity Formats

Environment From a Tag

Alert Identity

Getting Help

Test Script