Version: 1.4 (Current)

Hotfixes

A hotfix bundle is a small .zip file, signed by tobeit, that fixes a release you already have installed — for example an unhandled error — without deploying a new image. Support sends you the file and an administrator uploads it in Management → Hotfixes. The whole flow works offline: the instance does not contact any external server.


What it fixes	Backend code, the Web UI (frontend), and, when needed, additive SQL changes. One bundle can carry any combination of the three.
What it is not	An upgrade path. A hotfix never moves you to another release (for example 1.3 → 1.4). Upgrade the images for that.
Which release	Each bundle is built for exact release(s). A bundle for another release is rejected.
Availability	From the release that includes Management → Hotfixes onwards. Older images cannot receive hotfixes.

Hotfixes are cumulative per release: <release>-hf1, <release>-hf2, … The newest bundle contains the previous ones, and only one is active at a time. Bundle ids are immutable — a rebuilt bundle always gets a new id.

For the rest of the install, see Docker or Kubernetes.

What you need

A release that includes Management → Hotfixes
An admin account (API scope admin:management)
The bundle .zip exactly as support sent it (maximum 50 MB) — do not unzip, edit, or repack it
A persistent volume mounted at /tobealert/patches in the backend container (see below)
For Web UI fixes: a frontend image from a release that supports hotfixes too — see Web UI fixes

The feature is blocked in demo mode.

Persistent volume for `/tobealert/patches`

The backend stores uploaded bundles, the active hotfix, and the history under /tobealert/patches.

warning

Without a persistent volume at /tobealert/patches, hotfixes are lost when the container is recreated. The application goes back to the code in the image — harmless, but the fix disappears until you upload and apply the bundle again.

The Docker and Kubernetes guides already include this volume. If your installation predates it, add it before you apply the first hotfix.

Docker Compose

services:
  tobealert-backend:
    volumes:
      # ...existing volumes...
      - patches_data:/tobealert/patches

volumes:
  # ...existing volumes...
  patches_data:

Then recreate the backend from $INSTALL_PATH:

docker compose up -d tobealert-backend

Kubernetes

Add the tobealerta-patches-pv / tobealerta-patches-pvc pair and the patches volume mount shown in the Kubernetes guide, then apply the manifests.

To use another directory inside the container, set the backend environment variable TOBEALERT_PATCHES_DIR and mount the volume there instead.

OpenShift and random user ids

Everything the backend writes under /tobealert/patches is created group-writable for GID 0 (the same convention as the other backend volumes), so a pod that starts later with a different random UID can still load, delete and roll back the hotfixes written by the previous one. No extra setting is needed.

Apply a hotfix from the UI (recommended)

Sign in as an admin and open Management → Hotfixes (/hotfixes).
Under Upload a bundle, click Select zip, choose the file, and click Upload.
The server verifies the bundle before storing it: the signature (Ed25519, tobeit key shipped in the image), the target release, and the checksum of every file. If any check fails, nothing is stored — see Troubleshooting.
Review the Uploaded bundle preview: Id, Target releases, Description, Changelog, and Contents (how many Backend files, Frontend files, and SQL files it carries).
Click Apply and confirm.
If the bundle contains backend code, the backend restarts by itself: the page shows Hotfix applied, restarting… and reloads on its own when the backend is back — typically 10–30 seconds. A bundle with only Web UI fixes and/or SQL is applied without a restart — the page just reloads.

An uploaded bundle that you have not applied yet stays in Staged bundles, with its Backend / Frontend / SQL file counts. You can apply it later.

What restarts

The backend restarts only when backend code changes — that is, when the bundle you apply, or the one it replaces or rolls back, contains backend files. Bundles with only Web UI fixes and/or SQL never restart it.

When it does restart, only the backend processes are relaunched inside the same container: the API server, the Celery worker, and Celery beat. The container itself does not restart, and the black-box keeps running. The application is unavailable during those seconds.

Automatic restart needs the image entrypoint

The automatic restart only happens when the backend runs with the image's own entrypoint.sh. If you start the backend another way (custom command or entrypoint), the change is saved and the UI tells you to restart the backend container yourself:

# Docker Compose
docker compose restart tobealert-backend

# Kubernetes
kubectl rollout restart deployment/tobealerta-backend -n tobealerta-interno

SQL in a bundle

Some bundles carry SQL files. They run when you click Apply, in a single transaction:

If any statement fails, nothing is applied or activated. The page shows the error and the file that failed.
SQL changes are additive and are not undone by a rollback.

Web UI fixes

A bundle with Frontend files fixes the Web UI. You apply it exactly like any other bundle; this section covers what is different.

Requirements

Both images must come from a release that supports hotfixes: the backend image stores and serves the bundle, and the frontend image knows how to load a patched UI.
The frontend container stays stateless: no extra volume and no new configuration. The only volume is still /tobealert/patches on the backend.
A reverse proxy or ingress in front of ToBeAlert needs no change, as long as /tobealert/… keeps going to the frontend container. The frontend's nginx forwards /tobealert/hf/ to the backend by itself, with the same BACKEND_SERVICE / BACKEND_PORT variables it already uses for /api/.

What users see

After Apply, the page reloads. Every browser gets the fixed UI on its next page load or refresh; users who already have the application open keep the previous UI until they reload. No cache clearing is needed.
Roll back or Turn off hotfixes returns everyone to the UI in the image, also on the next page load.
A UI fix is loaded only when it was built for exactly the frontend version in the image. If you upgrade the frontend image, the old UI fix is ignored automatically and the UI in the image is used.
If the backend is unreachable or slow when a page loads, the UI in the image is loaded after about 1.5 seconds — the same behaviour as without hotfixes (the application then shows its usual backend error page).
The bundle is treated as a unit: if the hotfix is Broken, its UI fix is not served either.

A bundle with UI fixes contains only the files that changed (a CSS fix is a single file of about 100 KB); everything else keeps coming from the image.

How the frontend's nginx serves a patched UI:

Each UI file is requested from the backend once per hotfix id and the answer is cached for 90 days, whether the file was in the bundle (200) or not (404). A 404 means "use the copy in the image", which nginx then serves directly, so files the bundle does not carry are never requested from the backend again.
Caching a 404 is safe because hotfix ids are immutable: the files of a given id can never change later. A new hotfix has a new id, so it starts from an empty cache.
The cache lives in /var/cache/nginx/hf inside the frontend container (300 MB at most, entries unused for 90 days are dropped). Recreating the container just means the first page load asks the backend again.
Installations without a UI hotfix make no extra requests at all.

Custom nginx configuration in the frontend container

Only if you replaced the nginx configuration shipped in the frontend image with your own: add these blocks, or UI fixes are never loaded (the app falls back to the UI in the image). This is what the image ships — adapt it to your file, or ask support:

# http level (outside "server")
proxy_cache_path /var/cache/nginx/hf levels=1:2 keys_zone=hf:10m max_size=300m inactive=90d use_temp_path=off;

# inside "server", before "location /tobealert"
# Patched UI files. A bundle only carries the files that changed: each file is asked to the backend once
# per hotfix id and the answer is cached (200 and 404 alike); what the bundle does not carry is then served
# from the image without asking again.
location ~ ^/tobealert/hf/[^/]+/assets/.+$ {
    rewrite ^/tobealert/hf/(.*)$ /ui/$1 break;
    # same backend host and port as your /api/ block
    proxy_pass http://${BACKEND_SERVICE}:${BACKEND_PORT};
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_cache hf;
    proxy_cache_key $uri;
    proxy_cache_valid 200 404 90d;
    proxy_cache_use_stale error timeout updating http_502 http_503 http_504;
    proxy_intercept_errors on;
    error_page 404 = @hotfix_baked;
}

# the file is not in the hotfix: serve the copy shipped in the image
location @hotfix_baked {
    root /usr/share/nginx/html;
    rewrite ^/(?:ui|tobealert/hf)/[^/]+/(assets/.+)$ /tobealert/$1 break;
    add_header Cache-Control "public, max-age=31536000, immutable";
}

# bundle upload: a zip of up to 50 MB (nginx rejects bodies over 1 MB by default)
location = /api/hotfix {
    client_max_body_size 64m;
    proxy_pass http://${BACKEND_SERVICE}:${BACKEND_PORT};
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

# application assets carry ?v=<build id> in every URL, so they can be cached without revalidation
location /tobealert/assets/ {
    alias /usr/share/nginx/html/tobealert/assets/;
    add_header Cache-Control "public, max-age=31536000, immutable";
}

# index.html must not be cached by browsers: it decides which UI is loaded
location = /tobealert/index.html {
    alias /usr/share/nginx/html/tobealert/index.html;
    add_header Cache-Control "no-cache";
}

Confirm the hotfix

Management → Hotfixes → Status shows Active hotfix, when it was applied, its state (Active, Disabled, or Broken), and its contents.
About (sidebar footer) → Software versions shows the Backend version as release (hotfix id) while a hotfix is active, and the Frontend version as version (hotfix id) when the patched UI is the one running in your browser. See About.
The backend startup log has a Hotfix line with the id that was loaded.

Roll back, turn off, delete

Action	What it does	Restarts the backend
Roll back	Returns to the previously applied hotfix, or to the code and UI in the image if there was none	Only if backend code changes
Turn off hotfixes	Keeps the bundle on disk but stops loading it. The backend runs the code in the image, and browsers load the UI in the image	Only if backend code changes
Turn on hotfixes	Loads the active hotfix again	Only if backend code changes
Delete	Removes a staged bundle that is not active. The active one cannot be deleted — roll back first	No

"Only if backend code changes" follows the rule in What restarts: a hotfix with only Web UI fixes and/or SQL takes effect on the next page load, with no restart.

Every action (upload, apply, roll back, turn off / on, delete) is written to the audit trail. The page also keeps a History table with date, event, bundle, and user.

Safety nets

A patched module fails to load — the backend uses the original module from the image and marks the hotfix Broken. You see it on the page and in the Hotfix line of the startup log. A Broken hotfix does not serve its Web UI fix either.
The server fails to start twice in a row with a hotfix — the hotfix is marked Broken and the backend starts with the code in the image.
A Web UI fix cannot be loaded — wrong frontend version, or the backend does not answer in time: the browser loads the UI in the image. See Web UI fixes.
Emergency kill switch — set TOBEALERT_DISABLE_HOTFIXES=1 on the backend container and recreate it. All hotfixes, including Web UI fixes, are ignored until you remove the variable.

When a hotfix is Broken, roll back or apply a fixed bundle from support.

Command line (no UI access)

Use the CLI inside the backend container when the UI is not reachable.

Command	What it does
`tobealertd hotfix list`	Show the active hotfix and the staged bundles
`tobealertd hotfix verify <zip>`	Check a bundle without installing it
`tobealertd hotfix stage <zip> [--replace]`	Verify a bundle and store it (does not apply it). `--replace` overwrites a staged bundle with the same id that was never applied
`tobealertd hotfix apply <id>`	Run the bundle SQL and make it the active hotfix
`tobealertd hotfix rollback`	Go back to the previous hotfix, or to the code in the image
`tobealertd hotfix disable` / `enable`	Stop / resume loading the active hotfix
`tobealertd hotfix history`	Show what was staged, applied, and rolled back

important

CLI changes do not restart the backend. Restart the backend container afterwards so the new code is loaded.

Docker Compose

The zip can be in any path the container can read — it does not need to be inside /tobealert/patches.

docker cp ./fix.zip tobealert-backend:/tmp/fix.zip
docker exec tobealert-backend tobealertd hotfix stage /tmp/fix.zip
docker exec tobealert-backend tobealertd hotfix apply <id>
docker compose restart tobealert-backend

Kubernetes

kubectl cp ./fix.zip tobealerta-interno/<backend-pod>:/tmp/fix.zip
kubectl exec -n tobealerta-interno <backend-pod> -- tobealertd hotfix stage /tmp/fix.zip
kubectl exec -n tobealerta-interno <backend-pod> -- tobealertd hotfix apply <id>
kubectl rollout restart deployment/tobealerta-backend -n tobealerta-interno

Upgrading to a new release

The new image already contains the fixes. Old bundles target the old release, so they stop loading automatically after the upgrade — you do not need to roll back first. Management → Hotfixes then shows the old hotfix as Not loaded (other release): use Roll back to clear it, and delete the old bundles.

The same applies to the Web UI: a UI fix is tied to the exact frontend version it was built for, so a newer frontend image ignores it and serves its own UI.

Environment variables

Set these on the backend container. You normally do not need any of them. The frontend container needs no hotfix variable.

Variable	Default	Description
`TOBEALERT_PATCHES_DIR`	`/tobealert/patches`	Directory where bundles, the active hotfix, and the history are stored
`TOBEALERT_DISABLE_HOTFIXES`	—	`1` ignores all hotfixes, Web UI fixes included (emergency kill switch)
`TOBEALERT_RESTART_EXIT_CODE`	`75`	Exit code the server uses to ask the image entrypoint for a restart after a hotfix change

Troubleshooting

Upload and apply errors carry a code. The UI shows a message for each one; the CLI prints the code.

Code	Meaning	What to do
`bad_signature`	The file is not signed by tobeit, or it is corrupted	Use the zip exactly as support sent it. Do not repack it
`version_mismatch`	The bundle is built for another release	Check the Backend version on About and ask support for the bundle for that release
`checksum_mismatch` / `invalid_bundle`	Corrupted or incomplete zip	Download the file again and retry
`too_large`	The file is larger than the allowed size (50 MB)	Confirm you are uploading the bundle and not another file
Upload rejected by a proxy (HTTP 413)	A reverse proxy or ingress in front of ToBeAlert limits the request body. Bundles with UI fixes can reach several MB	Raise the limit to at least 50 MB for `/api/hotfix` (nginx: `client_max_body_size`; ingress-nginx: annotation `nginx.ingress.kubernetes.io/proxy-body-size`), or load the bundle with the CLI
`already_exists`	A bundle with that id is already on the server	Apply the one already in Staged bundles. A rebuilt bundle comes with a new id
`forbidden_path` / `path_traversal` / `unknown_schema`	The bundle was built incorrectly, or for a newer server	Contact support
`sql_failed`	The SQL step failed on Apply. Nothing was applied	Send the error and the file name shown to support
`busy`	Another hotfix operation is running	Try again in a moment

Symptom	Likely cause
Management → Hotfixes is missing	The release does not include the feature, the account is not admin, or the instance runs in demo mode
The fix disappears after the container is recreated	`/tobealert/patches` is not on a persistent volume — check Compose volumes / PVC binding
Done, but this backend cannot restart by itself	The backend does not run with the image entrypoint. Restart the backend container
The backend has not come back after 2 minutes	Check the backend logs. After two failed starts the hotfix is marked Broken and the code in the image is used
The hotfix shows Broken	A patched module failed to load, or the server failed to start with it. Roll back or apply a fixed bundle
A CLI change has no effect	The backend container was not restarted after the command
A Web UI fix is applied but you still see the old UI	Reload the page (a normal reload is enough; Ctrl+F5 is not required). Check that About shows the hotfix id in the Frontend row. Check that the frontend image is from a release that supports hotfixes. With a custom proxy in front, check that `/tobealert/hf/` reaches the frontend container

Need help? support@tobealert.com.

Docker and Kubernetes — where the /tobealert/patches volume is declared
About — backend and frontend versions, with the hotfix id
Settings & environment variables — Compose and runtime configuration

What you need​

Persistent volume for /tobealert/patches​

Docker Compose​

Kubernetes​

OpenShift and random user ids​

Apply a hotfix from the UI (recommended)​

What restarts​

SQL in a bundle​

Web UI fixes​

Requirements​

What users see​

Custom nginx configuration in the frontend container​

Confirm the hotfix​

Roll back, turn off, delete​

Safety nets​

Command line (no UI access)​

Docker Compose​

Kubernetes​

Upgrading to a new release​

Environment variables​

Troubleshooting​

Related guides​

What you need

Persistent volume for `/tobealert/patches`

Docker Compose

Kubernetes

OpenShift and random user ids

Apply a hotfix from the UI (recommended)

What restarts

SQL in a bundle

Web UI fixes

Requirements

What users see

Custom nginx configuration in the frontend container

Confirm the hotfix

Roll back, turn off, delete

Safety nets

Command line (no UI access)

Docker Compose

Kubernetes

Upgrading to a new release

Environment variables

Troubleshooting

Related guides