Skip to content

Upgrade Guide

  1. Back up the database

    Terminal window
    ./scripts/backup.sh --db

    Database migrations are forward-only. Rolling the container tag back does not roll the schema back. See Backup & Restore.

  2. Diff your Compose files against the tagged release and merge the required changes before pulling images; follow Compose file changes, preserving your local configuration. Back up .env, the active Compose files, and overrides alongside the database backup.

  3. Update the version in .env

    Terminal window
    # Set to the version you want (e.g. 0.11.3)
    BREEZE_VERSION=0.11.3

    This also selects the agent release your fleet is offered, so set it to a version that is published on GitHub, not one that has only been tagged. See Binary Distribution.

  4. Pull and restart

    Terminal window
    docker compose pull
    docker compose up -d

    This pulls the new API, web, and binaries images and restarts all services. Database migrations run automatically on API startup when AUTO_MIGRATE=true (the default).

    Expected downtime: Typically 5–15 seconds while containers restart. If the new version includes database migrations, the API container will take additional time to apply them before accepting traffic – usually under 60 seconds for routine schema changes, but potentially several minutes for large data migrations. Agents continue running independently during this window and reconnect automatically.

  5. Verify

    Terminal window
    # Check API health
    curl -s https://breeze.yourdomain.com/health | jq .version
    # Check all running image versions
    docker compose images
    # Check agent version (on the device)
    breeze-agent version

    Also sign in through your public URL and confirm agents resume checking in. /health bypasses the HTTPS redirect and can stay healthy while a stale proxy address sends other requests into a redirect loop.

docker compose pull updates images; it does not download a new docker-compose.yml, override file, or .env.example. Current images can therefore run with old network settings or missing environment mappings. Review the configuration shipped with every target release, including when skipping versions; the entries below highlight changes that need operator action, not a complete Compose changelog.

Fetch a separate copy of the target release’s Compose file and compare it with the one you actually deploy. Replace the placeholder with the published release tag matching your intended BREEZE_VERSION:

Terminal window
release_tag="v<target-version>"
release_compose=$(mktemp)
curl --fail --show-error --location \
"https://raw.githubusercontent.com/LanternOps/breeze/${release_tag}/docker-compose.yml" \
--output "$release_compose" && diff -u docker-compose.yml "$release_compose"

diff exits with status 1 when differences exist; a failed download must be resolved before continuing. If you deploy deploy/docker-compose.prod.yml or additional override files, compare those same paths at the same tag too. Review the tagged .env.example and any configuration files referenced by changed services.

Merge the required changes into your backed-up local files, preserving custom services, domains, mounts, secrets, and proxy topology. Do not overwrite .env with .env.example. Add required variables to both .env and the appropriate service’s environment: mapping; setting .env alone does not pass a variable to the container. For digest-pinned deployments, update the image references to the target release as well as BREEZE_VERSION.

Validate the merged configuration before the pull/restart step:

Terminal window
docker compose config --quiet

Use the same -f options and profiles you normally deploy for validation, pull, and restart so your overrides are included. Validation checks the Compose configuration; it does not prove that a trusted proxy address matches the running proxy.

v0.95.0 — stable Caddy address and proxy trust

Section titled “v0.95.0 — stable Caddy address and proxy trust”

The v0.95.0 Compose file introduced these coupled settings. If your deployed file predates them, apply all three even if your images are already newer:

Compose setting Required action
networks.breeze.ipam.config.subnet uses BREEZE_DOCKER_SUBNET Merge the IPAM configuration. If the shipped subnet overlaps another Docker network, host route, or VPN, choose a non-overlapping subnet in .env.
services.caddy.networks.breeze.ipv4_address uses BREEZE_CADDY_IP Merge the static address pin. If you customize the subnet, choose an unused Caddy address inside it and set BREEZE_CADDY_IP alongside BREEZE_DOCKER_SUBNET.
services.api.environment.TRUSTED_PROXY_CIDRS defaults to BREEZE_CADDY_IP with a /32 mask Merge the derived default. An explicit TRUSTED_PROXY_CIDRS in .env or an override takes precedence: remove a stale override when using bundled Caddy, or set it to the actual immediate proxy’s stable address for a custom topology.

Keep TRUST_PROXY_HEADERS=true when using the bundled proxy. Do not widen trust to an entire private network to work around an address mismatch. A proxy’s dynamically assigned address can move after a container recreation or host reboot.

Changing an existing network’s IPAM/subnet requires recreating that network during a maintenance window; docker compose restart cannot apply it. For a standalone stack whose network is used only by that stack, stop/remove its containers and network with docker compose down, then recreate them with docker compose up -d after pulling the target images. Do not use down -v, which deletes named data volumes. If other containers share the network, coordinate their migration before recreating it. This network migration takes longer than the routine image-only restart described above.

Before upgrading a proxy-fronted deployment to v0.103.0 or later, verify that TRUSTED_PROXY_CIDRS matches the proxy that directly connects to the API. That release’s TRANSPORT-001 enforcement only honors forwarded HTTPS headers from trusted peers; a stale address can cause repeated 308 redirects for the UI and agents while /health still returns 200. After restarting, check public sign-in and agent check-ins as well as API logs for [proxy-trust] or [force-https] warnings (available from v0.104.0).

v0.110.0 — reboot deferral, device-set billing, QuickBooks payments, portal visibility

Section titled “v0.110.0 — reboot deferral, device-set billing, QuickBooks payments, portal visibility”

No breaking changes and no new required environment variables. Everything new is either off by default or optional. A number of new optional variables were added, though — see New optional settings below and the full Environment Variables reference.

Pull the portal container too. The customer portal runs in its own container and gained several new pages in this release. A bare docker compose pull covers it; if you pull services by name, make sure portal is on the list alongside api and web. If you have opted into the worker split, include worker as well.

53 migrations. Take a database backup first. All are idempotent and apply automatically on API startup (unless you have set AUTO_MIGRATE=false). Every index on a table that can be large is built concurrently, so writes are not blocked while they run — the trade-off is that an interrupted build leaves an invalid index behind. If that happens the next boot reports it loudly; recover with DROP INDEX CONCURRENTLY <name> and restart.

Three migrations clean up or stamp existing rows and record what they touched as a Postgres WARNING. Read these in the API log after the first boot:

Log line What to do if the count is above zero
cleaned N contract_lines rows whose site belonged to another org Re-scope those contract lines in the contract editor before your next billing run.
The site-name stamp on contract_lines This records how many existing site-scoped lines were protected. A line whose site was deleted before this release cannot be recovered and keeps billing organization-wide until a technician re-scopes it.
rls-scoped replay Informational. Five earlier migrations wrote rows without the right database scope and are replayed correctly here; the counts tell you how many rows were fixed.

None of these are required and none of them fail boot when missing. Full descriptions are in Environment Variables.

Variable Why you might set it
QBO_WEBHOOK_VERIFIER_TOKEN Makes recorded QuickBooks payments come back within seconds instead of on the 15-minute sweep.
FIREBASE_SERVICE_ACCOUNT Enables push notifications to the Android app.
AUDIT_CHAIN_VERIFY_MODE, AUDIT_CHAIN_VERIFY_RESCAN_SLICES Nightly audit-chain verification is now incremental by default rather than a full-table walk. AUDIT_CHAIN_VERIFY_ENABLED=false now also stops a sweep that is already running.
TICKET_OUTBOX_RETENTION_DAYS, INTENT_OUTBOX_RETENTION_DAYS, METRIC_ANOMALY_INCIDENT_RETENTION_DAYS (and their batch knobs) Tune how long the new nightly retention sweeps keep delivery and incident history. Defaults to 14 days.
SMTP_TIMEOUT_MS, MAILGUN_TIMEOUT_MS Bound how long a wedged mail host can hold an outbound send open.
BREEZE_AUTHENTICATOR_ATTESTATION_ENFORCED Already on by default. Only set it if you need to accept an approver device’s unverified platform-bound claim.
PARTNER_TRUST_MODE and friends Hosted service only — inert on a self-hosted install.
  • End-user restart prompts ship off. Turn them on per policy on the Patch tab with Let users postpone the restart. The deadline is the guarantee; the postponement count is a courtesy budget. Linux endpoints need zenity installed to show a dialog the user can postpone from — without it they get a plain notification and the restart proceeds on schedule. See Patch Management.
  • Unrated patches are never treated as Low and are excluded from auto-approval unless an update ring opts in. Expect a backlog of pending patches to appear that were previously auto-approved as Low. See Update Rings.
  • Deleting a site used by a site-scoped contract line now fails invoice generation with a clear error instead of quietly billing every device in the organization. Re-scope or remove the line, then generate again.
  • Audit-log retention now runs for every organization that has a policy. New organizations get a default policy automatically; existing organizations have no policy until an operator saves one, and nothing is deleted until then. See Audit Logs → Retention.
  • New partners are created with inbound email-to-ticket turned off. Existing partners are unchanged.
  • Windows 10 and Windows Server 2016 are the declared agent OS floor. Older Windows builds are refused by the installer rather than enrolling and failing later.

Both features run under the flags you already have — BREEZE_AI_AGENTS_ENABLED for the nightly impact rollup, and BREEZE_AI_AGENTS_POLICY_DECIDE_ENABLED (still off, still not generally available) for promoting an operation to pre-authorized execution. Automatic demotion is not gated by anything and is always on.

Three new daily background jobs are registered. They run in the container that owns background work — the api container by default, or the worker container if you have enabled the worker split:

Job Runs daily at (UTC) What it does
Graduation evaluation 18:28 Re-evaluates which agent operations have earned enough clean evidence to be promoted. Read-only with respect to what agents may do — it never grants or revokes anything.
Impact rollup 18:33 Rebuilds the last 7 complete UTC days of the AI Impact report per organization, or the full 90 days the first time it sees an organization. Skipped entirely while BREEZE_AI_AGENTS_ENABLED is off.
Evidence retention 18:48 Prunes agent outcome evidence older than 400 days.

All three sit in the existing daily lane and are deliberately spaced apart from each other and from the retention sweeps, so they do not add a new nightly spike.

v0.109.0 — AI agents, ticket attachments and org archiving

Section titled “v0.109.0 — AI agents, ticket attachments and org archiving”

Mostly additive. Nothing in this release requires a new setting before you upgrade, but note the following:

  • Migrations. 53 idempotent migrations auto-apply on boot. Two touch tables that can be very large on a long-running self-hosted instance: one builds an index on remote-session history, and one adds a validated constraint to time entries. Both take a brief lock while they run, so expect a slightly longer boot than a routine upgrade — proportional to how much remote-session and time-entry history you hold. Take the database backup in step 1 above and upgrade during a quiet window.
  • New AI agent capabilities are off by default. BREEZE_AI_AGENTS_ENABLED remains false, and even with it on, each agent starts in Off — an agent only acts after you deliberately move it to Act and confirm with multi-factor. See AI Agents.
  • New partners start with inbound email-to-ticket turned off. Existing partners are unchanged. See Ticketing.
  • Optional worker split. You can now move background jobs to their own container. Entirely opt-in and off unless you enable it — see Worker Split.

v0.95.0 — authentication & tenancy hardening

Section titled “v0.95.0 — authentication & tenancy hardening”

This release resets the authentication surface. Plan for a short low-traffic window and expect the following the first time the new API boots:

  • Everyone is signed out once. Existing web and mobile sessions are invalidated and everyone signs in again — there is no silent-refresh recovery and no kill switch. Confirm you and your admins have a working multi-factor device or recovery codes before upgrading, since signing in is the only way back in. Agents, the Helper, the portal, and installers are unaffected.
  • Connected AI / automation apps must re-authorize. Legacy OAuth refresh tokens are removed; any MCP/OAuth client reconnects and re-consents once.
  • Microsoft 365 ticket mailboxes need re-consent. Non-disabled mailboxes drop to a re-auth state; a Partner Admin re-approves each one. Mail arriving during the gap is picked up after re-consent, not lost.
  • New required database credential. Production now runs request handlers as an unprivileged, RLS-enforced role and will not start unless one of DATABASE_URL_APP, BREEZE_APP_DB_PASSWORD, or POSTGRES_PASSWORD is set, and it refuses to run as a SUPERUSER/BYPASSRLS role. See Environment Variables → Database. This check runs even when AUTO_MIGRATE=false.
  • Behind Cloudflare? Set TRUST_CF_CONNECTING_IP=true (see Environment Variables).
  • Migrations. 26 idempotent migrations auto-apply on boot. Most are metadata-only, but one backfills the refresh_token_families table — on a large, long-running deployment, check that table’s size first and expect a slightly longer boot while it applies.

Agents update themselves automatically. On each heartbeat (~60s), the API checks if a newer version is available and signals the agent to upgrade. The agent:

  1. Downloads the new binary and verifies its SHA-256 checksum
  2. Backs up the current binary
  3. Replaces the executable and restarts the service
  4. Rolls back automatically if the new binary fails to start
  5. Swaps the bundled breeze-backup helper to the matching version in the same upgrade; a periodic reconcile re-fetches it later if the swap was skipped or the helper is missing or stale

Since v0.105.0, release manifests can label each asset with an agent edition (self-host or hosted); the updater refuses an asset whose declared edition doesn’t match the running build, and accepts manifests without edition labels for backward compatibility. See Self-Host Agent Migration.

During an update, the device status changes to Updating in the dashboard so you can distinguish active updates from offline agents. If the update takes longer than 5 minutes, the device reverts to Offline. After a successful restart, the agent immediately reports its new version (bypassing the normal 0–60 second heartbeat jitter).

On Linux, the updater handles the case where the agent binary is still in use (common with manual installs that don’t use a package manager). The updater removes the old binary before writing the new one to avoid ETXTBSY (“text file busy”) errors. If a file lock error still occurs, it is treated as transient and retried on the next heartbeat.

The watchdog service automatically restarts the agent after a failed update or crash. If an update leaves the agent in a bad state, the watchdog detects the failure and restarts the previous (backed-up) binary.

The binary source depends on your BINARY_SOURCE setting:

Mode Agent downloads from
github (default) GitHub Releases CDN
local Your API server (or S3 if configured)

See Binary Distribution for details.

Some devices — production database servers, devices in a change-controlled window — should not self-update on the normal heartbeat cadence. Toggle auto-update per device without editing config on the machine:

Terminal window
# Disable auto-update for one device
curl -X POST "https://breeze.example.com/api/v1/devices/<device-id>/auto-update" \
-H "Authorization: Bearer <api-key>" \
-H "Content-Type: application/json" \
-d '{"enabled": false}'
# Re-enable later
curl -X POST "https://breeze.example.com/api/v1/devices/<device-id>/auto-update" \
-H "Authorization: Bearer <api-key>" \
-H "Content-Type: application/json" \
-d '{"enabled": true}'

The agent persists this setting across restarts. This is also the way to recover agents that were left with auto-update disabled by a manual or recovery update — see the set_auto_update command.

By default (AGENT_AUTO_PROMOTE=true), syncing a new agent release immediately makes it the fleet’s upgrade target — every auto-updating device moves to it on its next heartbeat. Boot-time sync only ever registers the release matching your own BREEZE_VERSION (or BINARY_VERSION if set), so a newly published GitHub release does not move your fleet until you upgrade your server images — the fleet follows the server, not the public release feed. If you’d rather stage rollouts (canary first, then the fleet), set AGENT_AUTO_PROMOTE=false to decouple registering a release from promoting it.

  1. Turn on controlled rollout. Set AGENT_AUTO_PROMOTE=false in .env and map it in the api service environment: block of your compose file, then restart the API. (Compose only passes through variables you list there.)

  2. Publish or sync the release. New binaries register as downloadable but are not promoted — the fleet stays on the current version. Existing devices are unaffected.

  3. Canary it. Update a few non-production devices to the new version and verify (you can target an individual device’s update directly).

  4. Promote to the fleet. When you’re satisfied, promote the version. This is a platform-admin action and requires MFA:

    Terminal window
    curl -X POST "https://breeze.example.com/api/v1/agent-versions/promote" \
    -H "Authorization: Bearer <platform-admin-api-key>" \
    -H "Content-Type: application/json" \
    -d '{"version": "0.84.0"}'

    Auto-updating devices then move to the promoted version on their next heartbeat. To promote a single component (e.g. just the helper or watchdog) while holding the rest, add "component": "helper" to the body.

Instead of always tracking the latest promoted release, you can pin the fleet — or a single customer — to a specific version. Settings → Partner Defaults and Settings → Org Defaults each expose an Update version targets card with independent Agent target and Watchdog target selectors. Each selector offers every registered version plus Latest promoted (track the globally promoted release, the default behavior).

Pin precedence is inherit-with-override — deliberately different from the lock model used by the update policy and maintenance window below:

  • A partner pin is the inherited default for all of that partner’s organizations.
  • Any org can override the partner pin with its own value — including explicitly re-selecting Latest promoted to move that org back to the promoted release while the rest of the fleet stays pinned.
  • Agent and watchdog pins resolve independently.

This is what lets a partner pilot a new version on one customer: pin the fleet to the known-good version at the partner level, then override a single org to the candidate version (or to Latest promoted).

Pins gate alongside the update policy and maintenance window, not instead of them — a pinned update still waits for the maintenance window and respects the update policy.

In addition to the per-device auto-update toggle and the partner-level controlled rollout, Breeze supports a structured maintenance window for agent updates. When a maintenance window is configured, agents only accept update signals from the server during the allowed window — outside of it, update prompts are deferred until the next eligible window.

Agent update maintenance windows are set in Settings → Org Defaults (and optionally in Settings → Partner Defaults to apply fleet-wide). The window configuration includes:

Field Description
Days Which days of the week updates are permitted (e.g., Tuesday and Thursday)
Start time Window start time in UTC
End time Window end time in UTC

The update policy and maintenance window follow the same partner-locks precedence that governs other policy settings: a field set at the Partner level wins and locks — child organizations cannot override it. The org value applies only where the partner has left that field unset, and each field is merged independently. The update gate that agents check on heartbeat honors this merged result.

If no maintenance window is set, agents update on their normal heartbeat cadence (~60 seconds after a new version is promoted). Configure a window when you need updates to avoid business hours or to concentrate restarts into a known change window.

If the API container fails to start after an upgrade due to a migration error, follow these steps:

  1. Check the container logs to identify the failing migration.

    Terminal window
    docker compose logs api --tail 100

    Look for lines starting with [auto-migrate]. Common failure messages include:

    • Migration checksum mismatch – a previously applied migration file was modified. This should never happen in normal operation.
    • SQL errors during a specific migration file (e.g., constraint violations, missing columns).
  2. Do not manually edit migration files. Migrations are tracked by checksum. Modifying a previously applied migration will cause all future startups to fail.

  3. Fix and restart. The most common causes and resolutions:

    • Incompatible extension (e.g., TimescaleDB not installed): Install the required extension and restart.
    • Permissions issue: Ensure the database user has sufficient privileges.
    • Data conflict: Manually resolve the conflicting data in the database, then restart the API container.
    Terminal window
    docker compose restart api
  4. If unrecoverable, restore your database from a pre-upgrade backup and re-attempt the upgrade after resolving the underlying issue.

The first release that enables shared remote-session ownership is a barrier deployment. Do not use a rolling or blue-green handoff for that release: legacy API processes can mint credentials that the new pre-upgrade admission path must not accept.

  1. Set REMOTE_ACCESS_ADMISSION_MODE=closed, REMOTE_WS_AUTH_MODE=post_upgrade, and REMOTE_WS_REDIS_TOPOLOGY=standalone-single-primary. Recreate Caddy and verify every terminal, desktop, VNC, WebSocket-tunnel, and HTTP-tunnel issuer/consumer returns 503 with Retry-After.

  2. Stop the entire old API pool. Verify no old API process, WebSocket, or tunnel-HTTP consumer remains reachable. Record the verified UTC stop times in REMOTE_WS_LEGACY_TICKET_WRITER_DRAINED_AT and REMOTE_WS_LEGACY_VIEWER_ISSUER_DRAINED_AT.

  3. Keep admission closed for the independent V0 credential drains: 60 seconds for legacy WebSocket tickets and 300 seconds for tunnel-HTTP tickets/cookies. The 60-second milestone does not permit reopen. At exactly 300 seconds, a uniform new pool may reopen in post_upgrade.

  4. Deploy the new pool, verify every API instance uses the release digest, and run the Redis preflight (standalone primary, cluster disabled, AOF enabled, maxmemory-policy=noeviction). Only then set REMOTE_ACCESS_ADMISSION_MODE=open and recreate Caddy.

  5. Independently wait 7,200 seconds plus a final 60 seconds for legacy viewer tokens and their last compatibility ticket. Only after that boundary may the complete pool switch to REMOTE_WS_AUTH_MODE=pre_upgrade. Startup rejects this mode when either verified timestamp is missing, malformed, future-dated, or too recent.

For production deployments where any downtime is unacceptable, you can perform a blue-green upgrade using two parallel Compose stacks:

  1. Deploy the new version alongside the old one. Use a separate Compose project name or directory with the new BREEZE_VERSION and point it at the same database and Redis.

    Terminal window
    COMPOSE_PROJECT_NAME=breeze-new BREEZE_VERSION=0.12.0 \
    docker compose -f docker-compose.yml -f docker-compose.override.yml up -d
  2. Wait for migrations and health check. The new API container applies any pending migrations on startup. Once the health endpoint returns the expected version, the new stack is ready.

    Terminal window
    curl -s http://localhost:NEW_PORT/health | jq .version
  3. Switch traffic. Update your reverse proxy (Caddy, nginx, Cloudflare tunnel) to route traffic to the new stack.

  4. Tear down the old stack. Once traffic is fully migrated, stop the old containers.

  1. Revert the version

    Terminal window
    # In .env
    BREEZE_VERSION=0.11.2
  2. Pull and restart

    Terminal window
    docker compose pull
    docker compose up -d
Terminal window
# API version
curl -s https://breeze.yourdomain.com/health | jq .version
# Agent version (on the device)
breeze-agent version
# Docker image versions
docker compose images

You can test a release candidate before making it available to users:

  1. Tag a prerelease

    Terminal window
    git tag v0.11.3-rc.1 && git push --tags

    CI builds all images but does not tag them as latest.

  2. Deploy the RC locally

    Terminal window
    # In .env
    BREEZE_VERSION=0.11.3-rc.1
    docker compose pull
    docker compose up -d
  3. Sync agents to the RC using the sync-github endpoint. This fetches the release assets from GitHub and registers them in the Breeze database so agents know a new version is available.

    Terminal window
    curl -X POST "https://your-domain/api/v1/agent-versions/sync-github?version=v0.11.3-rc.1" \
    -H "Authorization: Bearer <api-key>"

    The response includes the version that was synced and the list of platform/architecture targets that were registered:

    {
    "version": "0.11.3-rc.1",
    "synced": ["agent:linux/amd64", "agent:macos/arm64", "agent:windows/amd64", "helper:windows/amd64"]
    }

    Agents will pick up the new binary on their next heartbeat (~60 seconds). Omit the ?version= parameter to sync the latest stable release instead of a specific version. This endpoint requires system-level (admin) authentication.

  4. Test, then promote

    Terminal window
    git tag v0.11.3 && git push --tags

    This pushes latest tags — all users get the update.