Upgrades & Rollback

Applies to: ChengOS v0.2.0+ ｜ Last verified: 2026-09-25 ｜ Source: chengflow/docs/release-operations-guide.md, deploy/chengos.sh, deploy/lib/release-update.sh, CHANGELOG.md

Updates are upgrade-only, verified, and recoverable. Nothing irreversible happens before the new release has been checked, and nothing is recorded as installed until the new release has proven healthy.

Commands

./chengos.sh status              # version + health + latest-stable check
./chengos.sh update              # upgrade-only, verified, recoverable
./chengos.sh update --force      # reinstall or downgrade deliberately
./chengos.sh rollback            # restore the previous release
./chengos.sh version             # bare installed version, for scripts

Docker installs take --mode docker, and update additionally accepts --to <version> for an exact target.

What update does, in order

1. Read and normalize .chengos_version (v1.2.3 and 1.2.3 both read as 1.2.3).
2. Resolve the latest stable release. A failed lookup is a clear, non-destructive failure — never silently reported as "up to date".
3. Compare strictly remote > local. Equal or older does not update.
4. Download into a unique mktemp -d directory.
5. Verify the SHA-256 checksum, then the signature. A release that does not publish a .sha256 is refused — an unverifiable artifact is never installed. The signature step follows CHENGOS_REQUIRE_SIGNATURE: auto (default) verifies when a signature and a verifier are available, required refuses without a valid one, off is for development.
6. Verify archive layout, and that the embedded VERSION equals the selected release.
7. Check free space on both the staging and installation filesystems.
8. Back up current package-owned state into .chengos_backups/.
9. Stop services — and abort if the stop fails, before touching any file.
10. Swap package-owned directories atomically: each current directory is retired with a same-filesystem mv and the staged one moved into place, so a directory is always either the old release or the new one, never a mixture.
11. Start services and run a bounded health check.
12. Only now write .chengos_version and prune old backups.

If step 10, 11, or the health check fails, the previous state is restored and the previous service restarted automatically.

Two properties of that order are worth internalising: the version record is written last, so a crashed update never leaves a lie on disk; and the service stop is a gate, so a host that will not stop cleanly is never half-upgraded.

The ownership boundary

This boundary is never crossed, which is what makes updates safe to run on a system holding real data:

Backups contain package-owned content only. Your configuration, data, logs, and workspaces are never copied into a backup and never removed by a rollback.

Backups and retention

Backups live in .chengos_backups/<version>-<timestamp>/, and the newest CHENGOS_BACKUP_KEEP are kept (default 3).

A rollback consumes its rollback point, so a second rollback walks one release further back rather than repeating the same restore.

Docker updates

All four services resolve their tag from one CHENGOS_VERSION pin, so images from unrelated releases can never be combined. An update pins → pulls → recreates api ui app → verifies /health. On a failed pull, recreate, or health check, the previous pin is restored and the previous services recreated.

docker compose pull against latest is not an update. It destroys reproducibility and makes rollback ambiguous, because there is no specific version to return to.

While any CHENGOS_<SERVICE>_IMAGE override is set, coordinated release updates are disabled and status says so.

Data volumes — PostgreSQL, Valkey, Qdrant, and user volumes — are never deleted or recreated by an update or rollback.

When rollback is refused

Automatic rollback is a file-level operation. It is safe exactly as long as the previous release can still read the current database.

It becomes unsafe the moment a release applies a migration that drops a column or table, rewrites data destructively, or changes a type in a non-widening way. Such a release declares Migration policy: irreversible in its release notes, and both rollback paths refuse to run.

The declaration travels inside the package as config/.migration-policy, written from the release notes, so the check works offline. Rollback reads it from the currently installed release: when that file says irreversible, the updater refuses and prints the manual recovery procedure below. A package without the file is treated as reversible.

Every release section in the changelog must declare its policy:

Manual recovery from an irreversible release

1. ./chengos.sh stop
2. Restore the database backup taken before the update. The release notes give the exact dump/restore commands for that release.
3. Restore the package from the newest .chengos_backups/ directory (native), or set CHENGOS_VERSION back by hand in .env (Docker).
4. Write the restored bare version into .chengos_version.
5. ./chengos.sh start
6. Confirm /health reports the restored version.

Always take a database backup before updating a release that declares an irreversible migration. Nothing in the updater can recreate data a migration deleted.

Offline updates

Place a verified chengos-full-linux-amd64-v<version>.tar.gz (ideally with its .sha256) next to chengos.sh, and the updater uses it instead of the network. A local package with no embedded chengos/VERSION predates the release contract and is rejected.

Network timeouts

A metadata timeout in status prints latest version unavailable — which is not a claim of being up to date. A metadata timeout in update aborts without touching the installation.

After an upgrade

.env is instance-owned and never touched, which protects your configuration — and also means a release's new defaults do not reach an existing installation. v0.2.0, for example, binds the API to 127.0.0.1 and adds UI_BIND, APP_BIND, TRUSTED_PROXY_IPS, and the LAYA_* settings; an installation upgraded from v0.1.x keeps its old values and lacks the new keys.

After every upgrade:

diff <(grep -o '^[A-Z_]*=' .env.example | sort -u) <(grep -o '^[A-Z_]*=' .env | sort -u)
./chengos.sh doctor

The first line lists keys that exist in the new template but not in your .env (and the reverse). Add the ones you need, restart, and let doctor confirm what the host exposes. See Hardening a Public Deployment.

Incident recovery

Next steps

• Deployment modes — what you are updating.
• Installation problems — first-install failures.