# Upgrades & Rollback

> Applies to: ChengOS v0.2.0+ ｜ Last verified: 2026-09-25 ｜ Source: `chengflow/docs/release-operations-guide.md`, `deploy/chengos.sh`, `deploy/lib/release-update.sh`, `CHANGELOG.md`

Updates are **upgrade-only, verified, and recoverable**. Nothing irreversible happens before the
new release has been checked, and nothing is recorded as installed until the new release has
proven healthy.

## Commands

```bash
./chengos.sh status              # version + health + latest-stable check
./chengos.sh update              # upgrade-only, verified, recoverable
./chengos.sh update --force      # reinstall or downgrade deliberately
./chengos.sh rollback            # restore the previous release
./chengos.sh version             # bare installed version, for scripts
```

Docker installs take `--mode docker`, and `update` additionally accepts `--to <version>` for an
exact target.

## What `update` does, in order

1. Read and normalize `.chengos_version` (`v1.2.3` and `1.2.3` both read as `1.2.3`).
2. Resolve the latest stable release. **A failed lookup is a clear, non-destructive failure —
   never silently reported as "up to date".**
3. Compare strictly `remote > local`. Equal or older does not update.
4. Download into a unique `mktemp -d` directory.
5. Verify the SHA-256 checksum, then the signature. **A release that does not publish a
   `.sha256` is refused** — an unverifiable artifact is never installed. The signature step
   follows `CHENGOS_REQUIRE_SIGNATURE`: `auto` (default) verifies when a signature and a
   verifier are available, `required` refuses without a valid one, `off` is for development.
6. Verify archive layout, and that the embedded `VERSION` equals the selected release.
7. Check free space on both the staging and installation filesystems.
8. Back up current **package-owned** state into `.chengos_backups/`.
9. Stop services — **and abort if the stop fails, before touching any file.**
10. Swap package-owned directories atomically: each current directory is retired with a
    same-filesystem `mv` and the staged one moved into place, so a directory is always either
    the old release or the new one, never a mixture.
11. Start services and run a bounded health check.
12. **Only now** write `.chengos_version` and prune old backups.

If step 10, 11, or the health check fails, the previous state is restored and the previous service
restarted automatically.

Two properties of that order are worth internalising: the version record is written **last**, so
a crashed update never leaves a lie on disk; and the service stop is a **gate**, so a host that
will not stop cleanly is never half-upgraded.

## The ownership boundary

This boundary is never crossed, which is what makes updates safe to run on a system holding real
data:

| Package-owned (replaced and restored) | Instance-owned (never touched) |
|---|---|
| `bin/`, `ui/`, `app/`, `config/`, `skills/`, `node_skills/`, `workflow-templates/`, `infra/` | `.env` |
| `hybrid/{start,stop,status,generate-env}.sh` | Database volumes and `runtime/` |
| `chengos.sh`, `.env.example`, `VERSION` | `logs/`, `models/`, `workspace/`, `workspaces/` |

Backups contain package-owned content only. Your configuration, data, logs, and workspaces are
never copied into a backup and never removed by a rollback.

## Backups and retention

Backups live in `.chengos_backups/<version>-<timestamp>/`, and the newest
`CHENGOS_BACKUP_KEEP` are kept (default **3**).

A rollback **consumes its rollback point**, so a second rollback walks one release further back
rather than repeating the same restore.

## Docker updates

All four services resolve their tag from one `CHENGOS_VERSION` pin, so images from unrelated
releases can never be combined. An update pins → pulls → recreates `api ui app` → verifies
`/health`. On a failed pull, recreate, or health check, the **previous pin is restored** and the
previous services recreated.

`docker compose pull` against `latest` is **not an update.** It destroys reproducibility and makes
rollback ambiguous, because there is no specific version to return to.

While **any** `CHENGOS_<SERVICE>_IMAGE` override is set, coordinated release updates are disabled
and `status` says so.

Data volumes — PostgreSQL, Valkey, Qdrant, and user volumes — are **never** deleted or recreated
by an update or rollback.

## When rollback is refused

Automatic rollback is a file-level operation. It is safe exactly as long as the previous release
can still read the current database.

It becomes unsafe the moment a release applies a migration that **drops a column or table,
rewrites data destructively, or changes a type in a non-widening way.** Such a release declares
`Migration policy: irreversible` in its release notes, and both rollback paths refuse to run.

The declaration travels **inside the package** as `config/.migration-policy`, written from the
release notes, so the check works offline. Rollback reads it from the *currently installed*
release: when that file says `irreversible`, the updater refuses and prints the manual recovery
procedure below. A package without the file is treated as `reversible`.

Every release section in the changelog must declare its policy:

| Declaration | Meaning |
|---|---|
| `Migration policy: reversible` | The previous release can run against the migrated database. Automatic rollback permitted. |
| `Migration policy: irreversible` | The previous release cannot read the data. Automatic rollback refused. |

### Manual recovery from an irreversible release

1. `./chengos.sh stop`
2. Restore the database backup taken **before** the update. The release notes give the exact
   dump/restore commands for that release.
3. Restore the package from the newest `.chengos_backups/` directory (native), or set
   `CHENGOS_VERSION` back by hand in `.env` (Docker).
4. Write the restored bare version into `.chengos_version`.
5. `./chengos.sh start`
6. Confirm `/health` reports the restored version.

**Always take a database backup before updating a release that declares an irreversible
migration.** Nothing in the updater can recreate data a migration deleted.

## Offline updates

Place a verified `chengos-full-linux-amd64-v<version>.tar.gz` (ideally with its `.sha256`) next to
`chengos.sh`, and the updater uses it instead of the network. A local package with no embedded
`chengos/VERSION` predates the release contract and is rejected.

## Network timeouts

| Operation | Variable | Default |
|---|---|---|
| Release metadata lookup | `CHENGOS_META_TIMEOUT` | 10 s |
| Artifact download | `CHENGOS_DOWNLOAD_TIMEOUT` | 1800 s |
| Post-update health check | `CHENGOS_HEALTH_TIMEOUT` | 90 s |

A metadata timeout in `status` prints `latest version unavailable` — which is **not** a claim of
being up to date. A metadata timeout in `update` aborts without touching the installation.

## After an upgrade

`.env` is instance-owned and never touched, which protects your configuration — and also means a
release's **new defaults do not reach an existing installation.** v0.2.0, for example, binds the
API to `127.0.0.1` and adds `UI_BIND`, `APP_BIND`, `TRUSTED_PROXY_IPS`, and the `LAYA_*`
settings; an installation upgraded from v0.1.x keeps its old values and lacks the new keys.

After every upgrade:

```bash
diff <(grep -o '^[A-Z_]*=' .env.example | sort -u) <(grep -o '^[A-Z_]*=' .env | sort -u)
./chengos.sh doctor
```

The first line lists keys that exist in the new template but not in your `.env` (and the
reverse). Add the ones you need, restart, and let `doctor` confirm what the host exposes. See
[Hardening a Public Deployment](/docs/deploy-hardening).

## Incident recovery

| Symptom | Action |
|---|---|
| Update failed, service is down | The updater already restored the previous package. Run `./chengos.sh start`, then check `logs/cheng-api.log`. |
| Automatic restore also failed | The backup is intact. Copy `shared/*` back over the install root and `hybrid/*` over `hybrid/`, then start. |
| Running version differs from the record | The package was replaced by hand. Run `./chengos.sh update --force`. |
| `status` says `latest version unavailable` | Network or rate limit. Retry, or update offline. |
| Docker update left services unhealthy | The pin was restored automatically. Check `docker compose logs api`, then roll back to the previous version if needed. |
| A release must be withdrawn | Tags are immutable and never moved. Publish a new patch release. |

## Next steps

- [Deployment modes](/docs/deploy-modes) — what you are updating.
- [Installation problems](/docs/fix-install-issues) — first-install failures.