Upgrades and maintenance

Upgrade a single-node instance with a few commands, and keep an eye on it between upgrades. Umpteenth applies its database migrations when it starts, so an upgrade comes down to new images and a backup to fall back on.

Upgrade a single node

Read the release notes of every version since yours, on GitHub Releases or in CHANGELOG.md.

Note the version you run from the Umpteenth is starting line of docker compose logs umpteenth, since a rollback needs its tag. Look for renamed or removed options in the notes, since Umpteenth refuses to start while config.yml holds an option it doesn’t know.
Back up config.yml and data/ as Backups and migration describes.

Umpteenth migrates the database forward and has no downgrade command, so this backup is your way back.
Pull the new app image.
Terminal window
```
docker compose pull
```
Pull the new sandbox image.
Terminal window
```
docker pull ghcr.io/stonith404/umpteenth-sandbox:latest
```
Compose doesn’t manage this image, and Umpteenth never pulls a tag the engine already has, so without this step your sandboxes stay on the old image.
Recreate the container on the new image.
Terminal window
```
docker compose up -d
```
A run in flight ends when the old container stops, so pick a moment when nothing runs.
Check the log.
Terminal window
```
docker compose logs umpteenth
```
The Umpteenth is starting line names the new version, and Server listening means the server is up.
Rebuild the images of jobs with their own Dockerfile.

A job image stays on the sandbox image Umpteenth built it on until you click Rebuild on the job’s Environment tab (Sandboxes explains builds).

On Kubernetes, upgrade the Helm release instead, as Install on Kubernetes describes.

Pin a version

docker-compose.yml runs the latest tag, which moves to each new release. Each release also publishes the image under its version, such as 1.4.2, and moves the tags 1.4 and 1 to it. To upgrade on your own schedule, pin one of them:

services:
  umpteenth:
    image: ghcr.io/stonith404/umpteenth:1.4

The next tag follows the main branch and holds changes that no release contains yet, so keep it off instances you depend on.

Roll back

Umpteenth can’t undo a migration, so an older version needs the database from before the upgrade. Set image: in docker-compose.yml to the tag of the version you ran before, restore the backup from step 2 as Backups and migration describes, and start Umpteenth with docker compose up -d.

After a restart

Umpteenth comes back from an upgrade, a crash or a host reboot the same way. You stay signed in, and queued runs start once the server is up. A job whose schedule came due while Umpteenth was down runs once, so after a week offline your daily digest arrives once instead of seven times in a row.

Runs in flight don’t resume. If Umpteenth had no time to record how a run ended, it fails the run within two minutes of the next start, with an error that begins with interrupted:, and sends the run.failed notification, which is on by default. It never starts such a run again, since a second attempt could repeat what the first one did, such as posting a message twice. Click Retry on the run page once you know that’s safe.

Health check

GET /healthz on port 8080 needs no sign-in and answers 204 while the database responds within 3 seconds, or 503 with database unavailable. Point an uptime monitor or a load balancer at it. It checks the database alone, so it keeps answering 204 while the container engine or a model provider is down.

The image runs the same check every 30 seconds, and docker compose ps shows the result as healthy or unhealthy.

Logs

Follow the log with docker compose logs -f umpteenth. An error toast in the UI shows a Request ID, and the log line of that request carries the same value under request_id, so search the log for it to see the server’s side of the error.

At the default info level, Umpteenth logs failed API requests and leaves out successful ones. To log every API request for a while, set log.level to debug and restart, and see Configuration for the other log options, such as JSON output.