Install on Kubernetes

Install Umpteenth on a Kubernetes cluster with the Helm chart, where every run’s sandbox is a pod and Umpteenth needs no Docker socket. The chart starts two replicas on a bundled Postgres, so schedules and webhooks keep firing when a node goes down, and keeps files in the database, so you need no bucket. On a single host, install with Docker Compose instead.

Requirements

• Kubernetes 1.30 or later, with amd64 or arm64 nodes.
• A network plugin that enforces NetworkPolicies, such as Calico, Cilium or kindnet. Umpteenth refuses to start sandboxes on a cluster that ignores them, see Network policies.
• A default StorageClass for the bundled Postgres.
• A container registry, if jobs build their own images from a Dockerfile, see Job images.

Install

1. Create a values.yaml with the address you’ll open Umpteenth at.

   values.yaml

   app:
     url: https://umpteenth.example.com
   
   ingress:
     enabled: true
     className: nginx
     host: umpteenth.example.com
     tls:
       - secretName: umpteenth-tls
         hosts: [umpteenth.example.com]

   That’s enough to sign in with passkeys, and Sign-in providers adds an identity provider or GitHub. The chart’s values.yaml describes every value.

2. Create the namespace.

   kubectl create namespace umpteenth

3. Install the chart.

   helm install umpteenth oci://ghcr.io/stonith404/charts/umpteenth --namespace umpteenth --values values.yaml

4. Open app.url, create the first account with a passkey on Set up Umpteenth, and connect a model.

   Do this right after the install, since whoever opens a fresh instance first becomes its admin.

Without an ingress, kubectl --namespace umpteenth port-forward service/umpteenth 8080:8080 reaches Umpteenth at http://localhost:8080, which then has to be app.url. With a model connected, the instance is ready for your first job.

Sign-in providers

To sign in through an identity provider or GitHub as well, register Umpteenth with it as Sign-in shows, and add it under config in values.yaml. The chart renders config into config.yml, so every option from Configuration fits there. Pass the client secret from a Secret through extraEnv, so it stays out of your values:

values.yaml

config:
  auth:
    providers:
      pocket-id:
        type: oidc
        name: Pocket ID
        issuer: https://id.example.com
        client_id: umpteenth

extraEnv:
  - name: AUTH_PROVIDERS_POCKET_ID_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: umpteenth-oidc
        key: client-secret

kubectl --namespace umpteenth create secret generic umpteenth-oidc --from-literal=client-secret='<client secret>'

Apply the change with helm upgrade and the same arguments as helm install. The chart sets the database, replica and sandbox options itself, through environment variables that win over config, so you change those through the chart’s values instead.

The encryption key

The chart generates app.encryption_key on install into the Secret umpteenth, and keeps it on upgrades and when you uninstall the release. Copy it next to your backups, since Umpteenth can’t decrypt what it stored without it:

kubectl --namespace umpteenth get secret umpteenth --output jsonpath='{.data.encryption-key}' | base64 --decode

To bring your own key, put it under encryption-key in a Secret and name that Secret in app.encryptionKey.existingSecret.

The database

The bundled Postgres is a single pod with a PersistentVolumeClaim, which suits a small team. For anything larger, point the chart at a Postgres you run with backups and turn the bundled one off:

values.yaml

postgresql:
  enabled: false
database:
  # A Secret that holds the postgres:// URL under connection-string
  existingSecret: umpteenth-database

Connection poolers

Umpteenth keeps a Postgres connection open to listen for updates, and holds a session lock while it migrates the database. A pooler in transaction mode, such as PgBouncer with pool_mode=transaction, breaks both, so connect the replicas to Postgres itself or to a pool in session mode.

Sandboxes

Umpteenth creates a pod per run in the release’s namespace, or in sandbox.namespace, and deletes it after the run. Your commands run as the agent user (uid 1000), as with Docker, and the pod gets no service account token. The chart grants Umpteenth a Role in that namespace for pods, their exec and log, Secrets and ConfigMaps, and nothing outside it. Sandbox pods fit the Pod Security level baseline, and sandbox.createNamespace: true together with sandbox.namespace has the chart create the namespace with that level enforced.

Two things differ from Docker:

• A command that runs out of memory ends the whole sandbox, since the kubelet kills every process in the container, and the run fails with ran out of memory and stopped while running a command.
• Kubernetes limits processes per node, so the 256-process limit of a sandbox doesn’t apply. Set the kubelet’s podPidsLimit to cap them.

For stronger isolation, install gVisor or Kata Containers on your nodes and set sandbox.runtimeClass to its RuntimeClass, such as gvisor. Use sandbox.nodeSelector and sandbox.tolerations to keep sandboxes on nodes of their own.

Network policies

The chart installs NetworkPolicies that let sandbox pods reach the broker port (8081) of the Umpteenth pods and nothing else, and no pod can connect to a sandbox. A job’s Network setting works as it does on Docker: Internet access and Allowed domains only sandboxes go through the egress proxy on that port, and Unrestricted network sandboxes reach public addresses and the cluster DNS, but no address in networkPolicy.privateRanges.

At start, each replica starts a sandbox without network and checks that it reaches the broker but not the Kubernetes API server. Sandboxes stay blocked until the check passes, including if the replica cannot discover the API server address. Umpteenth retries failed checks during maintenance. On a cluster that ignores NetworkPolicies, the check fails, the log shows Sandboxes can't start until the network check passes, and runs fail with the cluster does not enforce the sandbox NetworkPolicies. Set sandbox.requireNetworkPolicy: false to run sandboxes anyway, for example on a laptop cluster, knowing that they then reach everything in the cluster.

List your cluster’s pod and Service networks in sandbox.clusterRanges, such as [10.244.0.0/16, 10.96.0.0/12]. The egress proxy then refuses them for every job, so a job with Allow private network reaches your LAN but not the database or other services in the cluster. While the list is empty, the proxy refuses every private address, and Allow private network changes nothing.

Job images

Jobs with a Dockerfile need sandbox.registry.repository, since nodes pull job images from a registry:

values.yaml

sandbox:
  registry:
    repository: ghcr.io/acme/umpteenth-jobs
    username: acme-bot
    existingSecret: umpteenth-registry

The Secret holds the password or token under registry-password. Give that credential access only to sandbox.registry.repository, since build clients and nodes use registry credentials at host scope. Without a registry, runs of a job with a Dockerfile fail with the active sandbox adapter cannot build job images.

The chart runs a rootless BuildKit daemon in the sandbox namespace to build the images. Build steps run in separate process namespaces, and each workspace has its own namespace for mutable build caches. BuildKit requires pod user namespaces (hostUsers: false) and an unmasked /proc, plus unconfined seccomp and AppArmor profiles, so your nodes and container runtime must support these settings. A sandbox namespace the chart creates gets the Pod Security level privileged while builds are on. Unsupported runtimes fail the build; Umpteenth does not fall back to sharing the builder’s process namespace. The chart limits the shared builder to 1 CPU, 1 GiB of memory and 10 GiB of ephemeral storage by default through sandbox.buildkit.resources. A daemonless build pod has the same limits, with a 10 GiB limit on its build volume. Set the kubelet’s podPidsLimit to bound builder processes. An external BuildKit daemon must enforce process isolation and resource limits too. For a registry inside the cluster, add a rule for it to networkPolicy.build.extraEgress and set sandbox.registry.insecure: true if it speaks plain HTTP.

Replicas

The replicas share one run queue, and each takes up to its own runs.max_concurrent runs at once, so your capacity is the sum over all of them. A schedule fires once across the cluster. Any replica serves any page: a run page streams the output of a run on another replica, and Stop reaches the replica that runs it. Scale with replicaCount.

A failed replica

A run lives on the replica that executes it. If that replica goes down, another one fails the run within two minutes with interrupted: the replica executing this run stopped responding and sends the run.failed notification. Umpteenth doesn’t retry it, since a second attempt could repeat a side effect of the first, such as a message it already posted. Click Retry on the run page once you know that’s safe. The other replicas take over the failed replica’s queued runs, schedules and webhooks.

Upgrades

helm upgrade umpteenth oci://ghcr.io/stonith404/charts/umpteenth --namespace umpteenth --values values.yaml

Kubernetes starts a new replica before it stops an old one, and the replica that stops ends the runs it executes, as A failed replica describes, so pick a quiet moment. The first upgraded replica migrates the database, and Umpteenth doesn’t guarantee that a replica on the old version works against the new schema.