How jobs learn

Reflection is a model call after a run that writes what worked into the job’s playbook: notes for the next run, tested scripts and, once the runs follow the same steps, a main script that does the whole job without the agent. You review each change on the run’s Learned tab, and edit or roll back what the job learned on its Playbook tab.

Three modes

Umpteenth picks each run’s mode from the job’s playbook when it creates the run:

• Explore: the playbook has no active learnings, toolkit scripts or main script, so the agent works from your instruction alone.
• Assisted: the agent reads the learnings and calls the toolkit scripts as tools.
• Scripted: a main script does the job without the agent, and if its checks fail, the agent takes over in the same sandbox.

A job’s first learning or script moves it from Explore to Assisted, and three successful Assisted runs with the same tool calls graduate it to Scripted. If the main script falls back to the agent in two finished runs in a row, the job drops back to Assisted until it graduates again.

To keep a job on the agent, turn off Graduate to a script in the job’s Learning card (Managing jobs), and its runs never go past Assisted.

The playbook

A playbook has five parts:

• Learnings are short notes for the agent, such as “The status API returns at most 100 items per page, follow the next link”, each with an optional When condition.
• Toolkit scripts live in /ump/toolkit, and the agent calls them as tools.
• Setup is a shell script that runs before every run, in every mode. A non-zero exit fails the run.
• Dockerfile is the job’s environment, which you edit on the Environment tab (Sandboxes).
• Main and Verify arrive with graduation: the script that does the whole job, and the checks its result must pass.

The agent gets the active learnings and a one-line description of each toolkit script in its prompt, and the same text as /ump/PLAYBOOK.md in the sandbox. Every change to the playbook, yours or reflection’s, saves a new numbered version.

Reflection

With Self-improve on, the default for new jobs, Umpteenth reflects on each run that ends Succeeded, Failed or Timed out. It skips a run that failed before any work started, such as on the daily spend limit, unless the setup script caused the failure. It also skips a scripted run that succeeded without the agent’s help, so a graduated job costs no reflection while its script works. After you change a graduated job’s instruction, Umpteenth reflects on the next run anyway, so main catches up with the change.

With Self-improve off, the playbook changes only through your own edits and through Learn from this run, which starts reflection on one run that succeeded, failed or timed out.

Reflection uses the Reflection model from Settings → General → Default models, or the model the job runs with if that shows Same as the agent model (Models and costs). The run’s Learned tab shows what the reflection cost, and the cost counts toward the daily spend limit.

Checks before a change applies

Reflection answers with a list of changes, such as Add learning L4 or Graduate to a main script, each with a one-sentence rationale. Umpteenth checks every change on its own, so one bad change doesn’t block the rest, and marks it Applied, Held for review or Rejected.

Umpteenth checks each change the reflection model proposes before a run can use it. A risky change waits for you to apply it by hand, and if Umpteenth rejects a change, the model gets one retry with the reasons.

Before a new or changed main script goes live, Umpteenth runs it once in a shadow run: a fresh sandbox with the input of the run that reflection learned from. Main must pass the same checks as a scripted run, and if the source run succeeded within the last hour, report the same outputs, except those that verify lists under varies. If main fails, Umpteenth rejects it together with the changes it depends on, and the change offers Show the shadow run’s output.

Umpteenth skips the shadow run for a job whose spec or scripts declare side effects, since it can’t hold back the requests main sends on its own. It also skips it after a run that changed the job state, since main would start from a different state than that run did. The change applies untried, and the Learned tab says why.

Held for review

Umpteenth holds a change back instead of applying it in three cases:

• A Dockerfile builds on a base image the job doesn’t use yet, pipes a download into a shell, ADDs a file from a URL, or copies files out of another image.
• A learning, script or Dockerfile would contain the value of one of the job’s secrets, or something shaped like a credential, such as a GitHub token or an AWS access key.
• A Dockerfile changes for a job set to No network or Allowed domains only, since image builds reach the internet.

Umpteenth has no approve button for held playbook changes, so they never apply by themselves. To apply one, copy what you trust from the change on the Learned tab into the playbook with Edit as JSON on the Playbook tab. A Dockerfile goes into the Environment tab instead, followed by Save & build.

Suggestions

Reflection can’t change a job’s limits, secrets or instruction, so if a run shows that one of them must change, it suggests the change to you. It asks for more memory after a command ran out of it, for example, or for a token with the workflow scope a push needs. If the instruction leaves open whether a side effect such as an email happens, or a template the job uses contradicts the instruction, reflection asks which reading applies.

Graduation

A job graduates when reflection writes it a main script. Umpteenth allows that only while reflecting on the job’s newest finished run, which must have done the job’s work, and only if that run and the two runs before it that did the work meet three rules:

• All three succeeded in Assisted mode.
• Each made at most two bash calls that do more than read files. Commands such as cat, grep or jq don’t count.
• All three called the same toolkit scripts and MCP tools in the same order, whatever the arguments.

A run that found nothing to do, such as a digest on a day without news, ends its timeline with Finished with nothing to do and doesn’t count toward the three. Neither does a run with extra instructions. Neither kind breaks the streak, as long as it succeeded in Assisted mode.

If the rules hold, reflection writes a main script and its verify checks from that tool sequence, and once that change applies, the job’s next run is Scripted. Reflection still graduates a job with a step that needs judgment in only some runs, as long as none of the three runs needed it and main can spot it from facts such as an exit code. A daily rebase of a fork is the typical case: main rebases and pushes, and on a conflict it hands the run to the agent with ump handover (see Handover). If reflection finds a step main can’t do, the job stays Assisted, and the reflection summary on the Learned tab starts with “Not graduating:”.

On a new job, run 5 is the earliest possible scripted run: run 1 explores, its reflection adds the first learnings, and runs 2 to 4 make the streak. Umpteenth checks the rules only during reflection, so graduation needs Self-improve on, or a click on Learn from this run on the newest run.

Main is an ordinary shell, Python or Node script, and its version history on the Playbook tab leaves it better documented than most cron jobs. It runs as /ump/main in /workspace and reads the run’s input from /ump/input.json. Main can call toolkit scripts, MCP tools through ump mcp call, and the Utility model through ump llm for steps that need judgment, such as a summary. For a case that needs more judgment than one model call, main stops with ump handover and leaves the case to the agent. It reports results with ump output set and ump summary, and the ump CLI reference lists every command.

Scripted runs

A scripted run starts like any other, with the setup script if the playbook has one. Main then gets half the time left in the run, and the other half stays in reserve for the agent in case main falls short.

Once main exits, the run turns Verifying. Umpteenth checks that main exited 0 and set every output the job declares, then runs the verify checks. If they pass and verify sets "llm": true, the Utility model also judges the result against your instruction and success criteria. A pass ends the run as Succeeded.

The verify checks of a digest job that must find at least one item and write a Markdown file:

{
  "checks": ["output.count >= 1", "file /ump/outputs/digest.md exists"],
  "llm": false,
  "varies": ["fetched_at"]
}

A check takes one of five forms: exit_code == 0, output.<name> exists, output.<name> <op> <JSON value> with ==, !=, >, >=, < or <=, file <path> exists for a path under /ump/outputs/ or /workspace/, and stdout contains "<text>". varies names outputs that differ between two runs moments apart, like the timestamp fetched_at, so a shadow run leaves them out of its comparison.

A scripted run that passes costs only the ump llm calls its main script makes, plus one Utility call if verify sets "llm": true. The Graduation chart on the job’s Overview tab shows the drop.

Fallback

Main falls short if a check fails, if it calls ump fail, or if the verifier rejects the result or can’t judge it. The agent then takes over in the same sandbox, with the time left, the job’s toolkit and MCP tools, and a brief of what main did so far. The brief lists the MCP calls with side effects that main made, so the agent doesn’t repeat them, while requests main sent on its own, such as a curl to a chat webhook, get only a general warning. The timeline shows Fell back to the agent with the reason, the mode reads Scripted · fell back, and the run ends with the agent’s result. A run you cancel, or one that hits its time limit while main runs, ends there without a fallback.

After a fallback, reflection repairs main with Update the main script and changes only what failed. That needs Self-improve on, or a click on Learn from this run.

Handover

Main hands a run to the agent with ump handover <reason> when it reaches a case it can’t handle, such as a merge conflict that needs someone to read both sides. Umpteenth skips verification, and the agent takes over in the same sandbox with main’s reason, the end of its output and /workspace as main left it. The agent handles the case, does the steps main didn’t get to, and its result decides the run.

The timeline shows Handed over to the agent with the reason, and the mode reads Scripted · handed over. A handover means main works as intended, so reflection leaves main alone and adds learnings for the agent instead, unless the transcript shows main handed over by mistake. If main calls ump fail as well, the run counts as a fallback.

Demotion

Umpteenth demotes a job after its main script fails twice in a row, or after it hands over in three of its last five finished scripted runs since graduation. The job header shows Demoted, and runs stay Assisted until the job graduates again with three new successful Assisted runs and a new main script. One scripted success between two fallbacks resets the count. A handover between them doesn’t, and neither do repairs through Update the main script.

To hear about fallbacks, turn on run.fell_back under Notifications, where job.demoted is on by default. Handovers send no notification of their own.

The Learned tab

Each run’s Learned tab shows what reflection did with that run.

What this run taught the job lists each change with its rationale and badge, and a rejected change gives its reason after “Not applied:”. Playbook changes below it compares the new playbook version with the one before.

If reflection skipped the run, the tab offers Learn from this run, and a reflection that failed shows Reflection failed with the error. Reflection applies its changes to the job’s current playbook, so learning from an old run changes today’s version.

The Playbook tab

The job’s Playbook tab opens on the current version, such as Version 4, and lets you change it by hand.

• Learnings: edit a learning, or Retire and Restore it.
• Toolkit: expand a script to read its code, then Edit or Delete it.
• Run scripts: Setup, Main, Verify and Dockerfile, where the Dockerfile row links to the Environment tab.
• History: every version with its author and summary. Click a row to see what changed, and roll back with Roll back to version N in the row’s ⋯ menu.

Edit as JSON opens the whole playbook in one editor, with an optional Summary for the new version. Your edits skip reflection’s checks, so Umpteenth holds nothing for review and starts no shadow run. A rollback saves a new version with the old content, so the history stays intact, and starts an image build if the Dockerfile differs.

Adding a main script to a playbook that had none, through Edit as JSON or a rollback, graduates the job by hand. With Graduate to a script on, its next run is Scripted.

If a job doesn’t graduate

Open the Learned tab of the job’s newest run and go through this list:

• Self-improve is off, and you haven’t clicked Learn from this run on the newest run.
• Graduate to a script is off.
• One of the latest finished runs failed, timed out or ran in Explore.
• The newest run found nothing to do, so its timeline ends in Finished with nothing to do, or it had extra instructions.
• A run made more than two bash calls that do more than read files. Toolkit script calls don’t count.
• The runs that did the work took different paths: other tools, or the same tools in another order. Writing instructions has tips for runs that repeat.
• Reflection’s summary starts with “Not graduating:” and names the step main can’t do.
• Reflection wrote a main script, and it failed its shadow run. The change shows Rejected with Show the shadow run’s output.
• Reflection failed, for example on the daily spend limit.