Control theory for unattended jobs

A job that runs while nobody is watching has to decide, from a measurement it takes itself, whether to act, and it has to survive that measurement being wrong. Control theory names the four parts any system needs in order to hold a state steady: a setpoint, a sensor, an error signal, and an actuator. The reason it transfers to software is narrow and specific: those four names work as a checklist for whether an unattended job has actually solved that problem.

I’ve spent about three weeks making every unattended job in Command, the operator I run my projects through, write those four fields down, plus a fifth. The answers live in one registry file, and 38 jobs have an entry in it now [measured, that registry, read 2026-08-15]. Thirty-eight is what Command owns rather than everything that runs on this machine: the same registry names another 21 unattended jobs it excludes because they belong to other projects.

What follows is nine of those jobs, mapped. For each one: what it holds steady, how its four parts are wired, what the mapping caught, and where it still falls down. After the inventory come the grading results, the places where the vocabulary is only a metaphor, and the claims I can’t make.

The four parts, and a fifth question

• Setpoint. What “correct” means, written so that something could check it. On a thermostat, 20°C.
• Sensor. How the system measures what is true right now. The thermometer.
• Error signal. The measured gap between those two.
• Actuator. The thing that changes reality to close the gap. The furnace.

Then a fifth field that isn’t from the theory: how would you tell “I couldn’t measure” apart from “everything’s fine”? A thermostat whose thermometer wire snaps reads zero degrees and runs the furnace all night.

That’s the whole apparatus, and it took an afternoon to write down. The fourth field is the one that does the work, because there is no vague way to fill it in. “Writes a report” is visibly not a furnace once it’s sitting next to a setpoint.

The rule I graded them against

Six conditions. A job is a real control loop when all six hold:

1. The target state is checkable.
2. The sensor reads the world, not the worker’s report of its own effort.
3. The error signal is the gap between actual and desired.
4. Something acts on that gap.
5. The system checks whether the action changed the state.
6. A failure to measure stays distinguishable from a healthy reading.

Two of those I applied more strictly than the wording forces, and I’ll say where that changes the count when the count arrives. I read condition five as checks, not could check. And I only counted jobs I have watched do it unattended.

Nine jobs, mapped

Keeping a local service on the current build

A service on this machine has to be running current code. A keeper process checks it every five minutes and rebuilds it when it isn’t.

• Setpoint. The service is listening, and its build stamp matches the main line.
• Sensor. Compare the running process’s build stamp against main. An open port does not count.
• Error signal. Not listening, or the stamps differ.
• Actuator. Build a replacement, start it, and stop the old process only once the new one answers.
• Silent-failure handling. Partial, and its card says so. Fetch and build failures go to timestamped logs and leave the running service alone, which is the right fail-safe. Nothing alarms on a keeper that has been failing to rebuild for days.

The previous version of this watched the port. The service answered health checks for weeks while running 28 commits behind the branch it was serving. The replacement’s own header puts it flatly: the old one “correctly did nothing because it measured the wrong variable” [measured, that script's header].

Where it falls down: it verifies the entrypoint and the stamp, not behaviour. A rebuilt service that starts, answers, and serves wrong responses passes every check here. It is also blind whenever the machine is off. It has been ticking every five minutes while I wrote this, logging served == target and taking no action [measured, its run log, 2026-08-15].

Removing abandoned worktrees without deleting live work

Agent sessions create throwaway git worktrees, which are extra checkouts of a repository. A weekly sweep removes the ones nobody is using and escalates the ones that need a decision.

• Setpoint. Leftover worktrees and parked checkouts trend down week over week, no repository is left on an abnormal branch, and no worktree still in use is ever removed.
• Sensor. Count worktrees across the fleet against the prior run, measure how long each has been waiting, and check session evidence for whether anything is live inside it.
• Error signal. A flat or rising count, unresolved retirement claims, and the age of the oldest worktree waiting on a person.
• Actuator. Remove the ones that pass every protection check, and file one deduplicated escalation for the rest.
• Silent-failure handling. Handled at both ends. A run names itself clean, would-act, acted, awaiting-human, or degraded, and its exit code carries that verdict, so a run that removed nothing because its probes failed is no longer identical to a quiet week. Both used to exit 0.

Two things came out of this one.

The first is a live deletion, caught rather than hypothetical. A paired test ran the same set of worktrees through the old code and the new: removed went from 1 to 3, held from 4 to 7, kept from 13 to 8, and one worktree the old code called removable is now held, because it holds ignored content outside the disposable set, including 12 KB of another job’s own run records [measured, that paired test, 2026-08-08].

The second is a defect in the removal step itself. Removing a git worktree is not atomic: the command deletes the directory, then deregisters the worktree whatever the delete did. With a live process standing in the checkout it exits non-zero, reports that it failed to delete the path, leaves an empty directory behind, and drops the registration anyway [measured, against the git version on this machine, with a process held open in the checkout]. Both things here that remove a worktree then asked one question afterwards: is it still registered? That is false for a clean removal and equally false for that half-failure. So the removal got recorded as done, and the run’s verdict read “acted”. By 2026-08-12 there were 41 orphaned directories under one repository’s worktree folder, 37 of them empty shells, every run having reported success [measured, a listing of that directory, 2026-08-12].

Code review did not find that, and I had read the code more than once. It surfaced because somebody had to write an answer to the fifth question and there wasn’t one. The card’s own summary is four words: the sensor was blind to its own actuator.

Since 2026-08-12 both halves of the postcondition are measured. Four outcomes are named and only one of them is a removal. A half-failure lands in the errors list, the run reads “degraded” instead of “acted”, and it enters the aged backlog on the day it happens, reported by the run that caused it rather than found later as anonymous residue.

Where it falls down: nothing deletes an orphaned directory, deliberately, because the process that blocked the removal is the likeliest thing still holding it open. Those husks are the largest single component of the backlog and the least likely to be acted on, and the count only falls when a person acts. A single unreadable file under one agent runtime’s session store also makes every worktree read unknown, which stalls the sweep entirely.

The repaired path ran unattended for the first time at 08:05 on 2026-08-15: 44 leftover worktrees against 56 the week before, across the same 48 repositories, no precondition failures [measured, the sweep's run history, read 2026-08-15]. That is the direction I wanted and I can’t hand it to the fix. The repair classifies the outcome; it does not prevent the deregistration, so the count I’m quoting was never able to see the difference. The orphaned directories were already deregistered, so they were never inside the 56, and most of that pile was cleared by hand. And 44 is a stock rather than a flow, over seven days instead of the three between the previous pair of readings.

Making sure every pull request has a verdict on its current commit

Continuous integration is supposed to fire on every push. When that trigger silently fails, a pull request can carry a green check that belongs to an older commit.

• Setpoint. Every open, non-draft pull request carries a completed CI run on its current head commit, produced by the automatic trigger rather than by hand.
• Sensor. Read the runs attached to each pull request’s exact head commit and classify them: success, failure, running, cancelled, missing, early, unknown.
• Error signal. The count of open pull requests whose head commit has no verdict past a ten-minute grace period.
• Actuator. Re-fire CI on each stalled branch, then measure again.
• Silent-failure handling. unknown is a first-class state. A failed read is never missing, which would make the actuator fire at pull requests whose state was never read, and it never folds into a healthy zero. The job also tracks manual dispatches, so a broken automatic trigger cannot look healthy because a person kept supplying runs by hand.

The defect class it exists for is a green check attached to an earlier commit while the current head has never run. Its actuator was used by hand on 2026-08-06 to unstick the pipeline.

Where it falls down: nothing runs it at a cadence matched to the fault. A nightly routine carries it once a day and the interactive check is attended, so a stall arriving mid-morning can sit for hours. A fifteen-minute scheduled workflow would close that. The directory it would live in is one I’ve reserved to myself, so it’s a decision I owe rather than a task somebody can pick up.

Merging only the revision that passed

• Setpoint. A pull request merges only if its current head has a clean review verdict pinned to that exact 40-character commit, successful CI, no unanswered inline review threads, no label asking for a decision, and the host still permits the merge.
• Sensor. Re-read the head, merge state, labels, CI run, conversation comments, and inline threads.
• Error signal. A pull request that satisfies every current-head proof and is still open.
• Actuator. Squash-merge, pinned to the reviewed commit, and delete the branch.
• Silent-failure handling. Every required read fails closed. A failed, schema-incomplete, partial, malformed or truncated read is never accepted as a healthy zero, and mutable state is checked again immediately before merging.

This is the same defect class as the entry above, from the other side. “CI succeeded somewhere” and “CI succeeded on this exact revision” are different observations, and treating them as one is how reviewed work and merged work come apart.

Where it falls down: signature enforcement is still in a transitional mode, and flipping it is sequenced behind staging a signing key, so until then the strongest guarantee here is procedural. It also sits awkwardly in my own taxonomy. It’s filed as one of the jobs claiming to be loops, and its error signal (proven ready and still open) is real, but the behaviour worth having is the refusal, and a refusal has no target state to converge on. By the classification in the grading section below, it has as good a claim to being an interlock as anything I filed as one.

Finding repositories the fleet map doesn’t know about

• Setpoint. Every repository with activity in the last 14 days is either in the fleet registry or declared excluded with a stated reason.
• Sensor. Discover repositories independently from three sources, local git activity, pull-request activity, and agent session activity, and compare against the registry only afterwards.
• Error signal. The count of active repositories that are neither registered nor deliberately excluded, each carrying its name, days since activity, and which of the three sources saw it.
• Actuator. File one deduplicated issue naming all of them, landing in the same backlog another job already drains. An escalation that cannot be filed writes a local artifact instead, so the finding survives the network being down.
• Silent-failure handling. Handled at every level, with a deliberate ordering: a finding outranks a blind spot, but a clean read from an incomplete scan is never a pass.

Run offline against the registry as it stood, it found five uncovered repositories. Three were added, and it still found two that needed a decision. The first real escalated run was 2026-08-08.

Where it falls down: it isn’t independently scheduled, so it only runs when something else carries it. Its actuator also ends at a person by design, because deciding whether a repository belongs in the fleet is judgment rather than clerical repair. The loop closes on getting the discrepancy in front of the right decision-maker, not on editing the registry. Under a looser rule than mine that counts as a closed loop with a human in it.

Supervising delegated agent tasks

• Setpoint. Every queued, started, or unknown task has one valid disposition, one live parent session in its own runtime, one matching heartbeat naming the exact roster, a named return coordinator, and verified closeout evidence.
• Sensor. A board reads the durable launcher rows and emits separate error channels for disposition, ownership, closeout, parsing, and coverage. Git and the code host supply artifact evidence; each runtime’s own tools supply liveness.
• Error signal. The counts in those channels, plus rows it could not parse and runtimes it could not query.
• Actuator. The supervising session steers or resumes work, stops or supersedes it when the evidence warrants, serializes overlapping ownership, and records the correction back onto the durable launcher row so it outlives the session.
• Silent-failure handling. A runtime that cannot be queried stays undetermined and is never converted into “nothing is running”. Missing inventory becomes an explicit coverage error. A heartbeat on its own is not evidence that work is healthy.

One session supervised roughly 30 child tasks, waking hourly to produce a notify-or-don’t decision, and preserved a rejected checkpoint as a rejection rather than retrying it blindly or lowering the acceptance target [personal experience, from that session's own records].

Where it falls down, and this is the largest gap in the nine: enforcement is instruction-only. Collecting each runtime’s inventory, carrying the unqueried one as undetermined, choosing valid handoff evidence, running the startup reconciliation, verifying closeout. All of that is a written procedure a language-model session follows, not code that fails closed. The board itself is filed as an instrument rather than a loop, because it observes and something else acts.

Draining a defect backlog

• Setpoint. An item the ledger records as shipped is closed on the tracker within one firing.
• Sensor. Read open issues and the durable ledger, then classify items.
• Error signal. Open-defect backlog depth, measured before acting and not after.
• Actuator. Fix exactly one authorized item and open a pull request into the merge gate above.
• Silent-failure handling. Not handled, and its card says so in those words. “Shipped” means a pull request was opened, not that the defect closed, and nothing compares the two.

A leftover branch from a finished item marked that item permanently in flight, hiding 11 of 21 open items from selection. Separately, one ledger row had read shipped since 2026-07-27 while its issue was still open.

Where it falls down: this is the clearest negative example in the fleet. Every part is present except the one that would separate “I attempted the work” from “the problem is gone”. It measures the world before it acts and never afterwards, so its error signal can only ever describe the situation it inherited.

Turning session history into a wiki

• Setpoint. Every real session up to a stored read bookmark has been distilled and ingested, and the bookmark advances each night.
• Sensor. Re-scan after ingesting, compare the bookmark before and after, and record the run’s lifecycle before starting work.
• Error signal. Sessions still behind the bookmark, and consecutive runs where the bookmark doesn’t move.
• Actuator. Ingest the material and send the result through the merge gate.
• Silent-failure handling. Handled. A missing completion time marks an unfinished run, and an unreadable predecessor record is kept as a sidecar rather than discarded. It still cannot separate “crashed” from “running right now”.

The bookmark had never been written. Every scan therefore started at the beginning of history, hit its cap, and truncated, and the rule that advances the bookmark requires a scan that didn’t truncate. The job had been doing work every night without moving the state that represents progress.

Where it falls down: that is a designed verification sitting on a broken path, which is a different failure from the one above. This job had the right idea about what to measure, and measured a variable nothing ever wrote.

The overnight shift that was right nine nights running

The most useful entry here isn’t a job that failed. It’s one whose sensor worked perfectly and whose actuator could not fire, ever, by construction.

• Setpoint. Every self-check passes. Until 2026-08-08 it also had to close one pre-authorized item per firing; I retired that half after 14 firings produced nothing merged, so it no longer owns a target state for that work.
• Sensor. A self-check probing credential health, session continuity, the morning report’s health, loop-card coverage, CI liveness, unanswered review comments, fleet registry coverage, and follow-through on started work. Each probe returns pass, fail, or unknown.
• Error signal. Failing checks.
• Actuator. For most of what it finds, none that it owns. It reports or escalates.
• Silent-failure handling. Per-probe unknown stays distinct from both pass and fail.

Seventeen overnight shifts ran between 2026-07-27 and 2026-08-13. Nine of them detected a real problem, correctly, and went read-only without advancing anything [measured, its own run store, read 2026-08-13]. Don’t read that as nine out of seventeen. All nine fall before 2026-08-08, which is when I made the shift monitor-only, and after that date advancing nothing is the correct behaviour rather than a failure. The nine sit inside the window where it was still supposed to act, so the rate there is worse than the ratio suggests, not better.

The failure was the same one every time. The primary checkout was parked on a feature branch or carried a dirty tree, its preflight refused to let it write, and the run ended. The blockers were different every night: three different parked branches across the nine, assorted untracked directories, and one night a single untracked file. Its own log for 2026-08-06 names the pattern better than I would have: “the instance changes nightly, the class does not.”

The reason it couldn’t act is the transferable part. Repairing that fault meant writing to the one checkout the shift is forbidden to touch, and that prohibition is correct. So the sensor was right nine times out of nine and the actuator could not fire.

Where it falls down: one of those seventeen runs has been stuck reporting running since 2026-08-07, and nothing flagged it. It turned up when the runs were counted by hand on 2026-08-13. That is the exact silent-failure mode the fifth question exists to catch, sitting inside the job I was using to illustrate it. What the measurement bought here was an honest retirement rather than a repair: I kept the monitoring half, with the label on it saying that’s all it is.

What the grading found

Sixteen of the 38 used the form to stop claiming to be loops, and declared what they are instead, in a named category [measured, the loop registry, read 2026-08-15]:

The remaining 22 claim to be loops, and all 22 record a known gap. That says more about the authoring rule than about the fleet: an entry has to describe what a job is today, gaps included, and an aspirational entry is a worse violation than a missing one. A hundred percent disclosure rate tells you the rule is being followed. It tells you nothing about how big the gaps are.

Two of the 22 clear the first five conditions: the service keeper and the worktree sweep. On the sixth they differ, so the count depends on how strictly you read it. The sweep’s card rates silent-failure handling handled at both ends. The keeper’s rates it “PARTIALLY handled” and says why: nothing alarms on a keeper that has been failing to rebuild for days, so it degrades quietly to stale-but-serving [measured, that card's silent-failure field]. Read condition six strictly and the answer is one, not two. I’m counting the keeper anyway, because a job that verifies its own action and can still go blind is a different animal from one that never verifies at all, and that is a judgment about where to put a line rather than a measurement.

The sweep only started clearing the first five on 2026-08-12, when the removal repair landed. Before that date this section had one job in it under either reading.

The count is a function of the rule, so here is where else the rule can move and what happens when it does.

• Read condition five as could check instead of checks, and two near-misses come in. A nightly job rewrites a status page, re-reads it, and reverts if the rewrite didn’t take, which satisfies condition five on its face. I left it out because its actuator is a language-model session following a written procedure, and a night where that session skips the step leaves the page stale with the check never running to notice. The task supervisor above is out for the same reason.
• Loosen what counts as an actuator and the number rises further. An interlock that refuses a bad action is doing real control work. A job that files an issue a person then acts on is, to plenty of people, a closed loop with a human in it. Two more have a real actuator and a re-read designed in, and are out only because nothing runs them at a cadence matched to the fault.

The ones that aren’t close are easier to describe. Several act on the world and then take a self-report as confirmation: one records an item as shipped when what it did was open a pull request, one closes an issue on the strength of that pull request’s own body claiming the work is done, and one verifies a specification rather than the page actually published from it. Four are panels that only report while I’m looking at them. One has been dormant since I turned its schedule off.

What I’d defend is the rule, not the number. Two out of 22 is not a failure rate, and the other 16 were never candidates. Before this, the answer to “how many of these actually fix anything” was a guess.

The failure modes that showed up more than once

Four patterns turned up in more than one job, and they’re the part worth carrying to a fleet that isn’t mine.

1. The sensor measures availability instead of the state you care about. An open port is not a current build. A green check is not a green check on this commit.
2. The sensor accepts a self-report as the reading. “Shipped” meaning a pull request was opened. A removal reporting success because the registration is gone, without checking whether the directory went with it.
3. The actuator runs and the state never moves. The wiki digest ingested every night against a bookmark nothing ever wrote. The overnight shift detected correctly nine times with nothing wired to act.
4. The clock doesn’t match the fault. A nightly check against a fault that arrives mid-morning leaves hours uncovered. A weekly sweep against a backlog that grows once per session is always behind.

The first three are all failures of the same kind: the job measures its own effort rather than the world. The fourth is the one you can have while every other part is correct.

Where the vocabulary is metaphor

None of these are control systems in the sense a control engineer means. There is no continuous dynamics here, no plant identification, no optimal control, no tuned PID. They are digital, event-driven and rule-based. What’s actually borrowed is a design habit: name the target, measure the world rather than the worker, act on the gap, and refuse to let an unreadable measurement become a healthy zero. Claiming more than that would be dressing up a scheduled script.

The looser use is worse. “Use control theory to drive this to completion” means iterate until done, which is ordinary project management with borrowed words. Any piece of work can be described as a loop with its goal as the setpoint, which means the description excludes nothing and predicts nothing. My rule now: if a sentence with “setpoint” in it survives replacing that word with “goal”, use “goal”.

There’s a third category that is neither metaphor nor a feedback loop. The mechanism that decides how much authority automation gets is a governor: it proposes more freedom after a clean execution streak, stages that for my approval, and demotes immediately after an incident, with decision records, override rates and recorded incidents as its sensors. That’s supervisory control. Calling it a feedback controller would overstate it, because promotion needs a person and incidents are recorded by callers rather than inferred. Its own measurement is the one I’d point a reader at: on 2026-08-04 all seven items sitting on my desk for signature were work that was already authorized without me, an escalation precision of 0 for 7, with the same problem re-filed on four separate days [measured, that mechanism's own metrics section]. Nothing was measuring it, which is why it ran for nine days instead of two.

And the four questions can’t tell you which kind of job you’re looking at. Some of my loops end at me by design: anything that publishes under my name, anything that spends money, anything that decides something I should own. Those are open at the acting end, and closing them would be a governance failure rather than an upgrade. “This loop terminates at a human” is a defect in a nightly cleanup job and a feature in a publish gate, and nothing in the four questions distinguishes them.

What I can’t claim

The framing is about three weeks old in my repositories, dated from the first commit that uses the word “actuator” [measured, that commit, 2026-07-26]. Several of those days went into building the apparatus rather than fixing anything with it.

The registry isn’t complete either, and its own coverage check says so. I ran that check while writing this and it came back not-ok, naming six scheduled launchers on this machine that are carded nowhere [measured, the registry's coverage lint, run 2026-08-15]. So “every unattended job” means every one the lint could see on the day I read it. The detector that caught the gap is one of the 38, which is the fifth question working on the apparatus itself.

The obvious outcome measure would be my own count of known-broken items, and it can’t answer. Daily, from the day the framing arrived [measured, re-derived 2026-08-08 from the tracker's full issue history]:

19  17  17  18  17  17  16  15  17  22  24  25  26  19

That series stops on 2026-08-08, a week before the sweep reading above. The gap covers exactly the week holding the repair two of these entries describe, so the one number that might judge the whole exercise is the one I haven’t refreshed. It ranges from 15 to 26, and it was 15 two days before the mechanism I’d want to credit or blame even merged. The last reading is 19 because I closed ten issues in eighteen seconds that afternoon, twenty-two minutes before I took it. The series drops seven there, not ten, so about three more opened the same day: even the bookkeeping event doesn’t come through cleanly. A number that moves eleven across a fortnight and seven in an afternoon on ten closures is not an error signal yet, whatever else it is.

The job that cost me something outside my own machine isn’t one of the 38, and it never got a card at all. It lives in a different repository, and the registry’s reach stops at the edge of what Command owns. A checker on a comparison site I ran fetched each vendor’s product page weekly and compared the live price against what my site published. It failed on seven consecutive runs, from 2026-06-22 to 08-03 [measured, that workflow's run history]. Going through that batch by hand: five products had really changed price, two manufacturers’ sites had partly or completely vanished, one alert was the checker’s own mistake rather than a price change, and three of the wrong numbers were on pages Google had indexed [measured, the pull request that fixed it]. The measurement was good enough to act on and never clean. The alert went to a place I don’t read, so the checker sat dead for seven weeks and the site carried wrong prices somewhere inside that window. I can’t tell you how much of the window, because nothing was watching. The audit stopped at a repository boundary the failure didn’t.

One fleet, one operator, 38 jobs, three weeks. I think the cheap-first-three-parts problem is general, because a sensor is a script and an actuator is a decision about blast radius, and those costs aren’t specific to me. I can’t show you that from this data.

Try it on one of yours

Take one automated thing you run and fill in five fields. What does correct mean, written so something could check it. How do you measure what’s true now. What’s the gap. What acts on it. And how would you tell “I couldn’t check” apart from “everything’s fine”?

The fourth field is the one that’s hard to fake, which is why writing it next to the other three works better than asking about it on its own. If the answer is a log line, a dashboard, or an email, and a person isn’t meant to be the actuator, that’s where the surprises live. You don’t need the vocabulary for any of this. You need the fourth field to be hard to leave vague.