Jake Van Clief on handing automation back to people

Key ideas

• Manual isn't a fixed thing. It's whatever layer of other people's automation you happen to be standing on.
• A program is three parts: instructions, something that runs them, and the state they leave behind. AI is the part that runs them, not the program.
• Instructions someone else wrote carry their choices, so borrow the neutral ones and write your own wherever your judgment matters.
• The tools that last augment people instead of replacing them, and every layer of automation so far has been handed back to a person.
• Prompts and context files are turning English into a set of small programming languages, one per domain.
• Working out where a person should take over from the automation is hard, and that difficulty is what makes a product last.

Jake Van Clief’s answer to what we should build now that AI can do the work is to treat AI as one part of a program, the part that runs instructions, and build tools that hand control back to the person using them. A program is instructions, something that runs them, and whatever they leave behind, and the instructions are where your own reasons live. These are my notes on his 34-minute talk, with the three older sources he builds it on: Gerald Weinberg’s 1971 book on programming, Steve Jobs’s bicycle for the mind, and Douglas Engelbart’s 1962 paper on augmenting human intellect.

What manual means

He opens by saying he wrote the whole script himself, no AI, and then takes the boast apart. Every cut, paste and bold runs code written over decades, so nothing he did was manual in any strict sense. Someone from an earlier age would say manual means cutting paper and using white-out. His point is that manual just means whatever layer you’re standing on, and the layer keeps moving up.

Levine the tailor

His anchor story comes from Weinberg’s The Psychology of Computer Programming. A customer picks up a suit from Levine, a famous tailor, and every flaw gets the same fix: hunch your back, lean left, lift a leg. The customer limps out bent into shape, and a stranger stops him, amazed that Levine could fit a suit so well on a man built like that. Van Clief’s reading: a lot of software, AI included, is sold by showing that people can be made to fit it. The best software fits the person instead, or gets as close as it can.

He pairs it with a line he attributes to Weinberg: because machines are rigid, the people who use them have to supply more than their fair share of flexibility. His question for the AI era is whether that need goes away now that software can bend further toward the user, or just moves up a layer.

What a program is

A program, he says, doesn’t need a computer. It has three parts: instructions, something that runs them, and the output or state that survives the run. The instructions can be a company’s standard operating procedures or the creative brief for an animation. The runner can be a room of clerks, an animator, a processor or an AI model. The state can be files in a folder, a slide deck or cloth off a loom. The looms of the 1800s ran on punched cards (pasteboard, in Jacquard’s 1804 loom), and the loom, the weaver and the cards together were the program.

The same goes for AI work today. A rules file or a prompt is part of the instructions. The structure of your folders is instructions too, and what lands in them is the state. AI is the part in the middle that reads instructions and hands back output, so the model only does well when it sees the right instructions, in the right order, and not too many at once.

Whose instructions

His recurring example is asking an AI to check your email. A few years ago that failed; now nearly every AI tool does it. The model didn’t change the email, so he asks which part of the program got better, and his answer is the instructions: someone wrote down what checking means and how to do it.

Most of those instructions are written by someone else, a company, another programmer or the people who trained the model, and every one of them carries that person’s choices. He calls programming a human behavior, closer to poetry than math. So his rule is to take the instructions that carry no opinion, like a connector to your mailbox, and keep your own reasons in the ones that do: why you’re reading the email and what you want done with it. He uses the same idea to explain why tools that make AI writing sound human get close and still miss. Good writing depends on who’s writing and who’s reading, and borrowed instructions are someone else’s suit.

Augmentation, not automation

Jobs called the computer the equivalent of a bicycle for the mind, in a 1990 documentary for the Library of Congress, Memory and Imagination, and he’d used the image since about 1980. Van Clief’s reading of the word choice is that a bicycle doesn’t replace the rider. It lets one person go further and decide where.

Engelbart’s 1962 report, Augmenting Human Intellect, defines augmentation as increasing a person’s capability to approach a complex problem, understand it and solve it. Van Clief points to Engelbart’s warning against what he called one-pass improvements, where the tool does the task and that’s the end of it, and to the loop Engelbart described instead: a better tool makes a more capable person, who builds better tools. Engelbart showed the mouse to the public in his 1968 demo; it had been built about four years earlier. Van Clief’s case is that the mouse, the bicycle, and files and folders lasted for decades because they let people point the tool at what they want. (Keyboards themselves go back to typewriters, well before Engelbart.)

He ties this back to layers. Every layer of automation so far, from machine code up to programming languages, ended up in a person’s hands. Machine code never went away; it just runs underneath. Two hundred years ago manual meant punching the card, 40 years ago it meant writing the code, and today it means writing the instructions, which he calls arguably the same job each time.

A new language

He thinks we’re inventing a new programming language, though not the one people expect. Weinberg, he notes, argued natural language was too loose to program in. Van Clief half agrees: what’s forming is a scattered set of domain-specific languages, English shaped into structured instructions a model can follow. In his telling, every instruction to a model, such as “tighten this paragraph”, carries a goal, a constraint, an assumption and the thing you want made, and the model’s reply carries its own. Write those down in a markdown file and you’re programming whether you meant to or not. He compares the likely result to legalese: a kind of English we use for speaking to computers.

He closes the argument with the question he opened on. Can people now bend software to themselves? Yes, more than ever, and he points to the big AI companies’ custom interfaces and shareable skills. Does that end the need for human flexibility? No: it moves up a layer. He’s clear this is an efficiency argument, not a moral one, and that working out where a person should take over is hard, which is exactly why it’s a moat.

His last image is a stack of seats. Everything around you was built by someone, so when you’re stuck on what to build, look at whose work you’re sitting on and build a seat for the next person who doesn’t have one.

How to try it

Four ways to use the model on something you’re building:

• List the parts of one AI workflow as instructions, runner and state, and see which instructions you actually wrote.
• Borrow the neutral parts, such as connectors, and write your own instructions wherever your judgment or taste comes in.
• Write down the why for an automation (what it’s for and who reads the result) where the model can see it, not just in your head.
• Find the step where a person should take back control, and build that hand-off on purpose rather than leaving it out.

Key terms

Program

Instructions, something that runs them, and the output or state that survives the run. It doesn't need a computer: a loom and its punch cards is one.

State

What a program leaves behind after it runs, such as files, memory, notes or a finished video.

Abstraction layer

A level of tooling that hides the one below it, as Python hides machine code, so a person can work higher up.

Augmentation

Engelbart's term for raising a person's ability to understand and solve a complex problem, as opposed to automating the person out.

One-pass improvement

Engelbart's phrase for a tool that just does a task once, with no loop where a better tool makes a better person who makes better tools.

Domain-specific language

A small language built for one kind of work. He argues good prompts and context files are becoming these, written in English.

Quotes from the talk