What Will Be Left for Us to Work On?

Calling AI a normal technology can sound dismissive. Narayanan and Kapoor use the word differently: AI may prove as consequential as the Industrial Revolution while still following a familiar route from technical invention to social change.

They divide that route into four stages. New methods and capabilities are packaged into products and applications. Early adopters experiment with them. Over time, adaptation reshapes organizations, professions, laws, and markets.

Models may improve quickly, but the later stages move on human and institutional timescales. A model release cannot integrate itself into a hospital, rewrite professional regulation, capture an experienced worker's tacit knowledge, or reorganize a company. Capability alone cannot tell us when economic change will arrive.

The electrification of factories makes the point. Early factory owners replaced steam engines with electric generators while leaving everything else unchanged, and saw little benefit. The real gains came when factories were redesigned around distributed motors and assembly lines. That redesign brought new worker roles, management practices, training, and labor rules. It took decades.

AI is likely to demand its own reinvention of work. Treating an agent as a drop-in substitute for an employee repeats the factory owners' mistake. Larger gains will come from new ways of organizing work around the technology, and an AI lab cannot discover those arrangements by itself.

Benchmark progress says little about whether an agent is dependable enough to own a task.

Narayanan's research group breaks reliability into consistency, robustness, calibration, and operational safety. Does the agent succeed predictably? Can it cope with imperfect conditions? Does it recognize failure? When something goes wrong, can the damage be contained?

A single accuracy score hides these distinctions. An agent that always succeeds on a stable subset of tasks can be deployed selectively. One that fails unpredictably 30 percent of the time poses a different risk, even if both receive “70 percent accuracy.” Work presented at ICML 2026 found that capability rose sharply across recent frontier models while measured reliability improved by much less.

The gap matters because automation and collaboration call for different systems. An automation agent acts without continuous human involvement and needs consistency, calibration, and safe failure. A collaboration agent works with a person who can redirect it, inspect its output, and absorb some unpredictability. Creative work may even benefit from variation.

For now, general-purpose agents face a trilemma: they can be automated, used in high-stakes settings, or remain broadly capable, but combining all three is unsafe. Collaboration has therefore advanced faster than full automation. The most useful systems remain in the hands of people who can supervise their work.

Software engineering is the talk's leading case study because coding agents arrived there early. The usual prediction is that higher productivity means fewer engineers. Both the history and the structure of the job complicate that prediction.

Writing code was never the whole job. Narayanan divides software work into a decide-execute-deliver sandwich. Teams turn customer needs into requirements and plans, write and debug the code, then integrate, test, maintain, and stand behind what they release.

AI compresses the middle, perhaps only a third of the job. The surrounding layers can expand as cheaper implementation lets teams attempt more projects, explore alternatives, serve narrower needs, and maintain more software. Deciding what deserves to exist, and ensuring it works in the world, becomes scarcer than producing code.

This pattern is older than AI. Successive programming abstractions made individual developers enormously more productive, while demand for software engineers grew with the amount of software society wanted. ATMs made routine banking cheaper but also helped banks open more branches, which still needed people. AI-assisted radiology has not made radiologists vanish; cheaper analysis can increase demand for analysis. Even human translation has remained more resilient than simple substitution stories predicted, partly because there is no fixed quantity of material worth translating.

Economists call the assumption of a fixed quantity of useful work the lump-of-labor fallacy. Greater efficiency can reduce labor for one task while expanding the field around it. Roles change, and demand can grow with the new capacity.

Narayanan compares the resulting role to a crane operator. The machine supplies the force; the operator understands it, directs it, and takes responsibility for its movement. AI can extend cognitive reach in the same way without choosing what should be built or where it should be placed.

Recursive self-improvement seems to threaten this gradual account. An AI system might begin improving its successors, accelerating capability beyond every historical analogy.

Narayanan takes the possibility seriously but separates four dimensions that advanced-AI debates often collapse into one sequence: recursive self-improvement, humanlike intelligence, economic transformation, and superintelligence. Progress in one dimension does not guarantee progress in the others.

A system might automate experiments that improve model speed or efficiency. That achievement could resemble highly scaled hyperparameter search rather than the replacement of an entire research community. Creativity, judgment, and the ability to form better representations of an unfamiliar problem present a harder challenge. We still do not know how to specify or verify those qualities cleanly.

At the same time, current AI may already be capable enough to drive change on the scale of the Industrial Revolution. The remaining barriers lie downstream in reliability, product design, organizational integration, tacit knowledge, regulation, and adoption. Economic transformation could be both likely and gradual, with no need to cross a finish line called AGI first.

Even imagined superintelligence runs into constraints outside computation. A model cannot compress a decade of clinical trials into an afternoon merely by thinking faster. It cannot predict chaotic systems beyond their inherent limits. Intelligence operates inside a world with physical processes, institutions, uncertainty, and other people.

AI improvements also extend human abilities. Human intelligence has always depended on accumulated knowledge and tools as much as biology. A person today appears “superintelligent” beside a time traveler from the distant past because they inherit language, institutions, science, and digital technology. AI joins that inheritance. The useful comparison is between AI systems acting alone and people whose reach is extended by AI.

That distinction returns politics and agency to a debate that often treats deployment as fate. Permitting AI systems to own companies, hire and fire people, or exercise unchecked institutional power would threaten human dignity and democratic control regardless of whether the models were “aligned.” Those arrangements are choices. Model safety cannot compensate for irresponsible institutional design.

Human roles will change. Narayanan's central prediction concerns where the work will go: as verifiable tasks are automated, effort shifts from building to evaluation.

Technical execution is often easier to verify: code compiles, a test passes, a model reaches a score. AI systems thrive where feedback is legible and abundant. Evaluation begins where those signals become insufficient. Someone must decide what to measure, which failures matter, whether an output serves its user, and what trade-offs society should accept.

The talk's second metaphor is a ship. Rowers provide propulsion and direction at the same time. Once an engine supplies the force, the crew's work divides into navigation, safety, maintenance, coordination, and command. Less effort goes into moving the vessel; more goes into deciding where it should go.

AI and machine-learning research may be undergoing the same transition. When building models was difficult, building itself dominated the field's status and attention. As more implementation becomes automatable, evaluating agents becomes a discipline of its own. Agents are stateful, act over long horizons, interact with changing environments, and can fail in ways that ordinary model benchmarks were not designed to capture. Serious evaluation requires domain knowledge, models of real users, and judgments that do not scale as neatly as model inference.

Benchmark-based evaluation is efficient, but it can narrow research to questions that fit beneath the benchmark's streetlight. If the community rewards only what can be ranked cheaply, it may become highly capable at traveling in directions it never consciously chose.

Thoughtful evaluation is alignment at the level of a research community.

Narayanan provocatively suggests that perhaps half of a conference like ICML should concern evaluation, or at least that evaluation deserves a dedicated track. Evaluation can bring technical progress closer to the destination people actually want.

Visions of “automated science” often describe a machine moving directly from problem to solution, as though a correct solution were the whole purpose of the enterprise.

Science also creates human understanding. Its explanations can be challenged, connected to other knowledge, taught, extended, and used to make decisions. When an agent produces a correct result through an opaque process, the missing understanding is a loss rather than a friction to be removed.

Narayanan therefore predicts that AI-enabled science will create tools and professions devoted to recovering human understanding from machine-produced solutions. The agent may search a vast space, run analyses, or suggest a result. People will still need to establish why it is true, what assumptions sustain it, when it fails, and what it changes.

Peer review should evolve in the same direction. Agents can handle reproduction checks, code inspection, and other routine work, leaving researchers more time for deeper evaluation. Automating the judgment itself would hand control over the direction of research to the systems being evaluated just as human attention to direction becomes more valuable.

Companies face a parallel challenge. If agents make building cheap, proprietary advantage moves toward knowing what “good” looks like in a specific business. Evals become a form of institutional knowledge. Narayanan proposes cross-functional evaluation teams with enough independence and authority to keep a company honest about whether its AI systems genuinely help users, merely improve an internal metric, or create risks that product enthusiasm has hidden.

Software, science, AI research, and business all show the same pressure. Implementation becomes more abundant while discernment becomes more scarce.

AI continually raises the floor, meaning what a system can accomplish on its own. It can also raise the ceiling of what a person working with AI can attempt. The floor rises with model capability. Raising the ceiling takes effort from the user.

Narayanan describes reinvesting some of the time AI saves into learning new subjects and experimenting with new workflows. This creates a tension between short-term productivity and long-term growth. Offload too little and the technology adds little value. Offload too much and the person stops developing the expertise needed to direct, question, and improve its work.

His phrase for the necessary effort is cognitive sweat. Some friction is how judgment develops. Using an agent as a black box can remove that experience, especially when we delegate a task before understanding it ourselves. Convenience can then produce a dependence spiral: weaker skill makes delegation more attractive, and delegation weakens the skill further.

Narayanan advocates deliberate augmentation: use AI aggressively, but stay close enough to inspect the work, learn from it, and remain accountable for it. He sees productivity, growth, and control as a three-legged stool that fails when any one is neglected.

This leads to Narayanan's final vision: co-superintelligence. The superintelligences of the future need not be autonomous systems operating above and apart from humanity. They can be humans whose capabilities are amplified by systems they understand, govern, and direct.

Computers have long been called bicycles for the mind. Narayanan proposes AI as a crane for the mind, capable of lifting human ambition higher while leaving people in control.

So what will be left for us to work on?

The work moves outward from execution. People will decide which problems deserve attention, determine what good work looks like, translate machine outputs into human understanding, and build institutions that keep powerful systems answerable to human purposes.

This is a cautious optimism, and it depends on practice. Human agency remains valuable only when people exercise it. Rather than waiting for AI to assign us a role, we can take responsibility for where it leads.