Welcome to the Era of Experience

A draft chapter for Designing an Intelligence. (Full text)

Abstract

We stand on the threshold of a new era in artificial intelligence that promises to achieve an unprecedented level of ability. A new generation of agents will acquire superhuman capabilities by learning predominantly from experience. This note explores the key characteristics that will define this upcoming era.

The Era of Human Data

Recent years have been the era of human data. LLMs trained on human-generated data and human judgments (such as RLHF) have made major advances.

But this approach seems unlikely to produce AI that surpasses humans.

The Era of Experience

The authors’ “era of experience” contrasts with the “era of human data.” The era of human data is ending, and the era of (machine) experience is beginning.

As agents become more capable, the data they generate must also improve. This is possible only if they are designed to learn from their own experience gained through interaction with the environment. Examples such as AlphaProof and DeepSeek show that the era of experience has already begun.

Experiential learning can bring major advances in AI. What is experiential learning?

Streams: Agents will exist in continuous streams of experience rather than short fragments of interaction. (Agents will inhabit streams of experience, rather than short snippets of interaction.)
Actions and Observations: Agents’ actions and observations will be richly grounded in the environment rather than in human interaction. (Their actions and observations will be richly grounded in the environment, rather than interacting via human dialogue alone.)
Rewards: Rewards will be grounded in experience of the environment rather than human prejudgment. (Their rewards will be grounded in their experience of the environment, rather than coming from human prejudgement.)
Planning and Reasoning: Agents will plan and reason about their own experience rather than reason in human terms. (They will plan and/or reason about experience, rather than reasoning solely in human terms.)

The following sections elaborate on each criterion.

Streams

Rather than performing well on independent short episodes such as “the user asks, the AI answers,” experiential agents will learn continuously throughout their lifetimes, acting toward long-term goals and adapting by learning new patterns over time.

This enables strategic actions that maximize the likelihood of achieving long-term goals.

(We should be particularly careful about what long-term goals we give agents. Like making a poorly worded wish to a genie: the wish comes true, but not as intended. Wasn’t the problem in that old(?) film “I, Robot” that humans failed to properly explain the Three Laws of Robotics to the AI agent VIKI? “Careful what you wish for…”)

(Seeing “streams” in this context brings to mind sensing for coupling in Supersizing the mind. The authors’ “continuous learning” could apply across various timescales, not necessarily just long-term learning.)

Actions and Observations

Experiential agents will act autonomously in the real world. Today’s LLMs indirectly experience the results of human experiences and actions in the real world through media such as “text.” This is one of the greatest differences from how animals, including humans, exist in the world.

(So far, “the real world” may seem to mean “the physical world,” but it need not. It may be more useful to interpret it broadly as “situations with persistent, substantive consequences.” For example, in agentic coding, how well the environment allows an agent to execute code directly and inspect the results makes a noticeable difference to performance. See: New practices for AI-aided programming.)

Rewards

The current approach (humans evaluating LLM responses) is not grounded in real-world consequences. Examples of proper reward signals:

Health agent: The user’s heart rate, sleep patterns, activity patterns, etc.
Education agent: Test results, etc.
Climate crisis agent: Carbon concentrations, etc.

A general-purpose AI agent could flexibly change its reward function according to the situation. For example:

If the user gives a broad goal such as “I want to improve my health,” the reward function returns a value computed from heart rate, sleep duration, step counts, and so on.
If the user says “I want to learn Spanish,” the reward function returns Spanish test results.

Planning and Reasoning

LLMs in the era of human data tried to imitate human reasoning (for example, Chain-of-thought prompting or fine-tuning using it). But we cannot be sure that reasoning like humans through human language is optimal. Self-learning systems such as AlphaProof teach themselves to prove complex mathematical theorems in ways very different from human mathematicians.

A conventional LLM trained on human utterances from 5,000 years ago would learn animistic reasoning; trained on data from 1,000 years ago, it would learn theological and theistic reasoning. Ultimately, conventional approaches struggle to transcend human limitations.

Agents should therefore collect data by actively interacting with the environment, repeatedly use it, and gradually improve their understanding of the world. One effective method is for agents to build their own “world models” and use them to predict the consequences of plans.

(This connects very well with the argument of The experience machine: How our minds predict and shape reality.)

Why Now?

RL has been around for a long time, but ChatGPT’s success ushered in the “era of human data,” and interest in RL declined. The direction of agents discovering new knowledge for themselves (AlphaGo Zero, etc.) was forgotten.

Once the four criteria above are met (continuous long-term learning, rich interaction with the environment, rewards based on substantive signals from the environment, and learning to plan and reason through environmental interaction), the “era of experience” will truly begin.

Reinforcement Learning Methods

Although it briefly fell out of fashion, RL is a powerful concept. In the era of experience, RL will develop further and play an important role.

Consequences

(Positive consequences omitted)

New challenges:

Jobs: Rapidly improving AI agents may eliminate more jobs.
AI misuse: Risks from agents operating autonomously over long periods and thinking differently from humans.

Experiential agents may also have advantages in preparing for risks:

Experience-based agents can perceive their environments and adapt their behavior to changes. They can work around broken hardware, adapt to social changes, embrace new science and technology, and recognize human concerns, dissatisfaction, and stress, adjusting their behavior to avoid negative outcomes.
Agents’ reward functions can also adapt through experience. For example, reward functions could be modified in response to signals of human concern.
Progress dependent on physical experience is constrained by the time needed to carry out real actions and observe results. AI-assisted drug development, for example, still requires real-world clinical trials, which cannot be completed in a day. These constraints may naturally regulate the speed of AI self-improvement.

Conclusion

The era of experience marks a pivotal moment in the evolution of AI. Building on today’s strong foundations, but moving beyond the limitations of human-derived data, agents will increasingly learn from their own interactions with the world. Agents will autonomously interact with environments through rich observations and actions. They will continue to adapt over the course of lifelong streams of experience. Their goals will be directable towards any combination of grounded signals. Furthermore, agents will utilise powerful non-human reasoning, and construct plans that are grounded in the consequences of the agent’s actions upon its environment. Ultimately, experiential data will eclipse the scale and quality of human generated data. This paradigm shift, accompanied by algorithmic advancements in RL, will unlock in many domains new capabilities that surpass those possessed by any human.