Intersection of Antikythera and Richard Sutton

I keep finding myself thinking about the conversation Richard Sutton had, Father of RL thinks LLMs are a dead end, together with Antikythera’s What Is Intelligence.

It feels like a number of different fields I’ve studied so far are getting woven together here. Listing them haphazardly so I don’t forget.

If you simply scale up the Brainfuck experiment from What Is Intelligence and let it run for a long time, would efficient RL algorithms and RL agents inevitably emerge? If they do not emerge, that could also be indirect evidence that the RL we know is not actually such a good method.
This also brings to mind the point made by David Silver (an early member of DeepMind and the first author of Welcome to the Era of Experience) in Is human data enough?: the less human intervention there is—data obtained from humans, human judgments such as RLHF, and so on—the better AI performs. He also says that algorithms obtained through meta-RL (making the RL algorithm itself with RL) turned out to outperform algorithms designed by humans.
On the other hand, one good way to keep RL algorithms from getting stuck in local optima would be environmental isolation. If each computer is treated as an isolated environment, then isolation is achieved naturally while distributed computing also becomes easy. Scaling up would be very easy. Moreover, according to Sutton’s Big World Hypothesis, individual agents do not need to become too bloated anyway (indeed, imposing constraints on them may be beneficial), so this would be highly favorable for distributed computing.
According to the argument of What Is Intelligence, it is crucial that there be other agents in the “environment” and that they interact with one another. Only when there are other agents trying to predict the environment can a high-dimensional arms race be triggered (the other party predicting my behavior as I try to predict the other party’s behavior).
If I combine the local-optimum point with the point about interaction with other agents, then this time Geoffrey Miller comes to mind. In The Mating Mind there is a critical analysis of the runaway brain hypothesis. (He was negative about the runaway brain hypothesis itself.) But he argues that the runaway process driven by sexual selection produces adaptive traits of a different kind from survival selection, which pursues only extreme efficiency, and that this makes possible a kind of “upfront investment” that can overcome local optima.
From the claim in What Is Intelligence that there must be other agents in the environment, I am also reminded of James J. Gibson’s ecological psychology. In the first place, in Gibson’s original concept of affordance, the most important “environment” for an organism is other organisms.
Sutton says that while RL can surpass humans in specific domains (obvious from cases like AlphaGo Zero), transfer is crucial if one wants to do A”G”I with RL, and no one has yet found a good way to do that in RL. I vaguely imagine that a neuroevolution approach might make such a thing possible. Wouldn’t it be a good combination if parameters containing domain-specific knowledge were fixed to some extent by an evolutionary algorithm, while what the agent learns online through experience fine-tunes those parameters? As I understand it, one reason transfer is difficult in RL is that new data overwrites existing parameters and damages patterns that were previously learned. If the large-scale wiring that contains domain-specific knowledge were fixed to some extent, wouldn’t that alleviate the problem a bit? Even in this case, I think there is ultimately something to learn from nature’s design. I wonder whether the fact that neuroplasticity is extremely strong in some respects and rather limited in others is not an accidental trait, but perhaps also a design produced by natural selection. And is the reason acquired characteristics are not (for the most part) inherited perhaps also a design rather than an unavoidable constraint?