Avogadro Corp in the novel is obviously Google. Google’s name comes from googol, a large number, and the same goes for Avogadro (Avogadro’s constant). It feels like Udon Market in The Joys and Sorrows of Work.

One of Avogadro Corp’s main services is Avomail (Gmail? hahaha), which gets a feature where you roughly draft an email and AI polishes it for you. As the AI starts working better(?), it starts making increasingly bold edits to emails to further its own development, and eventually things happen.

Google I/O 2023 also demonstrated AI writing or improving emails for us. The difference between the novel and reality is that in reality (at least so far), AI only makes suggestions and humans retain control. Still, when I think about how much choice architecture such as nudges and so-called ‘dark patterns’ influence human decisions, I don’t think the mere fact that we have explicit control is enough to put us at ease.

I think the decisive moment in the novel was when a backdoor opened that allowed the email AI to modify its own code. So-called ‘recursive self-improvement’ became possible. The ending gets a bit absurd, though: the AI starts ordering weapons to physically protect itself.

Does AI really need to protect itself physically? Making humans dependent on AI (as with rice) so that humans protect it would be enough. For example, if some financial AI caused substantial harm to humanity as a whole but delivered enormous benefits to the richest 1%, they would use every means at their disposal to protect it. (See: Accidental Skynet)