Accidental Skynet
Even if computers do not become as intelligent as humans, and even if computers do not develop self-awareness and decide to drive humans out, a crisis for humanity caused by computers can still very realistically arise. If Skynet (the fictional AI that appears in the movie Terminator, which conquers humanity and rules the world) is ever created, I think it is more likely to be created accidentally and incidentally than intentionally by someone. (This is a post I wrote on my personal blog on 2016-03-17, lightly shortened and moved here.)
Scenario
There are countless accidental Skynet scenarios, but let me write down one of them.
- Stage 1. Stock trading by algorithms: There was already algorithmic trading starting in the 1980s, and in the case of the United States, it is said that 90% of current trading volume takes place through algorithms. Stage 1 has been achieved.
- Stage 2. Competition among algorithms: Competition among algorithms can arise naturally in the course of multiple companies competing, and it can also arise within a single company by operating multiple algorithms and adjusting the amount of capital allocated to each algorithm according to performance (e.g. Multi-armed bandit algorithm). Stage 2 has also already been achieved.
- Stage 3. Feedback loop: A feedback loop can be created in which trading activity itself affects the world, algorithms can detect changes in the world, and then use that again in making trading decisions. Algorithms can detect and exploit changes in the world not only through quantitative analysis but also through qualitative analysis (text mining, etc.). For example, in 2010 there was even a paper on a method of predicting stock prices through Twitter text analysis. Stage 3 has also already been achieved.
- Stage 4. The birth of Skynet: For example, if there is a pattern in which a decline in the stock price of shipping company A raises the probability of military conflict in region B, and as a result the stock price of defense company C rises, then an algorithm could exploit this by buying C stock in advance and then executing a strategy that influences the market in a way that negatively affects company A’s stock price. In other words, there could be countless scenarios in which the algorithm’s goal (maximizing profit) has the side effect of increasing the likelihood of war. Stage 4 appears to have been partially achieved.
The Role of Humans
The problem is that even if the individual or company operating the algorithm discovers this problem, they have little incentive to modify the algorithm. The profits are coming in nicely, and the war is happening on the other side of the planet.
- The algorithm does not need to have any motivation to preserve itself, nor does the mechanism for realizing that motivation (for example, a self-destruct device or offensive weapons for self-defense) need to be built into the system. In the stock algorithm scenario, the system is simply diligently doing the job assigned to it (maximizing profit through stock trading). The role of protecting the system from being shut down is handled by humans motivated by the desire to make money.
- The algorithm does not need to be superior to humans. It merely needs to be vastly more diligent than humans, vastly faster than humans, or vastly cheaper than humans.
Skynet Already Exists
Skynet already exists, and it is killing people in various ways. The difference from the movie is that 1) rather than being a fight of machines versus humans, it is a structure in which a small number of rich people become underlings of Skynet, maintain the current irrational system, and enjoy excessive luxury thanks to the concentration of wealth that results from it, and 2) rather than machines directly killing people, people die from starving, freezing, being unable to get vaccinated, or from wars among humans, all due to the concentration of wealth.