Exploration-exploitation dilemma
The dilemma between exploration and exploitation. Choosing the best option known so far to maximize gains is exploitation; choosing an option other than the best known one to find out whether there might be something better is exploration.
It is a dilemma because too much exploration reduces opportunities for exploitation, while too much exploitation increases the likelihood of mistaking a suboptimal option for the best one.
Horizon
When the number of opportunities to explore or exploit is limited, the number of attempts remaining is called the horizon. In general, when the horizon is near (few attempts remain), it is advantageous to reduce exploration. Conversely, when the horizon is far away (many attempts remain), it is advantageous to increase exploration.
Multi-armed bandit
A representative formalization of the exploration-exploitation dilemma is the MAB problem, and solutions to it are called MAB algorithms.
Books
- Bandit algorithms for website optimization