Math guy, product person, enthusiast of startups, data consultant, and global markets aficionado - I have always been in a position where multi-armed bandits could be a game-changer in testing and optimization strategy, but it usually stays on the white board because it is perceived as too complex or maybe the business is just hesitant to try. So far, I have been successfully running MABs through different projects, and today I want to share those experiences, focusing on an interesting case from the gaming industry.
Understanding Multi-Armed Bandits
Let me consider two related but separate concepts: multi-armed bandits as algorithms of numeric decision-making and multi-armed bandit testing as an alternative to A/B testing. While the former is an analogy to classic testing, the latter is a fundamental algorithm that makes it all possible.
Imagine being in a casino and standing in front of a row of slot machines offering different rewards. The question is a classic dilemma: should you continue with the one which seems to give you the most, out of exploitation, or should you try others in hopes of finding an even better one, out of exploration?
This casino analogy perfectly illustrates the very basics of Multi-Armed Bandits. Each “arm” returns values we want to maximize or minimize, and in aggregate these form some sort of distribution over time. The difficulty is in accurately building those distributions while balancing data collection, exploration, with return maximization, exploitation.
Mathematical Foundation
The reward optimization over time is the problem that MAB solves. Mathematically speaking, ideally immediately identify the optimal distribution of the arm and then focus the remaining time in gaining rewards from that arm. Perhaps the most concrete implementation, epsilon-greedy, makes this a concrete concept with one simple rule: with probability 0.9, exploit-the best performing arm; with probability 0.1, explore-all other arms.
However, there are two big disadvantages to epsilon-greedy:
Solution Instability: It is hard to tune by using a fixed value of epsilon, and the performance can vary greatly for small variations.
Poor Adaptation: Epsilon-greedy does not adapt well in non-stationary systems where distributions of arms change with time; it takes much longer to converge to new optimal states.
MAB Testing vs. A/B Testing: A Detailed Comparison
Traditional A/B Testing
In classical A/B testing, there is a fixed division in an audience, analysis of metrics after a test period, and a rollout of the winning variant to the whole audience. While it is simple, it may be inefficient.
MAB Testing
A/B testing starts with equal-sized audiences, while MAB dynamically adjusts the sizes of each group based on the performance metrics. And instead of waiting for the complete rollout, MAB stops at the time it would show 95% of users the best-performing variation, retaining 5% for exploration.
More importantly, how well we balance exploration and exploitation determines the convergence time, i.e., the time when MAB reaches this ideal state. Also, the infrastructure for dynamic redistribution of users is needed for MAB testing. Thus, it does not fit into every testing scenario.
Contextual Enhancement
We enhanced the system’s performance further more by supplementing with additional contextual information relative to user behavior, including:
- Buying patterns relative to events in a game (Halloween, Christmas, etc.)
- Engagement with historical offers
- Player lifecycle stage
This contextual information will improve the quality of predictions made by the system, convergence speed, and reduce regret - the opportunity cost of suboptimal choices.
Key Takeaways and Best Practices
Test Duration: MAB can converge 2-3 times faster than A/B testing in appropriate cases.
Use Case Selection: One should refrain from over-engineering simple tests, such as button color changes, with MAB implementation. Choose your tools based upon specific needs and potential returns.
Synthetic Testing: Heavy backtesting before production deployment on synthetic examples emulating what is anticipated to be seen online.
Simplicity vs. Complexity: Whereas there are indeed many complex variants of MAB, sometimes simpler approaches work better. One should always consider whether extra complexity really serves one’s particular needs.
Conclusion
If implemented correctly, Multi-Armed Bandits are an extremely powerful optimization technique. The key to success is knowing what it can and cannot do, selecting appropriate problems to apply it to, and finding a balance in the degree of implementation complexity. Sophisticated mathematics can bring great power but does demand great care in its implementation not to make expensive mistakes.
Consider that not every test requires the complexity of MAB, but where dynamic environments and the need for real-time optimization arise, MAB can offer great benefits over traditional methods of testing.
SOCIAL SHARE CARD GENERATOR