I’ve heard it said that in 50 years time, Charles Darwin will be seen as the father of modern economics, rather than Adam Smith. Nowhere could this be more true than in social games, though with books such as The Lean Start Up in vogue, there is clearly a wider resonance.
Darwin’s theory of evolution by natural selection is a process by which systems optimize themselves if they have three properties:
- Variation between individuals
- Selection of just a few individuals to reproduce
- Inheritance of traits from one generation to the next
Whilst these three properties clearly apply to living organisms, they equally apply to social games using A/B testing. With this lens, the fastest ways to optimize your games by speeding their evolution becomes clear.
Variation must come at two levels in the games industry – tiny changes between different versions of the same game to optimize them when running, and huge changes between different games when starting a new project. In either case, the more variation you have, the faster you can evolve.
The former allows the perfection of a game over time once it is up and running. Selection between siblings with minor differences gradually optimizes a species to the niche it finds itself in. The tongues of anteaters could only become so specialised with variation in tongue length and stickiness of anteater antecedents. The more segments that you assess with A/B testing, the faster you narrow down on the optimal solution. Don’t A/B test when you can A/B/C/D/E test.
Having large variation in prototype games allows developers to avoid the games equivalent of evolutionary dead ends. No doubt Tyrannosaurus Rex was highly adapted to the Cretaceous period 65 million years ago, but it was mammals that took its place after the great extinction event. Similarly Pac-Man was a huge success in its time, but there aren’t many people playing it now. The only way that developers can avoid missing a new gaming trend is to constantly take big, innovative risks. Every time you start a new project you should start with a Cambrian explosion of diversity, and go from there.
Selection is the process that tells you what works, and what doesn’t. The only people that can tell you that are your customers – the people who are paying for your games. Great creative talent internally comes up with variation, but it’s the marketplace that tells you which variants are the best. Field mice rear each litter member the same – they have no idea which will survive to reproduce and which will not. It’s the foxes and owls that determine that once they leave the burrow.
Of course, not everything can be A/B tested, and not everything needs to be A/B tested, but there’s no point debating internally whether a feature is a good one or not. Whatever conclusion you come to will be qualitative and subjective, and cannot predict how players will respond to that feature when it goes live.
Running an A/B test doesn’t ensure your game immediate success. It allows you to determine the best solution from the options you’ve created. Evolution takes time, and a program of A/B testing needs to be part of a developer’s routine, with tests being run regularly, and for enough time for the compound effects to accumulate.
You probably look more or less like your parents, and they more or less like theirs – the amount of variation in biology is incredibly small. Give that variation 250,000 generations though, and you have enough variation to separate humans and chimps. Luckily, social games can evolve a bit faster, as both the variation between generations is greater, and the time between generations shorter.
At GDC this year Jens Begemann, CEO of Wooga, revealed that a 1-3% improvement was typical when they ran an A/B test, if they saw any difference between segments at all. That’s not going to turn your game into a hit overnight. But run A/B tests frequently enough and these tiny improvements compound. Get a 20% improvement after 20 A/B tests and slowly things are starting to look interesting.
Similarly, the shorter your generation time, the faster your games will evolve. If you’re running an A/B test every couple of days, then in a month you can test 10 different variations, and get the compound improvement of all of them. Run a test every 2 weeks and you’re looking at 5 months for the same level of improvement.
If you want to evolve your games towards their optimal design then you need to A/B test them. To make your A/B testing program as effective as possible make sure that:
- Variation: you have the greatest amount of variation between segments as possible
- Selection: you let your customers decide which variants are best
- Inheritance: you minimise time between A/B tests and give games enough A/B tests for results to compound.