June 3rd 2020
Marketing Manager @SplitMetrics & SearchAdsHQ. Content creator with 6+ years of experience in IT
Game publishers may have no idea that they have weak marketability until they soft launch their product, and at this point they have already invested huge resources into the development. Yet, there’s a way to make sure that the game can hit its business goals before writing a single line of code. I’m referring to game concept testing.
Concept testing happens on emulated pages of the App Store and Google Play. Basically, you show the product page of a game that you intend to develop to your target audience to see how many of them click the “Get” button.
Now let’s dig deeper into…
Why use game concept testing?
You can start with A/B testing the game subcategory to answer the question “Should I develop a card game or a puzzle game?” This is the most general thing you can A/B test.
A/B testing subcategories: card game vs puzzle game
Additionally, game publishers can compare a few mobile game concepts that they are considering and select the most well-performing one. You can also choose between a few positioning strategies or understand the most well-performing audience, like should it be a game for kids or adults, etc.
A/B testing game positioning: quiz on brand guessing vs quiz on movie guessing
Details can be validated too. For example, you can check which of the game features resonates better with your audience and once you understand them, get down to the development.
A/B testing game features
Key game concept testing applications
Overall, there are two major applications for game concept testing.
1. Theme testing.
Theme testing can be done early on, at the prototyping stage of the game development when you are figuring out what type of game you should be creating.
Here, you A/B test the core gameplay, or setting, as well as the elements in it. For example, the theme can be Middle Ages or Fantasy, and the elements can be dragons or zombies, etc.
A/B testing two settings: space vs water world
Tests help game publishers see what users are more interested in, as well as find a less resonating theme, which is an equally important insight.
2. Increasing IPM, or installs per mille
When you manage to increase the IPM (installs per thousand of ad impressions), you can lower your CPI. In other words, the better your ad and the app store page performs, the cheaper it is to acquire users.
By experimenting with different styles and app store optimization and tying the creative to the gameplay, you can arrive at something that might resonate better than your current elements, and thus impact your IPM in a positive way.
How to perform game concept testing
1. Form a hypothesis.
To build one, start with analyzing the market and check if there are similar games out there, whether they are in demand and what features they have. For the A/B tests to have really meaningful results, the tested concepts should differ distinctively, not just by the color of the button on the third screenshot.
At this stage, make sure you know your audience. One of the mistakes that some game publishers make using the SplitMetrics platform is A/B testing on a very small audience. For example, instead of targeting the United States or several countries in Europe, they run A/B tests on, let’s say, Switzerland. As a result, they get a very low conversion and, moreover, the conclusions based on such are far from being meaningful.
Another mistake related to the audience is choosing the wrong age. We saw some publishers failing their game concept testing for a younger audience, while they could launch the same concept for an older segment of users and that would work. The only recommendation to give here is to study and know your audience.
After you’re done with the hypothesis, you have to decide what your game should look like.
2. Create a pretotype of your game.
That’s right, a pretotype that comes before the prototype.
Let’s say you’re hesitating between the two settings – a cybercity and an alien planet. To understand the best direction cost-efficiently, you could purchase the two packages from an asset store. Then you may want to modify them, put on screenshots and run tests to see which one is more resonating with your audience.
Thus, instead of investing time and money in creating your assets from scratch, you just buy them for a hundred dollars and get numerical user feedback behind each of the environments under A/B testing.
3. Build creatives for an emulated app store page.
Once you’ve got the visuals and copy prepared, you’re all set for showing the app page variations to potential users.
4. Run A/B tests of your game concepts.
There are two ways to do that using SplitMetrics.
Firstly, you can run a one-variation experiment to check the overall performance and conversion rate. This type of test will give you the right direction in a very cost-effective way: you don’t need to waste money buying the traffic to test several variations while you’re not sure whether your overall concept will work or not.
Secondly, you can perform multivariate A/B tests, where one variation stands for one concept, and check which one would perform better.
After you’ve found the top-performing variation, build a strategy to improve it until you’re happy with your results and ready for the launch.
5. Analyze metrics.
To assess the performance of variations under A/B testing, you have to consider their conversion rate, time on page, bounce rate, engagement rate, direct installs against the screenshot view, and the screenshot full-screen view, which basically means an open rate.
At the highest level, look at the IPM, or installs per thousand impressions. When assessing the metric, you may benchmark your own IPM against the game genre standard, but most importantly, compare it against your past performance. The IPM will change either way as you’re making changes to your assets.
How to improve the app store conversion rate?
You can improve the conversion rate by adding the brand name to your game’s title on the App Store or Google Play. However, this might work for big brands in the first place.
Take into account seasonality: adding two snowflakes to your icon or screenshots before the Christmas holidays might make a big difference. Just test if it works for your game.
Another thing that you can test is localization: find out if you need to adjust your app page to a particular storefront, and that may give you a major boost in another region.
Concept testing is not a one-time event, but a continuous process of improving your game. It’s really important to keep running tests at the prototyping, pre-alpha and alpha development stages to make sure that the marketability suits the core loop of your game.