Cracking A/B Testing in Your Product – Hacker Noon


1. De-bottlenecking Choke Points:

When it comes to the in-app user journey, increasing conversions further down the path gives the highest bang for the buck.

The obvious target for checking optimisation potential was the final stage: converting users who open the “Lead Form” to ones who successfully submit the Lead. The “Lead Form” UI, in its current implementation, was 1.5 screen length long, so the users had to scroll to get to the CTA (Call-To-Action). There was a clear scope of turning the “Lead Form” UI into a single page and giving it a persistent CTA (i.e., fixing actionable button at the bottom so that it is hard to miss).

Result: 4 hours of development effort and 7% increase in Install to Lead conversions!

The experiment was run on 3100 users, achieving a p-value of 0.05 (i.e. a less than 5% chance of the observed trend being a fluke). It’s important to gather enough data points to achieve statistical significance and ensure that the trend is definitive and not just random variation.

The key to spotting this experiment was to look for the implementation of best design principles. Clear upfront CTA reduces the friction in moving users to the next stage.

2. In-app coachmark:

For features/action items in app that might be less clear, improving CTA or adding coach-mark can significantly improve click-through-rate (CTR).

The purpose of the entire in-app browsing experience was to entice the user to submit the lead. But, data showed that a disproportionate set of app users were not even clicking on the FAB icon (Floating Action Button) that is supposed to open the “Lead Form”. Since, the purpose of the FAB icon was not super-obvious to the users in its current implementation, educating them about ‘what it does’ by using a coach-mark could significantly increase CTR.

Result: Adding coach-mark took 1 day of development effort and yielded 25% increase in CTR. We further experimented with different FAB icon designs, which increased the CTR by another 4%.

Overall, the experiment increased the core business metric — install to lead submission — by 15%.

Note: FAB icon was intentionally checked for optimization potential as it serves the critical step that leads users from Browsing stage to Lead Form open stage.

3. Optimising Information Density:

Up to a certain level, adding more (meaningful) Call-to-Action (CTA) options on an app screen proportionally increases its Click-through-Rate (CTR).

Home page is seen by 100% of the app opening users. Ensuring that the user smoothly navigates to the next relevant sections in the app (i.e. increasing CTR of the Home Page) can dramatically improve engagement and have a follow-on impact on lead submissions.

In its current implementation, Home Page had a very sparse UI with 8 destination categories shown in 4 vertical screens. All 8 categories could easily be condensed to a single vertical frame, which will eliminate the need for the user to scroll and yet not add any extra cognitive load.

Result: 60% increase in CTR. 40% increase in app open rate. 15% increase in the install to lead submission metric.

read original article here