Firebase Tutorial: iOS A/B Testing

iOS A/B Testing is an effective way to test user experiences in iOS apps. Learn how Firebase makes it easy to run experiments and review results. By Ron Kliffer.

Leave a rating/review
Download materials
Save for later
Share
Update note: Ron Kliffer updated this tutorial iOS 14, Swift 5 and Firebase 7.x. Todd Kerpelman wrote the original.

We’ve all heard stories about how some developer changed the label of a button or the flow of their welcome screens and suddenly found their retention rate or their in-app purchase rate increased by some crazy amount.

Maybe your to-do list contains an “experiment with my purchase button” item still waiting to be checked off because you discovered that running these experiments correctly is a lot of work.

The Firebase Remote Config Tutorial for iOS showed you how to use Remote Config to update certain parts of your app on the fly and deliver customized content to users in particular countries.

This follow-up tutorial will explore how to use Firebase Remote Config to conduct A/B testing by experimenting with different values and viewing the results to find out which values work better.

Prerequisites: This tutorial is designed to follow the first Firebase Remote Config Tutorial for iOS. That said, if you’re already familiar with Firebase Remote Config and have used it before, you can probably jump into this one without having completed the first tutorial.

Getting Started

Download the materials by clicking the Download Materials button at the top or bottom of this tutorial. Even if you have the project from the previous Remote Config tutorial, it will be better to start with the new starter project because it uses the latest Firebase version. The necessary libraries were added as packages for Swift Package Manager and might take a few moments to load when opening the project the first time.

Note: At the time of writing, Firebase supports installation via SwiftPM in Beta. There are some known issues, and you might get an error if you try running the app on your device. If you do, please refer to this workaround to fix it.

Creating a Firebase Project

If you already created a project with Google Analytics for the previous Remote Config tutorial, you can skip this section. If not, follow these instructions to create your project:

  1. Go to https://console.firebase.google.com.
  2. If this is your first project, click Create a project. Otherwise, click Add project.
  3. Name your project Planet Tour and then click Continue:
    Name your project
  4. Add Google Analytics to your project (switched on by default) and click Continue:
    Enable Analytics
  5. Select your region, read and accept the terms of use. Then click Create project:
    Configure Analytics

Next, add an iOS app to the project. Click the iOS button and follow these instructions:

  1. Add the bundle ID of your project — com.raywenderlich.PlanetTour — give it a fun nickname, leave the App Store ID field blank and then click Register app:
    Add an iOS app
  2. Download the GoogleServices-info.plist file and click Next.
  3. Open PlanetTour.xcodeproj in Xcode and drag the downloaded GoogleServices-info.plist file into the Planet Tour project (select Copy Items if Needed).
  4. Head back to your browser to the Firebase console. Complete the next few steps (numbers 3 and 4) by clicking the Next button.
  5. Press Continue to the console on step number 5:
    Continue to the console

Build and run your app. You should see a lovely tour of our solar system. Click a few planets to see details about them:
App main screen

If you’re new to the project, review RCValues.swift to understand how Remote Config is being used in the project. Then, go back to the main screen and check out the banner at the bottom where the app asks you to sign up for the newsletter.

Click “Get our newsletter!”. You’re now going to look at optimizing this screen with some A/B testing.

An Introduction to iOS A/B Testing

The higher-ups at Planet Tour Inc. are concerned there aren’t enough people subscribing to the Planet Tour newsletter. After all, how can you build a successful business if you’re not able to email people about exciting new Planet Tour offers?

Any way we can make this Subscribe button more enticing?

App newsletter screen

The folks from marketing have a theory. You might be able to get more subscribers if you change the label of the Subscribe button to Continue. While you’re at it, maybe you should try changing the text of the banner from Get our newsletter to Get more facts.

These are easy changes to make now that your app gets its data from Remote Config. It would just be a few seconds of work publishing these new values in the Firebase console. Then after a week, you could see if you get more newsletter subscribers. Simple, right?

Well, hang on. How would you know the changes you made are directly responsible for the results you’re seeing? What happens if some influential blogger mentions your app’s newsletter in their latest post? Or you end up running an ad campaign in another app’s newsletter, thereby bringing in an audience more inclined to subscribe to newsletters in the first place?

You might end up drawing the wrong conclusions if you can’t separate the change from the change in newsletter sign-ups.

Wrong conclusions

Ideally, you’d want to release two versions of your app simultaneously. One random group of users gets to see the new newsletter labels, and the other group gets to see the current ones. As long as these two groups are unbiased, you can compare the results between them. The more users in each group, the more confident you can be that the differences are due to the changes you made and not some external factor.

Two groups

Well, that’s exactly what A/B testing is, and it’s a common way to run these kinds of experiments. Many larger companies have built their own infrastructure to run and measure these tests. You can join these larger companies with the handy tools that come with Firebase Remote Config.

Adding Analytics to Your App

One of the main steps in creating an A/B test is telling Firebase the goal of your experiment. Sometimes, it might be a high-level goal, such as increasing retention (how many of your users return after a few days) or user engagement (how much time your users spend with your app each day). But other times, it might be a specific goal, such as increasing the number of users who visit your in-app store. Or, in this case, increasing the number of users who sign up for the Planet Tour newsletter.

Of course, the only way Firebase knows whether your user has signed up for your newsletter is if you tell it. To do that, you’ll use Firebase Analytics. So before you start creating an A/B test in your app, you should add some analytics to your app to give you a few goals you can start working toward.

Note: For more on Firebase Analytics, check out Firebase Analytics: Getting Started.