Home iOS & Swift Tutorials

Firebase Tutorial: iOS A/B Testing

iOS A/B Testing is an effective way to test user experiences in iOS apps. Learn how Firebase makes it easy to run experiments and review results.


  • Swift 5, iOS 14, Xcode 12
Update note: Ron Kliffer updated this tutorial iOS 14, Swift 5 and Firebase 7.x. Todd Kerpelman wrote the original.

We’ve all heard stories about how some developer changed the label of a button or the flow of their welcome screens and suddenly found their retention rate or their in-app purchase rate increased by some crazy amount.

Maybe your to-do list contains an “experiment with my purchase button” item still waiting to be checked off because you discovered that running these experiments correctly is a lot of work.

The Firebase Remote Config Tutorial for iOS showed you how to use Remote Config to update certain parts of your app on the fly and deliver customized content to users in particular countries.

This follow-up tutorial will explore how to use Firebase Remote Config to conduct A/B testing by experimenting with different values and viewing the results to find out which values work better.

Prerequisites: This tutorial is designed to follow the first Firebase Remote Config Tutorial for iOS. That said, if you’re already familiar with Firebase Remote Config and have used it before, you can probably jump into this one without having completed the first tutorial.

Getting Started

Download the materials by clicking the Download Materials button at the top or bottom of this tutorial. Even if you have the project from the previous Remote Config tutorial, it will be better to start with the new starter project because it uses the latest Firebase version. The necessary libraries were added as packages for Swift Package Manager and might take a few moments to load when opening the project the first time.

Note: At the time of writing, Firebase supports installation via SwiftPM in Beta. There are some known issues, and you might get an error if you try running the app on your device. If you do, please refer to this workaround to fix it.

Creating a Firebase Project

If you already created a project with Google Analytics for the previous Remote Config tutorial, you can skip this section. If not, follow these instructions to create your project:

  1. Go to https://console.firebase.google.com.
  2. If this is your first project, click Create a project. Otherwise, click Add project.
  3. Name your project Planet Tour and then click Continue:
    Name your project
  4. Add Google Analytics to your project (switched on by default) and click Continue:
    Enable Analytics
  5. Select your region, read and accept the terms of use. Then click Create project:
    Configure Analytics

Next, add an iOS app to the project. Click the iOS button and follow these instructions:

  1. Add the bundle ID of your project — com.raywenderlich.PlanetTour — give it a fun nickname, leave the App Store ID field blank and then click Register app:
    Add an iOS app
  2. Download the GoogleServices-info.plist file and click Next.
  3. Open PlanetTour.xcodeproj in Xcode and drag the downloaded GoogleServices-info.plist file into the Planet Tour project (select Copy Items if Needed).
  4. Head back to your browser to the Firebase console. Complete the next few steps (numbers 3 and 4) by clicking the Next button.
  5. Press Continue to the console on step number 5:
    Continue to the console

Build and run your app. You should see a lovely tour of our solar system. Click a few planets to see details about them:
App main screen

If you’re new to the project, review RCValues.swift to understand how Remote Config is being used in the project. Then, go back to the main screen and check out the banner at the bottom where the app asks you to sign up for the newsletter.

Click “Get our newsletter!”. You’re now going to look at optimizing this screen with some A/B testing.

An Introduction to iOS A/B Testing

The higher-ups at Planet Tour Inc. are concerned there aren’t enough people subscribing to the Planet Tour newsletter. After all, how can you build a successful business if you’re not able to email people about exciting new Planet Tour offers?

App newsletter screen

Any way we can make this Subscribe button more enticing?

The folks from marketing have a theory. You might be able to get more subscribers if you change the label of the Subscribe button to Continue. While you’re at it, maybe you should try changing the text of the banner from Get our newsletter to Get more facts.

These are easy changes to make now that your app gets its data from Remote Config. It would just be a few seconds of work publishing these new values in the Firebase console. Then after a week, you could see if you get more newsletter subscribers. Simple, right?

Well, hang on. How would you know the changes you made are directly responsible for the results you’re seeing? What happens if some influential blogger mentions your app’s newsletter in their latest post? Or you end up running an ad campaign in another app’s newsletter, thereby bringing in an audience more inclined to subscribe to newsletters in the first place?

You might end up drawing the wrong conclusions if you can’t separate the change from the change in newsletter sign-ups.

Wrong conclusions

Ideally, you’d want to release two versions of your app simultaneously. One random group of users gets to see the new newsletter labels, and the other group gets to see the current ones. As long as these two groups are unbiased, you can compare the results between them. The more users in each group, the more confident you can be that the differences are due to the changes you made and not some external factor.

Two groups

Well, that’s exactly what A/B testing is, and it’s a common way to run these kinds of experiments. Many larger companies have built their own infrastructure to run and measure these tests. You can join these larger companies with the handy tools that come with Firebase Remote Config.

Adding Analytics to Your App

One of the main steps in creating an A/B test is telling Firebase the goal of your experiment. Sometimes, it might be a high-level goal, such as increasing retention (how many of your users return after a few days) or user engagement (how much time your users spend with your app each day). But other times, it might be a specific goal, such as increasing the number of users who visit your in-app store. Or, in this case, increasing the number of users who sign up for the Planet Tour newsletter.

Of course, the only way Firebase knows whether your user has signed up for your newsletter is if you tell it. To do that, you’ll use Firebase Analytics. So before you start creating an A/B test in your app, you should add some analytics to your app to give you a few goals you can start working toward.

Note: For more on Firebase Analytics, check out Firebase Analytics: Getting Started.

Adding Events

Firebase Analytics, like most other mobile analytics solutions, uses an event-based model. As users perform actions in your app, Firebase Analytics sends events to its servers. Sometime later, these servers process those events and turn them into meaningful graphs for you to analyze.

Open ContainerViewController.swift. Add the following to the top of the file:

import Firebase

Next, add the following to the end of viewDidLoad():

Analytics.logEvent("mainPageLoaded", parameters: nil)

This will send an event named mainPageLoaded when your user first starts the app and makes it to your main screen. The parameters argument is an optional dictionary of key/value pairs associated with this event. You don’t need any here, so leave this as nil.

Next, open GetNewsletterViewController.swift. Add the following to the top of the file:

import Firebase

Add the following to the end of viewDidLoad():

Analytics.logEvent("newsletterPageLoaded", parameters: nil)

Finally, add the following to the end of submitButtonWasPressed(_:):

Analytics.logEvent("newsletterSubscribed", parameters: nil)

Your app will now trigger different analytics events when users first visit the main page or the newsletter page and when they click the Submit button to sign up for the newsletter.

Enabling Debug View

Before you build and run, you’ll want to turn on Firebase Analytics debug mode, which lets you see the results of all these analytics calls in the console.

To do this, select Product ▸ Scheme ▸ Edit Scheme. In the Run scheme, select Arguments. In Arguments Passed On Launch, click the + symbol and enter the argument -FIRAnalyticsDebugEnabled. Make sure you include the dash at the beginning.

When you’re done, you should have something like the following:

Enabling Debug View in Xcode

Close the dialog, and build and run. Now, you’ll see much more output in the console. There will be text like the following:

[Firebase/Analytics][I-ACS023051] Logging event: origin, name, params: app, mainPageLoaded, {
    firebase_event_origin (_o) = app;
    firebase_screen_class (_sc) = WaitingViewController;
    firebase_screen_id (_si) = 5917351895999520221;

You can click into the console and press Command-F and then enter mainPageLoaded in the search box to find this if you can’t find it manually.

When you tap Get our newsletter! (or Get more facts!), the app will also log this to your console:

[Firebase/Analytics][I-ACS023051] Logging event: origin, name, params: app, newsletterPageLoaded, {
    firebase_event_origin (_o) = app;
    firebase_screen_class (_sc) = ContainerViewController;
    firebase_screen_id (_si) = 5917351895999520222;

It will log a similar event when you tap Subscribe.

You should also be able to see these results trickle in almost-real time by going to the Debug View section of the Firebase console. This will let you know about any events Firebase receives from devices that have debug mode enabled. These include not only the events you added but also other events Firebase measures for you, like screen_view and user_engagement.

Debug View

Debug Mode Considerations

Because you’ve turned on debug mode, Firebase Analytics is aggressive about sending data to its servers. It sends a batch either when it has data that’s more than 10 seconds old or when your app moves into the background. In a production app, this behavior would probably kill your phone’s battery life. Therefore, when you don’t have debug mode turned on, Firebase Analytics sends data either when it has data that’s more than an hour old or when your app goes into the background.

Incidentally, this debug setting does persist. So if you want to turn off Analytics debug mode (because, say, you want to stress-test your app’s battery performance), you either disable the flag and then delete and reinstall the app, or explicitly turn off debug mode by changing the flag to -noFIRAnalyticsDebugEnabled.

Note: At the time of writing this tutorial, it takes about 24 hours for the events that you created to propagate to the A/B testing system. You can continue the tutorial without waiting that long. But keep in mind you might need to pick a different goal than one of these events.
Note: In RCValues.swift, you have minimumFetchInterval = 0 set so Firebase doesn’t throttle updates. However sometimes, particularly with a new project, you’ll still be throttled. If this happens, either wait for an hour or reset the simulator and try again.

Creating Your First A/B Test

Head to the A/B Testing section of the Firebase console. If you’re prompted to select a project, pick the Planet Tour project you created. Click Create experiment:

Create experiment

Select Remote Config to create an experiment with app behavior using remote config server-side configuration parameters:
Experiment service

You’ll see a form to create a new experiment. The first few fields should be pretty easy to fill out. Give your experiment a name like Newsletter sign-ups and whatever description you want, then click Next.

Note: You’ll see a note here that Firebase A/B tests last for a maximum of 90 days. This is worth taking note of, since after this time you would need to start a new A/B test if you wanted to continue collecting data.

Defining Targeting

For Targeting, make sure you select your iOS app from the drop-down list (see below for the completed panel). You don’t have an Android app, but even if you did, it makes sense to experiment on different platforms separately because your app — or your users — might behave quite differently on different platforms.

Because you’re experimenting with text labels, it makes sense to limit your experiment to English-speaking users. Click the and button, select Languages and then select English from the second drop-down list.

Finally, you can decide what percentage of users to put into this experiment. The more users who participate in your experiment, the more confident you’ll be in your results. But if you’re trying something risky that could anger your community or throw your in-app economy into chaos, it might make sense to keep this population small.

Because your changes seem pretty safe, you could probably afford to make this percentage larger. 30 seems like a good percentage to start with.

When you’re all done, the first part of your experiment panel should look like this:
Set targeting

Defining Goals

Click Next to move to the Goals section. This is the thing you’re looking to maximize in your app. In your case, you’re looking to increase the occurrence of the newsletterSubscribed event that you created earlier.

Assuming enough time has passed that the events you created in the previous section have made it into the A/B testing system, you should see newsletterSubscribed listed as one of your goals in the drop-down list. If it’s there, select it. If it’s not there yet, create it:
Add goal

Firebase has already included some other secondary goals to measure along with your main goal. These extra goals are useful in getting the “big picture” view of your experiment. They help make sure that you don’t accidentally hurt your retention because you’re too focused on improving your newsletter subscriptions.

Defining Variants

Click Next to move to the Variants section. Here, you’ll assign different Remote Config values to the different groups of people who are placed into your experiment.

Start by trying different variations of the text that appears in the little banner at the bottom of your main screen. This value is set by the Remote Config variable subscribeBannerButton.

Click Choose or create new. If you used this app in the previous Firebase Remote Config Tutorial for iOS tutorial, you’ll probably see the names of those parameters in a drop-down list. If you don’t have to use those, you can also create new ones! So enter subscribeBannerButton and select the Create parameter option:
Add the first parameter

You now have the option to add different values of subscribeBannerButton for the different variants in your experiment.

For your control group, leave this value as the default value of (empty string). This will tell Remote Config to use whatever value it would if the user weren’t in your experiment. In your case, this will be the default value of Get our newsletter! as set in RCValues.swift. As a general rule, it’s best if you keep your control group values set to (empty string). That’s the best way to compare your changes against whatever’s currently running in your app.

For the next variant, give the subscribeBannerButton a value of Get more facts!.

Your panel should now look like this:
Add another variant

You’ll also want to experiment with the button that appears in your GetNewsletterViewController — this is the parameter called subscribeVCButton. Click Choose or create new and create a new parameter with that name.

Clarifying the Cause of a Change

You could make this change in Variant A like so:
Add the second parameter

But this leaves you with a problem. Suppose you found that variant A did better than your control group. How would you know if the change was due to the Get more facts! button on the front page or your renaming of the Subscribe button to Continue? You really wouldn’t know. In fact, there’s always a chance variant A would have done even better if you hadn’t changed the Subscribe button.

A better option would be to try all the different combinations in several variants. This a technique called multi-variant testing, and you’re going to perform a simple version of it now. Leave subscribeVCButton set to (empty string) in Variant A.

Click Add Variant and, this time, leave subscribeBannerButton set to the default and give subscribeVCButton a value of Continue. Then, click Add Variant once more, and give subscribeBannerButton a value of Get more facts! and subscribeVCButton a value of Continue.

Your experiment panel should now look like this:
All four variants

You have successfully added three variants. Two variants where each of the two variables are changed alone, and one variant where both variables are changed at the same time. You could create other variants here to test other strings too!

You’re done creating your experiment! Click Review.

Testing Your Experiment

You probably want to test your experiment before you push it out to the world. Fortunately, Firebase makes it easy to try out all your experiment variations on a single device. To do this, you’re going to need your Instance ID token, which is a unique identifier assigned to every instance of your app.

To fetch it, go back to Planet Tour in Xcode and open AppDelegate.swift. Then, add the following right after the call to FirebaseApp.configure():

Installations.installations().authToken { result, _ in
  print("Your instance ID token is \(result?.authToken ?? "n/a")")

Build and run your app. You should see a line like the following somewhere in your Xcode console:

The instance ID in your console

Go back to your experiment in the Firebase Console and click the Menu button next to the Start experiment button. Then, click Manage test devices:
Experiment menu

In the dialog that appears, copy and paste the big instance ID token string from your Xcode console. Then, select a variant from the drop-down list — I like trying variant C because you can test both your changes at once:
Add test device

Click Add. Then, click Save.

Quit and restart your app. Because your app explicitly has a Remote Config cache time set to zero (see the previous Firebase Remote Config Tutorial for iOS tutorial if you need a refresher on this), you should see your new values right away:

App screens with tested parameters

Feel free to try your other variations, too. Simply revisit your Manage test devices dialog, select a different variant from the drop-down list and click Save, and you should see that variant on your test device (or simulator) after you restart the app.

Launching an Experiment

When you’ve tested all your variants and you’re happy with how each one looks on your test devices, you can start the experiment for real. Back in the Firebase A/B test console, click Start Experiment and then click Start.

At this point, 30 percent of your English-speaking audience will be randomly selected to use one of your four different variants. A/B testing will measure their progress, note how many of them reach your final goal of subscribing to the newsletter and tell you in the end which variant appeared to best guide your users toward that goal.

Understanding Your Experiment Results

Firebase A/B testing doesn’t just tell you what variant had the highest “user subscribed to newsletter” rate. It also uses some fancy math known as Bayesian statistics to let you know if these differences are due to the changes you made to your app instead of random chance. This is what people usually mean when they use the phrase “statistically significant”.

For this math to work, Firebase needs to try these different variants on a fairly large number of users. And although Planet Tour is a lovely app, it’s still a test app with a total user count of one. This means that for this tutorial, Firebase A/B testing won’t gather enough data to give you any meaningful results, and you’ll probably be left with a screen that looks like this:
Experiment results screen in pending state

Here’s an example of a completed experiment, from a real live app, that produced a clear improvement:
Experiment results screen in completed state

This has a lot of statistics, but here are the important ones:
Experiment header with a summary label

This giant label on top gives you the executive summary of your experiment. In this case, it tells you the variant called Guidebot and group improved the baseline by 8 percent. Note that your variants were boringly named Variant A, Variant B and Variant C, however you can actually rename them to give them nice names if you wish.

Below that, you’ll see a graph comparing each variant’s performance over time:
Experiment graph

Further down, you’ll see how each variant compares to your control group as a general improvement range in each of the goals you’re measuring:
Experiment results details

For example:

  • Under Modeled data, you can see it predicts Guidebot and group would perform between 0.3 percent and 16.6 percent better than the control group in driving retention up. Because this is definitely a change for the better, it’s presented in green with an up arrow.
  • Observed data measures more of the raw data of the experiment. In this case, you can see that Guidebot and group retained 1,145 users, while the control group only retained 1,064 users.

Rolling Out an Experiment

Depending on the results of your experiment, you might want to take one of your variants and push that out to all users of your app. If your experiment has a clear winner, you should see a button that says Roll out variant.

Clicking this button stops the experiment and presents you with a dialog that allows you to publish all the variables from the winning variant into Remote Config as the new default values for everybody. And just like that, your new newsletter options are available to the world!

Where to Go From Here?

Download the completed version of the project by clicking the Download Materials button at the top or bottom of this tutorial. However, please note that you still need to create a project in the Firebase Console and drag in your GoogleServices-info.plist file for the project to work.

There’s plenty more you can do with Firebase Analytics and Remote Config, and you can always read the documentation for more information. Both of these offer many options to manipulate and experiment with your app, and you should check out what they have to offer. Specifically, it’s worthwhile to learn about User Properties and how you can leverage these for more elaborate experiments that could bring even better insights.

In the meantime, think about elements of your app you’ve always wanted to experiment with. Try running one of them through iOS A/B testing and let us know what you come up with. If you have any questions or comments about this Firebase A/B Testing tutorial, please join the forum discussion below!




More like this