As indie software developers, sometimes the only type of testing we do is developer testing. And we all know that can be less than optimal ;]
But with larger apps, sometimes it can be useful to get some outside help. I recently did that with my latest app (Battle Map for iPad) by hosting a beta test, so thought I’d share some of the knowledge I learned along the way.
This beta test tutorial will address the questions that might arise as you’re setting up your beta test – or deciding whether you should have one in the first place. We will cover how to find beta testers, get them a copy of your app, respond to the feedback, and much more.
So let’s dig into the world of iOS beta testing!
Do I Need a Beta Test?
If you have a simple app or you’re in a rush to get your app out quickly, the answer is probably no. After all, hosting a beta test takes a lot of time and energy, and delays the time till you’re making money!
But you might want to consider hosting a beta test if:
- Your app has a lot of features. If you have a complicated app with a lot of features, this means there’s a higher chance of bugs! Beta testers can help with this by finding bugs before they’re released.
- You are making a high quality app. If you are making a high quality app, beta testers can help a lot by requesting features that you may not have thought of. By implementing some of these features before launch, your app can be much higher quality.
- You are making a game. If you’re making a game, beta testing can be particularly valuable because it’s hard to balance a game’s difficulty and fun properly without having players other than yourself give it a try!
How Can I Find Beta Testers?
If you just need a few beta testers, one good way is to simply write a post requesting beta testers on forums that app fans frequent. I’ve posted on the MacRumors Forums and subreddits of reddit.com in the past and found some great & helpful testers (thanks RPG reddit!).
If you’re on Twitter, that’s another good source to try! There are a lot of great iPhone developers on there willing to help out a fellow developer with their app, often providing an invaluable developer perspective (thanks @GeekAndDad & @wadevondoom!).
And of course, don’t forget family & friends. For games, this can be especially valuable because you can watch the testers use your app in person, which is often worth 1000 words.
Do I Need a Beta Signup Form?
If you just need a couple beta testers for your app, a beta signup form is probably overkill. But there are several advantages of having a beta signup form:
- It makes it easier to have multiple rounds of testing. Sometimes you might want to go through multiple rounds of testing for your app, which is useful to avoid burnout of beta testers as you make upgrades & improvements. By having a beta signup form, testers can sign up anytime and you can send invites as you’re ready.
- It allows you to ask questions to the beta testers. When people sign up to beta test, you have a chance to ask them some questions that might help you decide who to invite. For example, in my case I wanted to make sure both Game Masters and Players tried out Battle Map before launch to get both perspectives.
- It helps you keep your information organized. Since your beta testers will be in a database, you can keep track of their UDIDs, whether they’ve responded with feedback, and other useful bookkeeping information in a central place.
- It allows people to ask to be notified when your app launches. A lot of times, the people who sign up for your beta test are interested in knowing when your app launches, so give them the chance to sign up to find out! It is a simple way to help spread the word on launch day.
How Do I Make a Beta Signup Form?
Aha, I thought you might ask that! A n00b’s guide to making a beta signup form is coming to this blog soon, so stay tuned! :]
Update: My guide is good if you want to learn how to make a signup form yourself or have your own database on your web site, but if you don’t care about all that and want to just get something working ASAP, Jindrich from the comments section recommends using Google Docs forms, which looks like a quick & easy way to go!
What Do I Do Once I Find Some Beta Testers?
Once you have some volunteers, before you can do anything you need to get them to send you their device’s unique id (UDID) so you can build them a copy for their device.
Here are the instructions I usually send the beta testers on how to send me their UDID:
- Search for the free app called “UDID Tool” on the App store
- Download it onto your [insert device type here]
- Run app, click Send email, email to [insert email address here] (and please put your name in the email so I know who it’s from)
Update: If you don’t want to make your users download an app to get the UDID, there’s a perhaps easier way pointed out by Charles Tidswell in the comments section below. Just check out this tutorial on an alternate method for testers to send you their UDID from iTunes!
How Do I Send the Beta Testers an Ad-Hoc Build?
If you haven’t done an Ad-Hoc build before, you might think that it’s some hugely complicated process (like I once did), but never fear: it’s super easy!
Just take the following steps:
- Log on to the iPhone Provisioning Portal inside the iPhone Dev Center.
- Set up an App ID for your App if you haven’t already and make sure you set up your app to use it as well.
- Go to the Devices tab, and add an entry for each UDID that you receive from your beta testers. Note you’re limited to only 100 per year, so you have to use some discretion in how many people you invite to your beta tests to make sure you don’t run out!
- Go to the Provisioning tab, and create a new profile. Choose your App ID and select all of the devices that you want the app to be able to run on.
- Follow the instructions in the “Building your Application with Xcode for Distribution” section of the Distribution tab to set your project up to use the new provisioning profile + make an Ad-Hoc build (don’t worry, it’s easy).
- Build the project with your new Ad-Hoc configuration, and locate the .app that was built. Next, we need to package it so it’s easily installable by your beta testers:
- Make a folder named “Payload”
- Copy your app into the “Payload” folder
- Zip up the “Payload” folder
- Rename the resulting .zip with an .ipa extension. So if it was “MyApp.zip”, name it “MyApp.ipa”.
- Take your .ipa and the .mobileprovision, and zip up both of those so there’s just a single file needed by the beta testers.
Update: There’s an even easier way to create the .ipa from the .app than the manual instructions shown above. Jim Murff from the comments section below pointed out that @gaminghorror has written an excellent tutorial series on creating a Cocos2D project template that covers now to tweak the build settings to automatically build an IPA file for you!
And that’s it! Now you just need to send the .zip to the beta testers and they can install your app.
How Can the Beta Testers Install the App?
When I send the beta testers the .zip and .mobileprovision (which I usually do by uploading to my web site and sending them a link rather than clogging their email with a large attachment), I also send them the following instructions:
- Download [insert link to zip file here]
- Unzip the file. There should be two files inside: a file ending with “.ipa” and a file ending with “.mobileprovision”.
- Drag-and-drop the .mobileprovision file and the .ipa file onto Applications in your iTunes library on Windows, or the iTunes icon on a Mac
- In iTunes, select your device and open the Applications tab. Sync applications and the new app should be checked.
- Sync your device and try out the app!
Also, here’s a link to a useful guide that you can refer the beta testers to if they get stuck.
How Should I Solicit Feedback?
When you send your beta testers the link to the app zip and the instructions, that’s also a good time to explain what sort of feedback you’re looking for.
For example, you could point out particular areas you’re hoping for them to test or particular things you’re uncertain about/are hoping for feedback on. You could even list particular questions you’d like them to answer.
Here are the questions that I ask for my beta test:
- Things you like
- Things you don’t like, or things that are confusing or frustrating
- Ideas for improvements you have
- Bugs you may have found (with instructions for how to reproduce if possible)
Basically I try to keep the questions to a minimum in order to encourage free-flow thoughts and responses, which I find the most valuable kind of feedback to get.
What Should I Do With the Feedback?
If all goes well, at this point you should have several emails back from your beta testers with a bunch of feedback. Some feedback will be positive encouragment, some will be bugs they found, and MOST of what you’ll probably get is ideas for new features and improvements.
This is valuable stuff, so the first thing you should do is make sure you keep good track of it! I like using Evernote to keep track of all ideas for improvements people suggest for any of my apps (as well as potential blog posts! :])
Next, you should look over the ideas and implement as many as you have time for before launch. Of course, you probably can’t do everything, so you can prioritize by doing the ones that are requested by several different people, would be especially awesome to add, or are easy low-hanging fruit to implement.
It’s not necessary, but is often a nice touch to let the beta testers who came up with the idea know that you actually implemented their idea, they’re sure to appreciate that their ideas came in handy!
How Many Responses Should I Expect?
As far as I can tell, the number of responses you get varies a lot based on the people you invite and your app. But for a point of reference, in my most recent beta test I had:
- 45 signups
- 16 invites sent
- 12 responded with their UDIDs
- 9 responded with feedback
So, as you can see not everyone will respond with feedback. So in my case, for future betas I’ll probably plan on inviting around double the amount of people than the amount of feedback emails I’m hoping to end up with.
All in all, having a beta test is a lot of work, but can help lead to a more polished app if you take up some of the beta tester’s advice and suggestions.
But is it worth it given all of the time and effort? I think that is completely debatable, and probably dependent on the app too. What do you guys think – what do you do for your apps?