Beginning Automated Testing With Xcode Part 1/2

Automating xcodebuild from Jenkins

Let’s update your Jenkins job to build the project. Go to the Jenkins Dashboard\GuildBrowser job\Configure. In the Build section, click on the Add build step dropdown select Execute shell.

Enter the following code into the Command window:

export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer/

xcrun xcodebuild clean build

Your command window should look like this:

Now click Save.
Let’s test the job again by clicking on Build Now. When the job starts executing, pay attention! You might see the following screen, asking you if you want to allow Jenkins access to run codesign so that it can sign your code. Select Always Allow.

If you missed the prompt (because you were chasing your two year-old who was running off with your iPhone, for example), you will see this message in your failed job’s console output:

If this happens, simply try running the job again, but don’t miss the prompt (or let your two year-old run away with your iPhone) this time. :]

While writing this feast post, I seriously just had to use find my phone to locate my iPhone, the little rascal had stuffed it into the trunk of his toy car!!

Assuming you didn’t miss the prompt, then in your output you will see a message similar to this:

/Users/charlie/.jenkins/workspace/GuildBrowser/build/Release-iphoneos/GuildBrowser.app
Unable to validate your application. – (null)

This means the build was successful! You can safely ignore the “unable to validate” message – it’s a known bug that we were able to verify at WWDC this year.

Note: There is a Jenkins plugin for building iOS apps named Xcodeplugin. It works great, but I think it’s best to understand exactly what is going on when building your app. You can achieve the same results as the plugin with some simple build scripts of your own.

What’s more, becoming dependent on the plugin carries some risks. If Apple changes how things work and that breaks the plugin, and it’s no longer updated, you will still be able to adapt if you know how things work.

Unit testing

Unit testing ensures that the output of your code remains unchanged as you are developing it. After establishing expectations of what a class will do, you write unit tests to verify these expectations remain true. Unit testing also allows you to make changes incrementally, seeing results step-by-step, which makes it easier to develop new features and make big changes.

Some other benefits of including unit tests in your projects are:

• Reliability – With a good set of tests, you will most likely have fewer bugs. This is especially true with code that was developed with tests from the beginning, verifying every change and new bit of code along the way.
• Code confidence – It’s a lot less nerve-wracking to go in and refactor a massive amount of code with unit tests in place, knowing that your code should still be producing the same results and passing tests.
• Stability – When bugs are found, new tests can be added ensuring the same bugs don’t harass you again.

Xcode has built-in support for unit tests via the SenTestingKit framework. When writing unit tests, there are two terms to be aware of:

• Test case – The smallest component of a unit test is the test case. It is what verifies the expectations of what your unit of code should produce. For example, a test case might focus on testing a single method, or a small group of methods in your class. In Xcode, all test cases must start with the name test. You will see an example of this below.
• Test Suite – A group of test cases. This is usually a group of test cases against a particular class. In Xcode, there is a template for creating a test suite named Objective-C test case class. It produces a class that is a subclass of SenTestCase.

There are two general approaches to setting up unit tests: you can either test from the bottom up (testing each method/class individually), or the top down (testing functionality of the app as a whole). Both approaches are important – it’s usually good to have a combination of both.

In this tutorial, you are going to test from the bottom up, focusing on a few of the model classes. In iOS 6 by Tutorials, there is another chapter where you’ll try your hand at testing from the top down, from the User Interface to the model code via UI Automation.

Application vs. logic test targets

Next let’s take a look at the two different kinds of unit tests available in Xcode.

Application unit tests

An application unit test basically means any code that needs to use something in UIKit, or really needs to run in the context of a host app. This is the type of unit test target you get by default when creating a new project and including unit tests.

All of the code in this target will be run within your app bundle. You can see the difference by looking in the build settings for each target and checking the settings for Test Host and Bundle Loader:

Logic unit tests

For now, this is the only type of unit test you can run on your units of code from the command line. This code is tested in isolation from the app.

These are unit tests that test your own code, basically anything that doesn’t need to run in the Simulator. It’s great for testing custom logic, network access code, or your data model classes.

Running tests from command line notes – no touching!

Unfortunately, as of Xcode 4.5 we are still not “officially” allowed to run application unit tests via the xcodebuild command. This is very frustrating, because you obviously can run application unit tests from within Xcode. For this reason, you will create a special logic test target in the next section.

Some creative developers have made patches to the testing scripts inside of the Xcode.app contents. For this tutorial, you are going to stick to running your logic tests from Jenkins, and application tests from Xcode. If you’re curious about the patches, take a peek at these files:

/Applications/Xcode.app/Contents/Developer/Tools/RunUnitTests
/Applications/Xcode.app/Contents/Developer/Tools/RunPlatformUnitTests.include

This is the script that runs by default when you run unit tests via ⌘-U:

/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests

Here is the error message you’ll get, and the reason we can’t run application unit tests in the Simulator from the command line (remember that this works fine from within Xcode):

Some clever hacks have gotten application tests to work, but the scripts have changed from Xcode 4.4 to 4.5, so do so at your own risk.