With every new release of iOS, Apple seems to introduce at least one new library or technology that generates extra excitement among developers. A new technology like GPS or a gyroscope can make new types of apps possible. And to be honest, sometimes it’s because the new tech just looks like a whole lot of fun!
ARKit is one of those technologies that’s fun, but also opens up many new avenues for iOS app development. Since ARKit was announced, the buzz about augmented reality has only continued to build.
Maybe you’ve heard of augmented reality, but aren’t sure what it means. Or maybe you’re interested in augmented reality but not sure how ARKit can help you as a developer. In this article, we’ll examine what ARKit brings to the table and what you can do with it.
What is Augmented Reality?
Augmented reality, at its core, is the process of combining virtual objects with the real world. It’s not the same as virtual reality, which completely replaces the real world with a simulated one. Where virtual reality blocks out the real world, augmented reality seeks to enhance reality.
An augmented reality app superimposes 2D or 3D objects on the live camera view of the device. The result is an illusion that places these virtual objects in the real world, and gives the user extra information about the world that is not normally present, or invisible to the unaided eye.
Augmented Reality Before ARKit for Apple iOS 11
Augmented reality is a not a new technology; in fact, it’s existed in many forms before Apple came along and provided an SDK for iOS app development.
As an example, the heads up display (HUD) in some aircraft aids pilots by displaying important flight information on the cockpit window. That way, the pilot can see the information without looking away from the view. Similar technologies exist in vehicles to show speed and other pieces of data on the windshield.
Scenic look-offs usually have signs that identify key points of interest visible from the overlook. These signs provide extra context and information or metadata about the view, such as the height of peaks in the distance.
Even your Monday night football games uses a form of augmented reality: the current status, score, and time remaining in the game display on top of the video feed, along with stats about the star players in the game. Announcers will often use virtual pens to draw on top of replays when describing what worked, or went wrong, during a play.
The Current State of AR Apps
ARKit is an exciting new library for iOS app development, but augmented reality apps have been on Apple devices for years. Early iOS devices lacked some of the sensors needed to make augmented reality work well, such as a compass and gyroscope. As these sensors arrived, augmented reality apps began to appear on the App Store.
You would have had to try quite hard to avoid hearing about Pokémon Go during the summer of 2016, as it seemed to be the game that everyone was playing. Pokémon Go combines Nintendo’s second most popular video game franchise with an augmented reality interface. For many, it served as an introduction to augmented reality gaming on their phone.
PeakFinder Earth shows the names of nearby mountains as you view them through your phone. It also overlays a panoramic drawing — along with the paths of the sun and moon — on top of the view. It’s basically an augmented reality version of the scenic overlook sign I mentioned earlier.
The first augmented reality app I remember using on the iPhone was the stargazing app Sky View. Identifying what’s in the sky above from a paper map while standing outside at night can be difficult. When looking through your phone and seeing the night sky labeled, identifying becomes a simple process.
I use Gymaholic regularly to track my weight training routines. When adapting a new exercise, learning the proper form is important to reduce chances of injury and get the most from the workout. While many apps will show an exercise, Gymaholic can work with your Apple Watch or a sheet of paper printed with a target to display the exercise in question in augmented reality. Unlike a standard video or diagram, I can move and shift around the 3D-animated exerciser and view the exercise from any angle to ensure I understand the proper technique.
Getting a tattoo is a personal and somewhat permanent decision. Before committing to a tattoo, why not use your phone to see how it will look? Ink Hunter lets you do that. The app will allow you to search for tattoo designs and then see what it would look like on your skin.
I enjoy photography, and I especially enjoy taking photos of landscapes. The time of day, season of the year and even your location on the planet often makes the difference between a nice photo and a great one. PhotoPills provides several tools to help you plan your photo shoot: it will overlay the path and location of the Sun, Moon, and Milky Way at any time, past or future, based on your location. Seeing how the sun will move through the sky for a given season and location makes planning the perfect photo easier.
Augmented Reality in iOS Before ARKit
To no one’s surprise, building augmented reality libraries is a complex problem. I once wrote an iPhone app to overlay a text description of nearby points of interest onto the view from the device’s camera. This simple augmented reality app required finding the current location of the device. It then needed to determine where the device is pointing in space. From the location, the app could determine nearby points of interest. From the orientation it could determine which points are actually in view. The app then had to combine these elements to display text at the correct point on the display.
It took a lot of math to combine all these items together to a the result for the screen. In the case of that app, most of the calculations were 3D calculations that involved matrix manipulations and linear algebra. More complex augmented reality apps may also need computer vision algorithms to detect surfaces, map points on the display back to virtual objects, and calculate lighting effects.
So make no mistake, although augmented reality was possible before ARKit, it wasn’t always easy.
Other Augmented Reality Libraries
ARKit for Apple iOS 11 isn’t the first library created for augmented reality apps. Once iOS devices became capable of augmented reality, open source libraries weren’t far behind. The iphonearkit project on Github dates to 2009 when the iPhone 3GS was state-of-the-art. Our Augmented Reality iOS tutorial focuses on location-based AR and uses the HDAugmentedReality project.
The popular OpenCV project handles many computer vision aspects helpful in augmented reality. A tutorial on this site shows you how to make an augmented reality target shooter. It demonstrates this library’s capabilities.
Authors of open-source projects generally develop libraries for the author’s specific need. Most libraries support a specific usage and type of augmented reality application, and are seldom updated to take advantage of new iOS abilities and devices. They hide the complex calculations from the developer, but also make assumptions about how apps built on top of those libraries will work. If you find one that fits your need, it’s great. But for many apps, the right library doesn’t exist.
There are commercial libraries that help create augmented reality apps. These commercial libraries offer better support, but can have costs that are prohibitive for a single developer or small team. They’re often general purpose libraries written in C, C++, or other languages. This crossing of languages adds complexity, increases the learning curve for a library, and can cause integration headaches for developers not familiar with bridging libraries in other languages.
What ARKit Does
What makes ARKit special is that it reflects Apple at its best — it just works!
With ARKit, the creation of augmented reality apps no longer requires a custom engine or finding the perfect library. Apple built the engine to work on all existing devices with at least an A9 processor that run iOS 11. That covers the iPhone 6s or later, including the iPhone SE, all iPad Pros, and the 2017 iPad.
ARKit uses a process called visual-inertial odometry to build the correspondence between the real and virtual in your app. It combines the motion sensors in the device with analysis of the scene gathered through the device’s camera and produces a high-precision model of the device’s position and motion within the world for you. As a result, it recognizes notable features in the scene image, tracks differences in the positions of those features, and compares that information with motion sensing data during movement.
So ARKit doesn’t only determine that you’re pointing your phone to the east. It also analyzes and tries to understand the scene to the east. ARKit can find real-world surfaces that correspond to points in the camera image. It can detect flat surfaces (though not vertical ones) and provide information on the position and size of these surfaces. Your app can place virtual objects on and interact with these points and surfaces.
The tight integration with iOS means that you, as a developer, can leverage skills you’ve already gained in your years of iOS app development. You can build virtual objects using SceneKit and SpriteKit, or you can use Metal to develop a virtual scene with near complete control of the visual elements.
What can you do with ARKit?
Now that you understand a bit more about augmented reality you may be wondering, “What is this actually useful for?”
ARKit saw general release to the public in late September as part of the release of Apple’s iOS 11. In the months between, developers have already explored the technology and produced demos for inspiration. While ARKit is mere weeks old, developers have been considering how to use it for months. There seems to be a new ARKit demo on Twitter every day, and I’ve tried to highlight some of the best ones below.
Objects in World
The first demo I saw for ARKit used augmented reality to measure a physical space viewed through the phone. The ability to translate distances on a screen to distances in the real world provides a foundation for apps that need to place objects accurately onto a view.
Developers have expanded on this with apps that allow a user to determine a room’s dimensions and layout by moving a phone around the room. Other demos measure the distance between objects or along paths by drawing lines on a screen with a finger.
Ikea demonstrated an app allowing customers to preview furniture in a room before heading to the store. It’s a lot better to learn that a couch won’t fit in your room before you’ve had it delivered. Design and home decor companies likely will add similar abilities to their apps. This will make online shopping for these items feel a little less risky.
Games and Fun
Games in particular seem a natural fit for this functionality. It’s not a coincidence that both WWDC and the Apple iPhone launch events demonstrated augmented reality games. I expect game developers will embrace ARKit, as games are a good fit for new visual technologies. Even just having a glimpse at the demos during the Apple event, you can tell there’s something enticing about playing a game in augmented reality. The ability to move around and interact with a game as though it’s truly happening in the real world opens up a whole new category of gaming.
I suspect early games will likely resemble Pokémon Go, or the games demoed at Apple events like The Machines. In time, games that push the limits of augmented reality should arrive, and it’s likely that the most captivating games haven’t even been made public yet.
I’m most intrigued by the idea of games that bring people into a shared virtual environment. Instead of looking at a representation of a board on the screen, an augmented reality games can show the board on the desk in front of you. You could see avatars of other players around the board with you, much as if you were sitting in a room together. This would take the mildly-connected experience of multiplayer and make it much more immersive. The concept could also would work for card games, role playing games, and other shared game experiences.
Augmented reality has a lot of potential to revolutionize education at many levels. Apps that take an abstract or large scale concept and place it in front of someone’s eyes can be a powerful tool to help students understand new topics and concepts. As an example, ARSolarPlay shrinks the solar system to fit in the room with you. It’s one thing for a student to hear about the solar system. It’s another for a student to see it before them in the classroom and move around between the planets.
Beyond the classroom, historical locations and places can now augment their experience with AR apps. Instead of giving visitors a simple pamphlet at a historic site, each visitor’s phone could become their key to understanding the past. Points of interest could now provide relevant visual and auditory information when you visit a particular important location.
Today, you can merely read about a battle that took place in a field in front of you. With an augmented reality app, visitors could listen to narration while watching a virtual re-enactment of the battle. Or maybe you could watch a virtual recreation of a volcano eruption as you stand on the edge of the crater!
Mapping and Location
Current navigational apps simply tell you to turn left at the next intersection. An augmented reality app could draw the path for you, showing the turn around the corner. A hiking application can show you the fork to take for the desired trail, or lead you back to the trail when lost. For an example, check out the ARKit-CoreLocation library, which demonstrates combining augmented reality with the GPS data.
A businesses could help customers better navigate their stores with an app. Hospitals and universities in particular often have decades worth of remodels and additions that make finding certain rooms or floors rather difficult. Imagine an app that would lay out the path to a doctor’s office or waiting room or area right on your phone screen.
If you’ve ever become separated from friends at a concert, festival, or other event spread over a large and crowded space, imagine how useful a version of Find My Friends could be that shows you where your group is, and what path to take to get back to them.
Some Weakness of ARKit for iOS 11
As excited as I am about the potential of augmented reality on iOS, there are still a number of hurdles to realizing some of the full benefits of augmented reality in every area of your life.
The biggest hurdle I’ve run into with ARKit is its inability to detect vertical planes. ARKit can place your virtual object onto a table, but not hang it on the wall. It also doesn’t handle complex surfaces very well, such as such as curved walls or other non-flat surfaces.
The computer vision analysis that provides much of the “magic” of ARKit needs quite clear images to do its job. A dim room with few details may not work well in AR, so that haunted house app idea you had for Hallowe’en might just have to wait for later updates to ARKit.
I’ve also found that surfaces without texture or much contrast take longer to track, and once recognized, don’t track as well as textured surfaces with high contrast. This includes bare white walls, along with plain or simply-textured floors. These surfaces need users to move the camera around more to give the library enough data to process. The library though, does provide feedback in these cases that you can return to the user.
There are also concerns about how augmented reality apps will affect the battery life. An augmented reality app will be using the camera, displaying complex graphics on the screen, and performing complex calculations — all simultaneously, which can be a drain on the battery. Augmented reality should take a lesson learned from early GPS apps: a reputation for hurting battery lie can hurt adoption of a fantastic technology.
Since many users skirmish with their battery life daily (raises hand), power consumption is a barrier to broad adoption. Fortunately, power banks and portable chargers are cheap.
Where to go from here?
Augmented reality isn’t a new technology, but ARKit simplifies it enough that you can look forward to an exciting new wave of apps in the months to come.
Virtual reality has been “the next big thing” in technology for decades, but hasn’t fully arrived for a variety of reasons. Augmented reality feels… different. Naturally, not all apps will be winners. But there will also be amazing ones. Life-changing ones, even. The arrival of real apps using ARKit to solve real-world problems (let’s face it — it’s all about real-world entertainment, ) is just beginning.
If you’re excited to get started, Apple’s documentation provides a good reference and overview of the library. And as usual, we have some great tutorials and products to get you started:
- The book iOS 11 by Tutorials , available on our online store, contains a chapter introducing ARKit and shows you how to create an interior design app in augmented reality.
- 2D Games by Tutorials, also available on our store, has a ton of great projects in SceneKit and includes a bonus chapter on creating a game in ARKit.
- We also have several screencasts screencasts on ARKit for our video subscribers.
I’m excited to see what new places you take augmented reality in the years to come! Have a favorite AR app, or an idea you want to share? Come join the discussion below!