17
ECS & Collaborative Experiences
Written by Chris Language
Heads up... You're reading this book for free, with parts of this chapter shown beyond this point as text.
You can unlock the rest of this book, and our entire catalogue of books and videos, with a raywenderlich.com Professional subscription.
In the early days of ARKit, it quickly became apparent that something important was missing: the ability to share augmented reality experiences among multiple users.
Later versions of ARKit addressed the issue by introducing ARWorldMap. The map contains a space-mapping state along with a set of anchors from a world-tracking AR session. This map can be shared, allowing multiple users to experience persistent AR anchors within the same space.
With the assistance of a peer-to-peer network, multiple users can share an ARWorldMap in real time, creating a collaborative experience. Using ARKit, the process is somewhat painful, requiring vast amounts of manual labor from a coding perspective.
Apple created a fantastic ARKit example project that you can explore. Find the project here: https://apple.co/31ltm2u
However, since iOS 13, you’ve been able to pair RealityKit with ARKit to automate most of the manual effort that ARKit-based apps typically require.
In this chapter, you’ll create a modern take on the classic Tic-Tac-Toe game and deliver a RealityKit-based collaborative experience. The new project will borrow from Apple’s ARKit example project, but will mainly focus on the RealityKit side of things.
It’s time to get going!
Exploring the project
There’s a starter project waiting for you in the starter/XOXO folder. The project is a basic Swift-based app that uses a classic style storyboard UI.
Load the project in Xcode so you can take a quick tour of the important components within.
ViewController.swift
Open ViewController.swift. By now, you’re very familiar with the inner workings of the ViewController.
MultipeerController.swift
Open MultipeerController.swift.
Main.storyboard
Open Main.storyboard and flip the orientation to landscape.
Info.plist
Open Info.plist.
Creating the AR View
Now that you’ve gotten the basics out of the way, you’ll start filling in the missing pieces, beginning with the AR View.
Setting the player’s color
Add the following variable to the Properties section:
var playerColor = UIColor.blue
playerColor = UIColor.blue
playerColor = UIColor.red
Sending messages
Add the following helper function to Helper Functions:
func sendMessage(_ message: String) {
DispatchQueue.main.async {
self.message.text = message
}
}
Creating the AR configuration
You start all AR experiences by creating an ARConfiguration and running the AR session. Well, this AR experience is no different.
initARView()
func initARView() {
arView.session.delegate = self
arView.automaticallyConfigureSession = false
let arConfiguration = ARWorldTrackingConfiguration()
arConfiguration.planeDetection = [.horizontal]
arConfiguration.environmentTexturing = .automatic
arView.session.run(arConfiguration)
}
What is ECS?
When using the RealityKit framework to create content for your AR experiences, it’s important to note that the framework runs a CPU-based entity-component system (ECS) to manage physics, animations, audio processing and network synchronization. The framework then relies on Metal for GPU-based multithreaded rendering.
Predefined entities
With RealityKit, you can easily create your own custom entities with custom behaviors based on the various components you add to them.
Creating the game board
Now that you have some background, it’s time to create some of your very own entities, starting with the game board.
var gridModelEntityX:ModelEntity?
var gridModelEntityY:ModelEntity?
var tileModelEntity:ModelEntity?
Creating the model entities
Look carefully at the game board and you can see that the entire board is constructed out of just three distinct shapes: the two grid bars and the square tiles. You’ll create those shapes next.
initModelEntities()
func initModelEntities() {
// 1
gridModelEntityX = ModelEntity(
mesh: .generateBox(size: SIMD3(x: 0.3, y: 0.01, z: 0.01)),
materials: [SimpleMaterial(color: .white, isMetallic: false)]
)
// 2
gridModelEntityY = ModelEntity(
mesh: .generateBox(size: SIMD3(x: 0.01, y: 0.01, z: 0.3)),
materials: [SimpleMaterial(color: .white, isMetallic: false)]
)
// 3
tileModelEntity = ModelEntity(
mesh: .generateBox(size: SIMD3(x: 0.07, y: 0.01, z: 0.07)),
materials: [SimpleMaterial(color: .gray, isMetallic: true)]
)
// 4
tileModelEntity!.generateCollisionShapes(recursive: false)
}
Cloning model entities
Now that you’ve created the three main shapes, you’ll use them to construct the game board. Instead of re-creating each element from scratch, you’ll clone the original entities.
func cloneModelEntity(_ modelEntity: ModelEntity,
position: SIMD3<Float>) -> ModelEntity {
let newModelEntity = modelEntity.clone(recursive: false)
newModelEntity.position = position
return newModelEntity
}
Adding the grid
Now, you’re going to use the helper function above to create the grid. Add the following function to Model Entity Functions:
func addGameBoardAnchor(transform: simd_float4x4) {
// 1
let arAnchor = ARAnchor(name: "XOXO Grid", transform: transform)
let anchorEntity = AnchorEntity(anchor: arAnchor)
// 2
anchorEntity.addChild(cloneModelEntity(gridModelEntityY!,
position: SIMD3(x: 0.05, y: 0, z: 0)))
anchorEntity.addChild(cloneModelEntity(gridModelEntityY!,
position: SIMD3(x: -0.05, y: 0, z: 0)))
anchorEntity.addChild(cloneModelEntity(gridModelEntityX!,
position: SIMD3(x: 0.0, y: 0, z: 0.05)))
anchorEntity.addChild(cloneModelEntity(gridModelEntityX!,
position: SIMD3(x: 0.0, y: 0, z: -0.05)))
}
Adding the tiles
With the grid out of the way, you need to add the tiles to the game board. There are nine slots to fill.
anchorEntity.addChild(cloneModelEntity(tileModelEntity!,
position: SIMD3(x: -0.1, y: 0, z: -0.1)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!,
position: SIMD3(x: 0, y: 0, z: -0.1)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!,
position: SIMD3(x: 0.1, y: 0, z: -0.1)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!,
position: SIMD3(x: -0.1, y: 0, z: 0)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!,
position: SIMD3(x: 0, y: 0, z: 0)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!,
position: SIMD3(x: 0.1, y: 0, z: 0)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!,
position: SIMD3(x: -0.1, y: 0, z: 0.1)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!,
position: SIMD3(x: 0, y: 0, z: 0.1)))
anchorEntity.addChild(cloneModelEntity(tileModelEntity!,
position: SIMD3(x: 0.1, y: 0, z: 0.1)))
Adding the anchor
Now that you’ve now completed the grid and all the tiles, your next step is to add the game board to the AR scene.
// 1
anchorEntity.anchoring = AnchoringComponent(arAnchor)
// 2
arView.scene.addAnchor(anchorEntity)
// 3
arView.session.add(anchor: arAnchor)
Placing content
Now that your game board is ready to place in the scene, you need some user input to know where to place it. All the user needs to do is tap the horizontal surface and the game board should appear in that position. Your next step is to ensure the app recognizes the user’s tap.
Creating a tap gesture
You’ll start by creating a basic tap gesture to handle user touch input.
initGestures()
func initGestures() {
// 1
let tap = UITapGestureRecognizer(
target: self,
action: #selector(handleTap))
// 2
self.arView.addGestureRecognizer(tap)
}
Handling tap gestures
But hang, on there’s an error. You still need to define handleTap().
@objc func handleTap(recognizer: UITapGestureRecognizer?) {
}
Getting the touch location
After the user taps the screen, you’ll cast a ray into the scene to see where on the surface the tap actually occurred. This lets you position the game board just where they want it.
guard let touchLocation =
recognizer?.location(in: self.arView) else { return }
Tapping a surface
Now, to perform the actual ray-cast into the AR scene.
let results = self.arView.raycast(
from: touchLocation,
allowing: .estimatedPlane,
alignment: .horizontal)
if let firstResult = results.first {
self.addGameBoardAnchor(transform: firstResult.worldTransform)
} else {
self.message.text = "[WARNING] No surface detected!"
}
Tapping a tile
OK, now that the game board is visible in the scene, what’s next? Well, when the user touches a tile, that tile should change to the player’s color.
if let hitEntity = self.arView.entity(at: touchLocation) {
let modelEntity = hitEntity as! ModelEntity
modelEntity.model?.materials = [
SimpleMaterial(color: self.playerColor,
isMetallic: true)]
return
}
Collaborative experiences
When multiple people share an augmented reality experience from their own personal viewpoints on separate devices, it’s known as a collaborative experience. To achieve such an experience, all the devices should be connected to one another via a local network or Bluetooth. The devices share an AR world map, which localizes each device within the same space. During an active collaborative session, entities in the augmented space synchronize across all the other devices.
Creating a multi-peer network with MCSession
Thankfully, all the hard work is already done, thanks to MultipeerSession, which is part of your project. It acts as a basic wrapper class for MCSession, which is the network session class that connects multiple peers.
Adding multi-peer connectivity
Now that the network is ready, you’ll create the multi-peer session.
var multipeerSession: MultipeerSession?
var peerSessionIDs = [MCPeerID: String]()
var sessionIDObservation: NSKeyValueObservation?
initMultipeerSession()
func initMultipeerSession()
{
multipeerSession = MultipeerSession(
receivedDataHandler: receivedData,
peerJoinedHandler: peerJoined,
peerLeftHandler: peerLeft,
peerDiscoveredHandler: peerDiscovered)
}
func receivedData(_ data: Data, from peer: MCPeerID) {
}
func peerDiscovered(_ peer: MCPeerID) -> Bool {
}
func peerJoined(_ peer: MCPeerID) {
}
func peerLeft(_ peer: MCPeerID) {
}
Handling session ID changes
When a peer connects or when your session ID changes, you need to inform the connected peers of your current peer ID.
private func sendARSessionIDTo(peers: [MCPeerID]) {
guard let multipeerSession = multipeerSession else { return }
let idString = arView.session.identifier.uuidString
let command = "SessionID:" + idString
if let commandData = command.data(using: .utf8) {
multipeerSession.sendToPeers(commandData,
reliably: true,
peers: peers)
}
}
sessionIDObservation = observe(\.arView.session.identifier,
options: [.new]) { object, change in
print("Current SessionID: \(change.newValue!)")
guard let multipeerSession = self.multipeerSession else
{ return }
self.sendARSessionIDTo(peers: multipeerSession.connectedPeers)
}
Handling the “peer” discovered event
When the network session discovers a new peer, it triggers peerDiscovered(_:), asking it for permission to allow the new peer to connect.
guard let multipeerSession = multipeerSession else
{ return false }
sendMessage("Peer discovered!")
if multipeerSession.connectedPeers.count > 2 {
sendMessage("[WARNING] Max connections reached!")
return false
} else {
return true
}
Handling the “peer joined” event
When the peer is allowed to connect, the network session will trigger peerJoined(_:).
sendMessage("Hold phones together...")
sendARSessionIDTo(peers: [peer])
Handling the “peer left” event
When a peer leaves, you need to update peerSessionIDs. To do this, add the following to peerLeft(_:):
sendMessage("Peer left!")
peerSessionIDs.removeValue(forKey: peer)
Configuring RealityKit for collaboration
Well, that’s all you need to do to create a multi-peer network, but you’re not quite done yet. You still need to configure RealityKit for collaboration.
Enabling collaboration
To use collaboration, you need to enable it when you create the AR configuration. Do this by adding the following line of code to initARView(), just before running the AR session:
arConfiguration.isCollaborationEnabled = true
Setting the synchronization service
When you use RealityKit, you have to synchronize all of its entities and their components with all the connected peers.
extension MultipeerSession {
public var multipeerConnectivityService:
MultipeerConnectivityService? {
return try? MultipeerConnectivityService(
session: self.session)
}
}
// 1
guard let multipeerConnectivityService =
multipeerSession!.multipeerConnectivityService else {
fatalError("[FATAL ERROR] Unable to create Sync Service!")
}
// 2
arView.scene.synchronizationService = multipeerConnectivityService
self.message.text = "Waiting for peers..."
Handling a successful connection
Now that everything’s in place, once a new peer successfully joins, RealityKit will create an ARParticipationAnchor for that peer.
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
for anchor in anchors {
if let participantAnchor = anchor as? ARParticipantAnchor {
self.message.text = "Peer connected!"
let anchorEntity = AnchorEntity(anchor: participantAnchor)
arView.scene.addAnchor(anchorEntity)
}
}
}
Requesting network permissions
Oh, you’re not quite done yet. There’s one last thing that you have to do and that’s to request network permissions.
Managing ownership
During a collaborative experience, when you create an entity, you become the owner of that entity. Should another peer attempt to modify an entity that belongs to you, they’ll be blocked.
Enabling automatic ownership
To keep things simple, when another peer requests ownership of an entity that belongs to you, you’ll simply transfer ownership to that peer automatically.
anchorEntity.synchronization?.ownershipTransferMode = .autoAccept
Requesting ownership
Now, you need to make sure you request ownership when you tap on a tile.
if let hitEntity = self.arView.entity(at: touchLocation) {
if hitEntity.isOwner {
let modelEntity = hitEntity as! ModelEntity
modelEntity.model?.materials = [
SimpleMaterial(color: self.playerColor,
isMetallic: true)]
} else {
hitEntity.requestOwnership { result in
if result == .granted {
let modelEntity = hitEntity as! ModelEntity
modelEntity.model?.materials = [
SimpleMaterial(color: self.playerColor,
isMetallic: true)]
}
}
}
return
}
Removing anchors
As a final touch, when you’ve played a few games with a friend, it would be nice to clear out the playing field so that you can play some more.
func removeAnchors() {
guard let frame = arView.session.currentFrame else { return }
for anchor in frame.anchors {
arView.session.remove(anchor: anchor)
}
sendMessage("All anchors removed!")
}
removeAnchors()
Key points
Congratulations, you’ve reached the end of this chapter and section. You can find a copy of the project in its final state under final/XOXO.
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
Have feedback to share about the online reading experience? If you have feedback about the UI, UX, highlighting, or other features of our online readers, you can send them to the design team with the form below:
© 2021 Razeware LLC
