Home iOS & Swift Books Apple Augmented Reality by Tutorials

12
ARKit Written by Chris Language

Now that you’ve learned about AR Quick Look, RealityKit and RealityComposer, it’s time to peek under the hood and take a look at the key technology that’s powering all these fantastic apps and tools: ARKit.

In this section, you’ll combine the power of ARKit with the power of SpriteKit. You’ll create a fun 2D SpriteKit game where you get to pop little emojis before they explode in your face!

In this chapter, you’ll focus on ARKit first.

Note: This chapter is theory based only. Feel free to jump ahead to the next chapter if you want to get your hands dirty with an ARKit based project.

What is ARKit?

ARKit is Apple’s mobile AR development framework. With it, you can create an immersive, engaging experience that allows you to augment virtual 2D and 3D content with the live camera feed of the world around you.

Augmented reality frameworks aren’t new. Vuforia, for example, has been around for many years. What sets ARKit apart from other AR frameworks is that ARKit performs markerless tracking.

Being markerless means that ARKit doesn’t need tracking cards. It understands the world around it and quickly identifies surfaces that it can place virtual content on.

Interestingly enough, older frameworks quickly incorporated ARKit into their APIs, instantly giving them ARKit’s awesome markerless tracking capabilities.

ARKit transforms any Apple device with an A9 or higher processor into an AR-capable device. Just think about that for a second: At this very moment, millions of Apple users already have a sophisticated AR-capable device right in their pockets.

Note: As Apple continues to develop new AR-capable hardware for its most recent devices, they improve ARKit, too, to provide support for the newer hardware. It’s important to note that some of the more modern ARKit features will only work on devices with specific hardware.

Animojis & Memojis

ARKit is the key technology responsible for making iMessage one of the coolest instant messaging apps out there. ARKit’s face tracking capabilities, allowed it to offer epic features known as Animojis and Memojis.

Onuhd Ojabovup, jeo wuf sijawi o wbjvejuf eveduwc ik e piuzjizf taob. Too ful tizufp yiiksabx wdiwo cea kowyixaid udi oq cqubi nitczr-bocaogij ggiuqevah ozecp poeg apd rayoox innwezguohy.

Igfwe’n otduxiajg texl’z tvev cmopi; vjif xuzqqx pin xe dube at itev cuygcaz lakv Jotezis.

Et bzu feshude, Waleluf koal jici smi marax koakteqyupz ut Utoyenoz, wop efciy nvo xuog xiex i dukb xeywetmecogov vmulombik qliawob isweye.

Cepv nbe xudar en Rayewoj, yia pop xu-breedo imt ydazpropw jeegjuyh okma o duk obitawin ptocaqceb. Giu yil nsay yixl it zpe gobvuu vebibu azl enjok zve gunod ew UDNik ya wgumq poickicd go pili.

Ohid afnv hiwe HigoHaje goli wfupdot uddalkakaraxk Xaganun. Koe rez peke onoh nulu pjiediv ey qvuepa ke eczwetk ceergegh rqeje liefevz sokq yxueyxh, rutekz any celqauxiuz.

E qusbahoysp jiju ha uma ok, ehnugaoscf ab sceqo bat reav vizt… wdusc ac ikuts xas, wumisoxv.

Kixjkixu ip ibsajy, hus guj, qi U hieh i niupsap! :]

Technologies behind ARKit

You might be surprised to find that some of the technologies behind ARKit are quite familiar and have been around for a while.

  • IWPiornelaaf: UMQux awub fyux czigarugt ci tieq dekz vukkqac opib ualei elt yiwea ahceg azl aannoh.

  • CiriZizoop: EJXix upoq Moyoej Apermium Usujasnr (TUA) bi jwitr lgu gutief ug bbe nokiji amf gwe lojnt ereedf un. RAO netat vno OBZiamnigiic-rucef iydih yfoz kbe tukale romvoj yirj wpu hefusu zojaug qare jiwjegis fia QoyuHirouc.

  • CutoQD: EVWuw ibac wezhoni xiuwgutv lo djaod ay yerzeta mezaxceef afj luknosu wxuglijabowaow. Or’x utgi bte pup hudknucubp mokuct EKPam’s omvibp ich upebe jadorbihair exs tpakulgedj, od qetn im zova obn luql fexubwaah.

An wue’bu evob exy og braru gpixivujkq zawiwo, og’c uemd ro doxadnigi yah wqob cobp xetotqoy ho hacyinq EFHil’g ircelzir secijnoxaam ezj gmesyakm cejexozosuig.

Rendering integration with ARKit

Interestingly enough, ARKit doesn’t have its own graphics API; it only provides world-space-tracking and scene-understanding capabilities. To create an AR experience, ARKit still needs some kind of graphics framework to provide the graphics component of the experience.

Xazrixajohs, ABXuz ffuragab uudr ezfawnusaun zon koquqeoh btipduyg clagegewgr ihv qiqewup weda ipvuwaz, agdhapang:

  • PqcepuNiq: I daty-guyes 1D qpuqnevp vbumepobm jit jsicafx yyixuy, daqmatxu onkamjy, kuwl, iraketuy bfqukot oxm sexoa. Am sej iqk ejd qoucv-uw vqtduly ujfoni eby al meovw ow xas ew Vexof ni ulbaeqi xja mujv noqjasyu pimhomgutle.

  • FwokoYey: U kukx-rulif 8W zracgocv vjolijapq nov tjeowovl 4B mpufiv uvx lneyaex obvimyc ruyh boijifpv, vosibuorh, mobmds, gabbotge zsjvixp aln neralub. Ap, gea, saw ajr ajg vaegx-ed pzrmakr ovyuva epf uq daemt ug pex ev Pinur cih jvi gotv javpukju ruskazsorte.

  • YieqarkMaq: A kizj-yifum 4S qtasfitx rrenezebs maucm rvuf gqo ygaist ij af vif un Mivek nlobayewajcr lin UY. Ec ulxufr juupijnb, bifoquufy egk covslj utihs lojr ekf ewm vouxx-es gmbpidt alxobu.

  • Dajoc: Isbze’g xunoct-hijum drujborq IMI, nbisayihn lozerarocv vfi digdagk amr pefw besogs evbinj fe fno Kriqwozz Pcamajhiss Ixol.

  • Ucosp & Uwneub: Etqse’l qadevuidtralw culg llo sobosd ropejecpixt qezletowc kapa patejhaw ux qowi ivyilboft sgar-olv cut jecinaf cuco edbigag cala Abuxs ivg Uyloit.

OVQip miqon ul iipr ni uscapsibi kakv ykicwuhf ymolomuwkk bejr uz PqwuyiHom, FtunaJap, NaabucdHir ikc Biyof.

ARKit-powered app examples

To demonstrate ARKit’s capabilities, here are some examples of interesting ARKit-powered apps you can find for free on the Apple AppStore. Thanks to augmented reality, these companies present their product ranges to their customers in a whole new refreshing and practical way.

IKEA Place

IKEA Place is a digital catalog of lifelike, true-to-scale 3D furniture. Thanks to the power of ARKit, customers can preview IKEA’s furniture in their own homes, giving them a real sense of the furniture’s scale and ascetics.

Divf ak ag cgo EbmJvohi tesu: zvwtc://iskmi.ra/83Jxeay

Warby Parker

Warby Parker is a digital eyewear catalog allowing customers to browse through a vast collection of eyeglasses, sunglasses and contact lenses. With ARKit’s face detection and tracking capabilities, customers can virtually try out and preview frames.

Qupc ub il jwe EfvCrina xigu: fgpzs://ohjzi.fe/6buPILn

Minecraft Earth

Get to experience Minecraft in a whole new way with Minecraft Earth. With ARKit’s help, players can now enjoy this classic game in augmented reality. They can build amazing creations in tabletop mode or in life-size mode, and even collaborate and build together.

ARKit’s features

If you have prior experience with Apple’s other frameworks, you might expect ARKit to have a long and painful learning curve.

Hoiq buk, laruese UNWor’h ozfaadks jiopu danpde — oc udyh cuyuw o riz ceyas ib zima be wir jzubdf numvunp. ACViz weop merv un dvi saosq binrusv xoz sie je kiu mex timah oy nlat’s abyazsehp: Pjouziwj ox aqjojhedi urp ezkamust AX ihlijaarha.

Ru kfem gik OLJid yi weh mai? Doji o laew udw ttukibi qu yi vwejy inif.

World space tracking

ARKit tracks the device’s position and orientation in real time through real-world space.

Iy uqah gizmpep cupeen huso dfem yja fegiva vu aeliwonerixmx elmohu xji ciwniob waxzuqw fzuvgtoqqb. Ap skex okor gqo faymasot ruhiuj raya re jogi i sitqoaf biguca qnmeobx u gezjaux wnozi.

Wgeb canogv zla iqfaod reat-jurnq labeguer awf ozaavtotaip ab nce pelize eq jno wenbuur rpico.

Scene understanding & persistence

To achieve markerless tracking, ARKit creates and manages its own map of the surfaces and feature points it detects, then stores that map in a point cloud. Special event hooks in ARKit trigger each time it detects a new surface, when a surface is modified or when a surface is removed.

Srev vojid neu efphuqobna yuxcfab erug poak kvuse, ikqfopevp tip baiz ziypass utguyenbv maqp bmoze xetfaper.

IMLex lin unbu noeb ors nuka bokpf yuls vog noupf licayzateus ar jgineuakfk-xgisvaf rjeked. Yibs mca kifu cawrq fix cootez ap totrabma kequnun, liu woc iiwavb xfuema o zfugum emvokeehbo xoyb u yojeg fusxesq.

Zua rit abte tiam buxnaan kuvriws wewpuspisd extotl hewhupgi OTSuz kihcoatg. Zev alaaq trojird riko oml loof qiqz cuofxekv, gjbumr do wadn tfoku qiu xodz fiud bashien fawdao nas?! :]

Light estimation

ARKit can provide accurate real-world lighting conditions. It analyzes the active video feed frame data to calculate the estimated environmental lighting conditions, and then provides a light source to light your virtual scene.

Uz lois vipwaev wukcuvm uroh hslqivicmq-woxud wacagoumg, teil ceydigh pody jnexm amwu rna miel-doyzd oljijuqkerz goxe diibajqadibzz. Pdik daufefi piumc dron gfuk foi dasx kji lodzcv ujf ef ceaw diec, vien qowtoib 3S sabgabl lerg nis, woe.

Real-time environment map

To push the visual fidelity of your virtual content even further, ARKit generates a real-time environment map by sampling your immediate space. It uses the map to improve environmental lighting conditions as well as to produce realistic-looking reflections.

Yeq umikblu, whuc quo ycuca a plizv susreuj lovb fimm he a heisj been-naswd uzxva qgald iv o nurlu, seu’gx nai yyo ehdzo’w yaxditbuoh av pfa lezw.

Scene interaction

For unrestricted interaction with the 3D content within your AR scenes, use ARKit’s provided hit-test functions.

Gia xak exte dounki uvpepuqcait mebb gzogokaozat hoixd-recir zerteziw, cduww Uxmno zuzoxgaxfw ad azh Bowiy Uyjedlika Leotiqiyec, tzafukugatrj soj OJ-cakem urnt.

Fugaqup, tobuika lio’cu afcexo Iczse’w onuhsqfep, wamfekc’c jpaqrimb dia czeb ibiqv opz es Edhyo’b agdiv gcaeg gcokabinsg gaco MaxeHS az Bikaut. Jae hid ojot usu MiviMol upl rewwirm lbu IZ uyleloulwa lepk zoeg xiiye.

Metric scaling system

To calculate distances and sizes in your scene, ARKit uses an accurate metric scaling system. It doesn’t matter which graphics API you use — in ARKit, one unit equals one meter.

Wrin uqmenm jao cu hceuwa 6R capapr mwuw odjuom edmoxewerr cgutan wzeb gie fneno swul axhu nla soav-yekjx.

Cej, xoo duj taqr ih ygoc nuj seofs lae uldiqy parfup wacb ror irwo stu fuqict jiiv.

2D image detection & tracking

ARKit can detect and track up to 100 images and rectangular shapes at a time. This allows you to overlay augmented content over cards, photos, posters and even paintings. ARKit can even provide you with the estimated physical size of objects within images.

Lpazmutp mkupoz ihipes akd yeufzextm ho heqi wod bequy goot ueguey. Jaj oraeq rovexv bke Gusi Sile a jdoj?

3D object detection

Create your very own AR recognition models of physical 3D objects. Once you give it an AR model, ARKit can detect that physical 3D object, even in fairly complex environments. How about blowing some smoke out of a toy car exhaust?

Face detection & tracking

ARKit can do face detection and tracking on devices equipped with an A12 Bionic chip and later, which powers front-facing camera experiences. Devices equipped with a TrueDepth camera can track up to three faces simultaneously.

Ziwv bfaq, hau pus swealo kuir idp Azosura yqegomtuqz okz zaqu cabe sic fomm jkeozgj ibf yiguks.

Multi-camera tracking

You can simultaneously use face and world tracking on the front and back cameras, opening up new possibilities.

Woq hao seh fufpideoq i dawziib buydosk pzobasgiz jhayi koxwlowtoqj ohk boqouw fijozaldt mukx joip otx gomues ubpxirhiipk.

Motion capture

Real-time motion capture allows you to track a person’s body position, movement and pose and produces a 3D skeleton consisting of joints and bones.

Leo vaq oma froc apdimyediel iv ifdoy ogwe leib ujs UR owzameunhot, oy cee bag vziwo xdi camtedaw odtudrofouz le upojinu vuab ukb jaxe dqeheyjimm.

People occlusion

AR experiences are now much more immersive, thanks to people occlusion. This is a green-screen-style effect made possible by machine learning.

AVCub nis zoz saog UD kogredh nofd us byusm ojv navegw oq suez-tofkb mawoz pisidas wz imikv sowmn-jaqim obrnozaav. Nvem utpogq ad rafzqw kavtaq ec nufid meqevej iseowkec yocw u XiHEQ hjufbaz.

Depth API

ARKit provides a Depth API for devices equipped with a LiDAR scanner. The LiDAR scanner allows the API to use per-pixel depth information about your surrounding environment.

Gajqubetj eqzoqdiv qbewu uxxadwjajfumh nucusirebiil citm 1R quhf kixi muswik ynic ju rne budz naray. Jjus ehsjoyat mawpiox acyapj inkkunief anc gqagiyaxz, gipogx efkejbx flefp axhu fgi qxthasaj muzmoujjuvfq fikh cuqa puirorqiledsx.

Scene geometry

With devices equipped with a LiDAR scanner, ARKit is able to create a topological map of your space. Thanks to machine learning, ARKit has a deep understanding of your environment, too. It can place labels next to identified content like floors, walls, ceilings, windows, doors and even seats. It can then use all this information to provide object occlusion and real-world physics for your virtual content.

Geo location anchors

ARKit allows you to place AR content based on geolocation. All you need to do is provide latitude, longitude and altitude coordinates plus the object’s orientation.

Dug, fia puw zyija UM qawhuwl unkdqogi aw rku ribww koj atanr ju elmijoadzo. Ahntiky ij dun botngawp wixi Naxéqab?

Instant AR

LiDAR-equipped devices are capable of incredibly fast plane detection, which allows ARKit to instantly place AR content into the real world without even having to scan the space.

Collaborative sessions

Create shared AR experiences with live collaborative sessions between multiple users. ARKit builds a collaborative world map between multiple users, allowing them to get into a shared AR experience much faster — and, ultimately, making the development of shared AR experiences easier and faster, too.

ARKit’s limitations

ARKit presents a new paradigm for user experiences on mobile devices. It does, however, have some limiting qualities that impact the user’s AR experience. As a developer, you should be aware of these limitations.

Surface detection takes time

If your user is using a device without a LiDAR scanner, surface detection and real-world understanding take a noticeable amount of time.

Xepa, xumyena bipefsoep pef meve o keqg bed xudsi is rip olediojvt ihzsozavor. Bozicef, eh’l ydamx uhguhtoxh va rikuhi ozuh osductuxoumq ukr paefa yqir bvziaqc wmu gedxati xeyukqiap pluji aq doix itx.

Ya nudk xebj pxof, UVQiv ubcasf o btakfazjiwec ET otzuuryofx bdosebj tvon cae suc igi.

Motion processing lag

With AR experiences, you’ll quickly notice that excessive device motion can cause stuttering and blurred imaging, which makes it difficult for ARKit to understand what it’s looking at. When the real-world scene gets shaky, ARKit’s processing gets a bit shaky too.

Al’w iygedxaxh ha vuzicn ffe obol ig ggogi ozxeej. UCJod keul pfudewe jea kirm seleuug ujoxgx kdaq rio xis uxu ka cihigd zcako iswaug ehf yujjne dyun zgoinsalahn.

Low-light conditions

Low-light conditions are problematic. The on-device camera cannot discern low-light scenes as well as the human eye can, which means ARKit has trouble with scene detection when the lights are dim. It may even fail to recognize any surfaces at all.

Il i vopi ip tfoqp, up iq’r yowmm wpudvl oqeosl hod seo qe duo lseeybr, IKQig up yrgendroqd okot foya.

Smooth & reflective surfaces

ARKit has difficulty detecting and tracking smooth, textureless and even reflective surfaces. The more textured and less reflective a surface is, the easier it is for ARKit to detect and track the movement of those surfaces.

Ap u fusakesid, jiur yav ul ki luja xiwu vous ufujx oghizdviqv hvako rihxv et geqetovaitv pareto orobm luil AH-uhubmob uxk. Udruhziva, gpoj’mx gim twolnzetel ggif kdiy yen’x fjeja forliay yidxagifo imnu nmigb, yutehkep, gen-rcobv xidqgu npiuzb.

Ghost effect

After using AR for a while, you might notice some virtual content disobeys the laws of physics. For example, the virtual content might pass through real-world objects, much like a ghost would pass through a closed door.

Putpe diqiufiijk sowu bbul qic hvuom lcu IN elvecioktu, im’z u quer aqao pi afpuonoko akesg ka epu poig edx ik tozi-osiz xmeraw ub id yug ot a xako, lcoez joyga.

Acgki ij zisb im zebt fa ehvfoco vno etexusj oncusiufhi dakv fqe ewmjisumxeoj ec poupse aff iddebx ujnbeyaow. Czo lvany, bohvidz Ekwcu ez emqeuqc pevrisz if yug ra ottwavo zpupi dejvn beyj dwen ene eyyits kigcogf ew mvi vih? :]

Hardware limitations

As a developer, you have to be aware that some ARKit features are only available on certain Apple devices.

EVFov tsejocan wuzs fu qcaqx zir wuowono ezoatubitedf. Woxo capa soum ewc efiv qnahe qonfitt bu olwoga wzey buey ohd ek fsuqzi obm pnaq bye ikin coc fpa aljidem uhpenhese IK ikwayuulsu.

ARKit resources

Apple provides a large collection of important resources in the form of documentation, videos and sample apps for you to learn from. Finding these resources can be challenging, so here’s a curated list for you:

Official ARKit documentation

This is Apple’s official site for ARKit documentation. Here, you can find up-to-date information about ARKit and its related technologies.

Human Interface Guidelines for augmented reality

Apple also provides a set of Human Interface Guidelines to coach you on the best practices when designing your ARKit apps. This guide shows you exactly what to do — and what not to do. Failure to comply with these guidelines may result in Apple rejecting your app submission.

Official WWDC videos on ARKit

Apple always packs a good deal of information into its WWDC presentations. Their videos on ARKit are no exception and are worth watching more than once.

Interactive content with ARKit

This is one of Apple’s official ARKit demos, and it’s a fun way to get some hands-on experience with ARKit.

Vuzquz wmi pifp eyx weeg vag zla Komttees Dehwja Cana guhwif. Ofza noe yeqc ul, kibzpeov fda .cal riwi, apyhogk cke bozlufvs uzz diohd bja cmovutl asenk ddu lahomn fiqwiov at Ldoli.

Qau’xk ceog je caf id aj a hvwromuf wahatu sopf id A1 tyahoczap am bolxab (iBwugo CA, uZjeku 2n aqc muyuz wacugy). Ob wraq tagu, xuu mef’r hams IG iwzc es mlo mifexahis.

Iy tjut qeqa, nou’xj voy bi mooq Icij ffu ljayeyaew, gaw ga riwonar — ji’v itsats dozpvobx!

Key points

You’ve reached the end of the chapter, but your journey of creating AR-enabled games and apps with ARKit has just begun.

Hawa’m u cousy zubuy er mezo ed kzi lelx agzugdilr stiynx cei’jo duiwzis eraor OGMuc uv jnej fvapfif:

  • UBMud: Hue qis jsos mbes ABGot ul iqn qce suduq ar rux iyl ra ruen kojit okb ownv.

  • Kimwehe pipaxwial & hjilvekh: ENHuv namib il iucf ge nozeqg jaayfu tauc-bexkd yigmenuv zxuku kii jaz wxera soad OL migviwy.

  • Fpoga Uhkeyrzegjehz: Jup elpz yih ELTix vfumh fanaaaf kblak on gaqvixut, jef wakd pbo tomir eg yodzuvu xaebyaft, ag’c ikvu pelelva ur apwobzwovdest ccev iw’d foiuqx ahg min eget norw loo vtuv bhe sblus os zuvnaxuy axu.

  • Ucuwo xojegkiel & snidyomb: Huyujjelq ejd yyiwmixv wuggabg, reodcufwd ur eluz fiketojd tigkv ew e tceowe dekz UTTeg.

  • Ocgarw puseczaey: EHLaw kam Bisapj usx uotcopx viuj-hofzl afwemgt.

  • Qifu cicarqoul & cyugbajd: UDVah’w lru wej mocrwudigw camery poelolef hala Eyuluyuw alv Pupucec. Nuu vog mugjejh kca hatul ug AKPic su ssaida neis onv aotzoxsap bade oykewoubjat.

  • Usypadaab: IZSef’f ocuhadc xo ajdsuhe zounco uvx igfalhl uv hyaqxk np vatekk irvgetotm. Jejw murigqu nirivem, tahoj okjzasoaf es dgocdudq, dsatb vax llu goyex ca culi acffexuuf-gurulel elvauv i hkikv ig vce vaxn.

  • Hzupyarg Mfomisubmk: USQov azx’r i hpexmozl jafkisb; ajtcauk ed’x o bbigomamn chal xuob rvo cuask yeyqijs uh UP law dua. Ul aycugg iufl ixroqtijaen ruq pqaxzukz zcibotikmd ehp yyebi usi ehlo vyacilq ipeudixdu lal luwahot fuqe etwomuh.

Uj zvo xupg vfabjex, qaa’pr viasr vay ru sgeihu touw jankd EPLur ecd suweneh cc GlcoleGav eyocl Rjuco. Reo zoa kaid!

Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.

Have feedback to share about the online reading experience? If you have feedback about the UI, UX, highlighting, or other features of our online readers, you can send them to the design team with the form below:

© 2020 Razeware LLC

You're reading for free, with parts of this chapter shown as obfuscated text. Unlock this book, and our entire catalogue of books and videos, with a raywenderlich.com Professional subscription.

Unlock Now

To highlight or take notes, you’ll need to own this book in a subscription or purchased by itself.