All videos. All books. One low price.

Get unlimited access to all video courses and books on this site with the new raywenderlich.com Ultimate Subscription. Plans start at just $19.99/month.

Home iOS & Swift Tutorials

Vision Framework Tutorial for iOS: Scanning Barcodes

In this Vision Framework tutorial, you’ll learn how to use your iPhone’s camera to scan QR codes and automatically open encoded URLs in Safari.

5/5 4 Ratings

Version

  • Swift 5, iOS 14, Xcode 12

Barcodes are everywhere: on products, in advertising, on movie tickets. In this tutorial, you’ll learn how to scan barcodes with your iPhone using the Vision Framework. You’ll work with Vision Framework APIs such as VNDetectBarcodesRequest, VNBarcodeObservation and VNBarcodeSymbology, as well as learn how to use AVCaptureSession to perform real-time image capture.

That’s not all! You’ll also become familiar with:

  • Using the camera as an input device.
  • Generating and evaluating an image confidence score.
  • Opening a web page in SFSafariViewController.

Getting Started

Download the starter project by clicking the Download Materials button at the top or bottom of this tutorial. Open the project in Xcode from starter and explore the project.

Take a look at ViewController.swift. You’ll find some helper methods already in the code.

Note: To follow this tutorial, you’ll need a physical iPhone 5s or later running iOS 11 or later. To run the app on your physical device, be sure to set your team in the Signing and Capabilities section of the Xcode project settings. If you need help setting up your device and Xcode project, check out this app store tutorial.

Before you start scanning barcodes, you’d better get permission to use the camera.

Getting Permission to Use Camera

To protect user privacy, Apple requires developers to get permission from users before accessing their camera. There are two steps to prepare your app to ask for the right permission:

  1. Explain why and how your app will use the camera by adding a key and value in your Info.plist.
  2. Use AVCaptureDevice.requestAccess(for:completionHandler:) to prompt the user with your explanation and get user input for permission.

The starter project includes the key-value pair in Info.plist. You can find it under the key Privacy – Camera Usage Description.

To prompt the user for permission to use the camera, open ViewController.swift. Next, find // TODO: Checking permissions inside checkPermissions(). Add the following code inside the method:

switch AVCaptureDevice.authorizationStatus(for: .video) {
// 1
case .notDetermined:
  AVCaptureDevice.requestAccess(for: .video) { [self] granted in
    if !granted {
      showPermissionsAlert()
    }
  }

// 2
case .denied, .restricted:
  showPermissionsAlert()

// 3
default: 
  return
}

In the code above, you ask iOS for the current camera authorization status for your app.

  1. When the status isn’t determined, meaning the user hasn’t made a permissions selection yet, you call AVCaptureDevice.requestAccess(for:completionHandler:). It presents the user with a dialog asking for permission to use the camera. If the user denies your request, you show a pop-up message asking for permission again, this time in the iPhone settings.
  2. If the user previously provided restricted access to the camera, or denied the app access to the camera, you show an alert asking for an update to settings to allow access.
  3. Otherwise, the user already granted permission for your app to use the camera, so you don’t have to do anything.

Build and run and you’ll see the following:

Popup requesting user permissions to access the camera

With the camera permission in place, you can move on to starting a capturing session.

Starting an AVCaptureSession

Now you have permission to access the camera on your device. But when you dismiss the alert on your phone, nothing happens! You’ll fix that now by following these steps to start using the iPhone camera:

  1. Set quality for the capture session.
  2. Define a camera for input.
  3. Make an output for the camera.
  4. Run the Capture Session.

Setting Capture Session Quality

In Xcode, navigate to ViewController.swift. Find setupCameraLiveView() and add the following code after // TODO: Setup captureSession:

captureSession.sessionPreset = .hd1280x720

captureSession is an instance of AVCaptureSession. With an AVCaptureSession, you can manage capture activity and coordinate how data flows from input devices to capture outputs.

In the code above, you set the capture session quality to HD.

Next, you’ll define which of the many iPhone cameras you want your app to use and pass your selection to the capture session.

Defining a Camera for Input

Continuing in setupCameraLiveView(), add this code right after // TODO: Add input:

// 1
let videoDevice = AVCaptureDevice
  .default(.builtInWideAngleCamera, for: .video, position: .back)

// 2
guard
  let device = videoDevice,
  let videoDeviceInput = try? AVCaptureDeviceInput(device: device),
  captureSession.canAddInput(videoDeviceInput) 
  else {
    // 3
    showAlert(
      withTitle: "Cannot Find Camera",
      message: "There seems to be a problem with the camera on your device.")
    return
  }

// 4
captureSession.addInput(videoDeviceInput)

Here you:

  1. Find the default wide angle camera, located on the rear of the iPhone.
  2. Make sure your app can use the camera as an input device for the capture session.
  3. If there’s a problem with the camera, show the user an error message.
  4. Otherwise, set the rear wide angle camera as the input device for the capture session.

With the capturing session ready, you can now make the camera output.

Making an Output

Now that you have video coming in from the camera, you need a place for it to go. Continuing where you left off, after // TODO: Add output, add:

let captureOutput = AVCaptureVideoDataOutput()
// TODO: Set video sample rate
captureSession.addOutput(captureOutput)

Here, you set the output of your capture session to an instance of AVCaptureVideoDataOutput. AVCaptureVideoDataOutput is a capture output that records video and provides access to video frames for processing. You’ll add more to this later.

Finally, time to run the capturing session.

Running the Capture Session

Find the // TODO: Run session comment and add the following code directly after it:

captureSession.startRunning()

This starts your camera session and enables you to continue with the Vision framework.

But first! What many forget is stopping the camera session again.

To do this, find the // TODO: Stop Session comment in viewWillDisappear(_:) and add the following after it:

captureSession.stopRunning()

This stops your capture session if your view happens to disappear, freeing up some precious memory.

Build and run the project on your device. Just like that, your rear camera shows up on your iPhone screen! If you have your phone facing your computer, it’ll look similar to this:

Using Vision Framework, your camera session shows your rear camera on your iPhone screen

With that in place, it’s time to move on to the Vision framework.

Vision Framework

Apple created the Vision Framework to let developers apply computer vision algorithms to perform a variety of tasks on input images and video. For example, you can use Vision for:

  • Face and landmark detection
  • Text detection
  • Image registration
  • General feature tracking

Vision also lets you use custom Core ML models for tasks like image classification or object detection.

Vision and the Camera

The Vision Framework operates on still images. Of course, when you use the camera on your iPhone, the image moves smoothly, as you would expect from video. However, video is made up of a collection of still images played one after the other, almost like a flip book.

Cartoon iPhone holding a camera

When using your camera with the Vision Framework, Vision splits the moving video into its component images and processes one of those images at some frequency called the sample rate.

In this tutorial, you’ll use the Vision Framework to find barcodes in images. The Vision Framework can read 17 different barcode formats, including UPC and QR codes.

In the coming sections, you’ll instruct your app to find QR codes and read their contents. Time to get started!

Using the Vision Framework

To implement the Vision Framework in your app, you’ll follow three basic steps:

  1. Request: When you want to detect something using the framework, you use a subclass of VNRequest to define the request.
  2. Handler: You process that request and perform image analysis for any detection using a subclass of VNImageRequestHandler.
  3. Observation: You analyze the results of your handled request with a subclass of VNObservation.

Time to create your first Vision request.

Creating a Vision Request

Vision provides VNDetectBarcodesRequest to detect a barcode in an image. You’ll implement it now.

In ViewController.swift, find // TODO: Make VNDetectBarcodesRequest variable at the top of the file and add the following code right after it:

lazy var detectBarcodeRequest = VNDetectBarcodesRequest { request, error in
  guard error == nil else {
    self.showAlert(
      withTitle: "Barcode error",
      message: error?.localizedDescription ?? "error")
    return
  }
  self.processClassification(request)
}

In this code, you set up a VNDetectBarcodesRequest that will detect barcodes when called. When the method thinks it found a barcode, it’ll pass the barcode on to processClassification(_:). You’ll define processClassification(_:) in a moment.

But first, you need to revisit video and sample rates.

Vision Handler

Remember that video is a collection of images, and the Vision Framework processes one of those images at some frequency. To set up your video feed accordingly, find setupCameraLiveView() and locate the TODO you left earlier: // TODO: Set video sample rate. Then, add this code right after the comment, and before the call to addOutput(_:):

captureOutput.videoSettings = 
  [kCVPixelBufferPixelFormatTypeKey as String: Int(kCVPixelFormatType_32BGRA)]
captureOutput.setSampleBufferDelegate(
  self, 
  queue: DispatchQueue.global(qos: DispatchQoS.QoSClass.default))

In this code, you set your video stream’s pixel format to 32-bit BGRA. Then, you set self as the delegate for the sample buffer. When new images are available in the buffer, Vision calls the appropriate delegate method from AVCaptureVideoDataOutputSampleBufferDelegate.

Because you’ve passed self as the delegate, you must conform ViewController to AVCaptureVideoDataOutputSampleBufferDelegate. Your class already does this and has a single callback method defined: captureOutput(_:didOutput:from:). Find this method and insert the following after // TODO: Live Vision:

// 1
guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { 
  return 
}

// 2
let imageRequestHandler = VNImageRequestHandler(
  cvPixelBuffer: pixelBuffer,
  orientation: .right)

// 3
do {
  try imageRequestHandler.perform([detectBarcodeRequest])
} catch {
  print(error)
}

Here you:

  1. Get an image out of sample buffer, like a page out of a flip book.
  2. Make a new VNImageRequestHandler using that image.
  3. Perform the detectBarcodeRequest using the handler.

Live streaming with a cartoon iPhone

Vision Observation

Think back to the section on the Vision Request. There, you built the detectBarcodeRequest which calls processClassification(_:) if it thinks it found a barcode. For your last step, you’ll fill out processClassification(_:) to analyze the result of the handled request.

In processClassification(_:), locate // TODO: Main logic and add the following code right below it:

// 1
guard let barcodes = request.results else { return }
DispatchQueue.main.async { [self] in
  if captureSession.isRunning {
    view.layer.sublayers?.removeSubrange(1...)

    // 2
    for barcode in barcodes {
      guard
        // TODO: Check for QR Code symbology and confidence score
        let potentialQRCode = barcode as? VNBarcodeObservation 
        else { return }

      // 3
      showAlert(
        withTitle: potentialQRCode.symbology.rawValue,
        // TODO: Check the confidence score
        message: potentialQRCode.payloadStringValue ?? "" )
    }
  }
}

In this code, you:

  1. Get a list of potential barcodes from the request.
  2. Loop through the potential barcodes to analyze each one individually.
  3. If one of the results happens to be a barcode, you show an alert with the barcode type and the string encoded in the barcode.

Build and run again. This time, point your camera at a barcode.

Booooom you scanned it!

Note: You can find a sample barcode and sample QR code in the project folder, under Sample/barcode.png and Sample/qrcode.png respectively.

Scanning a barcode and a QR code with the Vision Framework

So far, so good. But what if there was a way for you to know with what certainty the object you point at is actually a barcode? More on this next…

Adding a Confidence Score

So far, you’ve worked extensively with AVCaptureSession and the Vision Framework. But there are more things you can do to tighten your implementation. Specifically, you can limit your Vision Observation to recognize only QR type barcodes and you can make sure the Vision Framework is certain it’s found a QR code in an image.

Cartoon iPhone scientist

Whenever your barcode observer analyzes the result of a handled request, it sets a property called confidence. This property tells you the result’s confidence level, normalized to [0, 1], where 1 is the most confident.

Inside processClassification(_:), find// TODO: Check for QR Code symbology and confidence score and replace the guard:

guard 
  let potentialQRCode = barcode as? VNBarcodeObservation 
  else { return }

with:

guard 
  // TODO: Check for QR Code symbology and confidence score
  let potentialQRCode = barcode as? VNBarcodeObservation,
  potentialQRCode.confidence > 0.9
  else { return }

Here you ensure Vision is at least 90% confident it’s found a barcode.

Now in the same method, locate // TODO: Check the confidence score. The message key’s value below is currently potentialQRCode.payloadStringValue ?? "". Change it to:

String(potentialQRCode.confidence)

Now, instead of showing the barcode’s payload in the alert, you’ll show the confidence score. Because the score is a number, you coalesced the value to a string so it can display in the alert.

Build and run. When you scan the sample QR code, you’ll see the confidence score in the alert that pops up.

Vision Framework is 100% confident it sees a QR code

Nicely done! As you see, the confidence score of the sample QR code is quite high, meaning Vision is certain this is actually a QR code.

Using Barcode Symbology

Currently, your scanner responds to all types of barcodes. But what if you want your scanner to respond only to QR codes? Fortunately, Vision has a solution for you!

In the Vision Observation section above, you used VNBarcodeObservation‘s confidence to find how sure Vision is it found a barcode. Similarly, you can use VNBarcodeObservation‘s symbology property to check the type of symbol found in the Vision Request.

Once again, in processClassification(_:) locate // TODO: Check for QR Code symbology and confidence score. Then inside the guard add the following condition after the first let clause:

potentialQRCode.symbology == .QR,

Your guard should now look as follows:

guard
  let potentialQRCode = barcode as? VNBarcodeObservation,
  potentialQRCode.symbology == .QR,
  potentialQRCode.confidence > 0.9
  else { return }

With this new addition, you’ve added a check to ensure the barcode is of type .QR. If the barcode is not a QR code, you ignore the request.

Build and run. Scan the sample QR code and see that your app now ignores the sample barcode.

Scanning a QR code with an iPhone

Finally, you’ll finish up this nifty new feature by opening an encoded URL in Safari.

Opening Encoded URLs in Safari

Your barcode scanner is now complete. Of course, simply scanning and reading a barcode is only half the battle.

Your users will want to do something with the result of the scan. They did give you permission to use their camera, after all!

Often, QR codes contain URLs that point to interesting websites. When users scan the QR code, they want to go to the page at the encoded address.

In the following sections, you’ll work on doing just that.

Setting Up Safari

First, you need to add support for opening a link in a SFSafariViewController.

Locate observationHandler(payload:) and after // TODO: Open it in Safari add the following:

// 1
guard 
  let payloadString = payload,
  let url = URL(string: payloadString),
  ["http", "https"].contains(url.scheme?.lowercased()) 
  else { return }

// 2
let config = SFSafariViewController.Configuration()
config.entersReaderIfAvailable = true

// 3
let safariVC = SFSafariViewController(url: url, configuration: config)
safariVC.delegate = self
present(safariVC, animated: true)

With this code, you:

  1. Make sure the encoded string in the QR code is a valid URL.
  2. Set up a new instance of SFSafariViewController.
  3. Open Safari to the URL encoded in the QR code.

Next, you’ll work on triggering this function after having scanned a valid QR code.

Opening Safari

To open a URL with this function, you must tell processClassification(_:) to use observationHandler(payload:) if it finds a QR code.

In processClassification(_:), replace:

showAlert(
  withTitle: potentialQRCode.symbology.rawValue,
  // TODO: Check the confidence score
  message: String(potentialQRCode.confidence)

with:

observationHandler(payload: potentialQRCode.payloadStringValue)

Here, instead of showing an alert when your barcode reader encounters a QR code, you open Safari to the URL encoded in the QR code.

Build and run and point your device towards the sample QR code.

Opening in Safari a URL encoded in a QR code

Congratulations! You developed a fully-featured app that uses the iPhone’s camera to scan and read QR codes, and opens encoded URLs in Safari!

Where to Go From Here?

You can download the completed project files by clicking the Download Materials button at the top or bottom of the tutorial.

In this tutorial, you learned how to:

  • Use the Vision Framework.
  • Make and perform VNRequests.
  • How to use SFSafariViewController.
  • How to limit the kind of barcodes to scan for.

This project is only the beginning of what you can do with the Vision Framework. If you want to learn more, check out our Face Detection Tutorial Using the Vision Framework for iOS. For a more advanced challenge, check out AR Face Tracking Tutorial for iOS: Getting Started.

You can also visit the developer documentation for the Vision Framework, AVCaptureSession, VNDetectBarcodesRequest, VNBarcodeObservation, and VNBarcodeSymbology.

I hope you enjoyed this tutorial. If you have any questions or comments, please join the forum discussion below!

Average Rating

5/5

Add a rating for this content

4 ratings

More like this

Contributors

Comments