Vision Framework Tutorial for iOS: Contour Detection

Art is a very subjective thing. However, coding is not. Sure, developers can be opinionated at times, but how a computer interprets your code is very much not a matter of opinion.

So how can you, a developer, use code to create art? Maybe it’s not how you code but rather what you choose to do with it. Getting creative with the tools available to you can significantly affect the output.

Think about how Apple has been pushing the limit on computational photography. Most digital photography is about post-processing the pixels that come from a sensor. Different sets of algorithms change how the final output looks and feels. That’s art!

You can even use computer vision algorithms to create exciting filters and effects for images and photos. For instance, if you detected all of the contours in an image, you might have some cool material to make an artsy-looking drawing of the image. And that’s what this tutorial is all about!

In this tutorial, you’ll learn how to use the Vision framework to:

• Create requests to perform contour detection.
• Tweak settings to get different contours.
• Simplify the contours to create an artistic effect.

Sounds like fun, right? Exactly… art should be fun!

Getting Started

Click the Download Materials button at the top or bottom of this tutorial. The starter project includes some extensions, model files, and the UI.

If you build and run now, you’ll see instructions for tapping the screen and a functional settings icon.

You might notice that tapping the screen doesn’t do anything right now, but before you can get to the Vision part of this tutorial, you need to display an image on the screen.

Displaying an Image

While going through the starter project, you might have noticed an ImageView connected to the image property of the ContentViewModel. If you open up ContentViewModel.swift, you’ll see that image is a published property, but nothing is assigned to it.

The first thing you’ll need to do is change that!

Start by adding the following code directly after the three defined published properties in ContentViewModel.swift:

init() {
  let uiImage = UIImage(named: "sample")
  let cgImage = uiImage?.cgImage
  self.image = cgImage  
}

This code loads the image called sample.png from the asset catalog and obtains a CGImage for it, before assigning it to the image published property.

With that small change, go ahead and build and rerun the app and you’ll see the image below on your screen:

Now, when you tap on the screen, it should toggle between the above image and the blank screen you initially saw.

The blank screen will eventually contain the contours you detect using the Vision framework.

Vision API Pipeline

Before you start writing some code to detect contours, it’ll be helpful to understand the Vision API pipeline. Once you know how it works, you can easily include any of the Vision algorithms in your future projects; that’s pretty slick.

The Vision API pipeline consists of three parts:

1. The first is the request, which is a subclass of VNRequest – the base class for all analysis requests. This request is then passed to a handler.
2. The handler can be one of two types, either a VNImageRequestHandler or a VNSequenceRequestHandler.
3. Finally, the result, a subclass of VNObservation, is returned as a property on the original request object.

Often, it’s quite easy to tell which result type goes with which request type, as they’re named similarly. For instance, if your request is a VNDetectFaceRectanglesRequest, then the result returned will be a VNFaceObservation.

For this project, the request will be a VNDetectContoursRequest, which will return the result as a VNContoursObservation.

Whenever you’re working with individual images, as opposed to frames in a sequence of images, you’ll use a VNImageRequestHandler. A VNSequenceRequestHandler is used when working on sequences of images where you want to apply requests to a sequence of related images, for example frames from a video stream. In this project, you’ll use the former for single image requests.

Now that you have the background theory, it’s time to put it into practice!

Contour Detection

To keep the project nicely organized, right-click on the the Contour Art group in the project navigator and select New Group. Name the new group Vision.

Right click on the new Vision group and select New File…. Choose
Swift File and name it ContourDetector.

Replace the contents of the file with the following code:

import Vision

class ContourDetector {
  static let shared = ContourDetector()
  
  private init() {}
}

All this code does is set up a new ContourDetector class as a Singleton. The singleton pattern isn’t strictly necessary, but it ensures that you only have one ContourDetector instance running around the app.

Performing Vision Requests

Now it’s time to make the detector class do something.

Add the following property to the ContourDetector class:

private lazy var request: VNDetectContoursRequest = {
  let req = VNDetectContoursRequest()
  return req
}()

This will lazily create a VNDetectContoursRequest the first time you need it. The Singleton structure also ensures there’s only one Vision request, which can be reused throughout the app’s lifecycle.

Now add the following method:

private func perform(request: VNRequest,
                     on image: CGImage) throws -> VNRequest {
  // 1
  let requestHandler = VNImageRequestHandler(cgImage: image, options: [:])
  
  // 2
  try requestHandler.perform([request])
  
  // 3
  return request
}

This method is simple but powerful. Here you:

1. Create the request handler and pass it a supplied CGImage.
2. Perform the request using the handler.
3. Return the request, which now has the results attached.

In order to use the results from the request, you’ll need to do a bit of processing. Below the previous method, add the following method to process the returned request:

private func postProcess(request: VNRequest) -> [Contour] {
  // 1
  guard let results = request.results as? [VNContoursObservation] else {
    return []
  }
    
  // 2
  let vnContours = results.flatMap { contour in
    (0..<contour.contourCount).compactMap { try? contour.contour(at: $0) }
  }
      
  // 3
  return vnContours.map { Contour(vnContour: $0) }
}

In this method, you:

• flatMap the results into a single flattened array.
• Iterate over the contours in contour using compactMap to ensure only non-nil values are kept.
• Use contour(at:) to retrieve a contour object at a specified index.

1. Check that results is an array of VNContoursObservation objects.
2. Convert each result into an array of VNContours.
3. Map the array of VNContours into an array of your custom Contour models.

Note: The reason you convert from VNContour to Contour is to simplify some SwiftUI code. Contour conforms to Identifiable, so it's easy to loop through an array of them. Check out the ContoursView.swift to see this in action.