Core Image: From CIImage to Metal and Beyond

Core Image is an oft-overlooked first-party framework on Apple platforms for the efficient processing of images and videos. It includes a wide range of built-in filters, as well as the ability to build your own custom filters using Metal. Discover how to integrate Core Image into your own apps, and how the optimized image processing can save your users time and battery life. Learn how to perform real-time video processing, and how you can extend Core Image by implementing your own custom filters directly in Metal. By Sam Davies.

Leave a rating/review
Save for later
Comments
Share

Who is this for?

iOS, macOS and tvOS developers that have heard of and may have used Core Image in the past, and are interested in learning more about this powerful framework.

Covered concepts

  • The Core Image Pipeline: image input, processing & output. Why is it efficient?
  • Common issues with importing, displaying and exporting images using Core Image.
  • Using built-in filters, and the new iOS 13 API.
  • Creating custom composite filters.
  • Using image mattes to perform segmented image processing.
  • Creating custom filters using Metal.
  • Color, warp and general kernels.
  • Video processing with Core Image.
  • How to perform real-time processing of a camera stream.

Part 1: The Core Image Pipeline

01
Toggle description

Create a CIImage from a file, discover how to handle rotation and see how to import a depth map in a portrait photo.

Toggle description

Instantiate a CIFilter and apply it to an image. Compare the string-based filter instantiation to the new iOS 13 static method approach.

Toggle description

Integrate multiple filters in a chain to create complex image processing effects. See how Core Image efficiently concatenates kernels.

Toggle description

Display the result of the Core Image pipeline in a UIImageView. Discover how to cope with issues with aspect ratio, by creating a CGImage.

Toggle description

Save the output from a Core Image pipeline into a file, using a Core Image context.

Part 2: Built-in Filters

Toggle description

List the categorized built-in filters, their documentation and available parameters.

Toggle description

Photos taken in portrait mode have powerful segmentation algorithms applied, splitting the foreground from the background. Apply background blurring with a single filter.

Toggle description

Use an image matte to selectively apply a complex composite chain of filters to the background of an image.

Toggle description

Apply a masked filter to the foreground of the image, and composite the foreground and background back together.

Toggle description

Use a Core Image generator filter to create a fake sun and then a blend filter to insert it into your photographic masterpiece.

Part 3: Creating Custom Filters with Metal

Toggle description

Filter kernels come in three forms, and represent highly optimised image processing routines that are performed on the GPU. Create your first kernel—for a passthrough filter.

Toggle description

To use a metal kernel, you must access it from inside a CIFilter. See how to create a filter that utilizes your freshly forged kernel.

Toggle description

Color kernels manipulate pixel values independent of spatial location. You can use them to change a colorspace.

Toggle description

Custom kernels give you the full power to create any filter you can imagine. See how to build a bilateral filter in Core Image.

Combining custom filters is very similar to combining built-in filters, and uses the same kernel optimization techniques. Combine the color space transform and bilateral filter kernel for some hardcore image processing!

Part 4: Video Processing with Core Image

Toggle description

Set up an AVFoundation session to access the camera and stream frames into Core Image, by creating a Core Image camera object.

Toggle description

See how to create a Metal-backed view that is designed to display video frames once they've been processed.

Toggle description

Create a simple Core Image pipeline and use it to process video frames from the camera in real time.