Enhance RAW image processing with Core Image - WWDC26 - Videos - Apple Developer

View in English

-->

More Videos

About

Summary

Transcript

Code

Enhance RAW image processing with Core Image

Harness the power of version 9 of the Core Image RAW processing APIs to dramatically improve image quality in your apps, with improved sharpness and more defined color, while using the Apple Neural Engine for optimal performance. Take advantage of the CIRAWFilter API to let your users edit RAW photos by changing exposure, noise reduction, sharpness, contrast and more. And explore new CIImageProcessor APIs that optimize performance by giving you precise control over tile sizing and buffer management.

Chapters

0:00 - Introduction

0:52 - How Core Image supports RAW

2:48 - The evolution of RAW support

3:33 - RAW 9 overview

3:56 - RAW 9 quality improvements

5:50 - Enable and edit RAW 9 with CIRAWFilter API

8:33 - RAW 9 performance overview

9:19 - Interactive editing

10:52 - Exporting to other formats

11:50 - New CIImageProcessor features

Resources

Extended Virtual Addressing Entitlement

HD Video

SD Video

Related Videos

WWDC22

Display EDR content with Core Image, Metal, and SwiftUI

WWDC21

Capture and process ProRAW images

Search this video…

Welcome everyone. My name is David Hayward and I am excited to talk about enhancements to Core Image, and its support for RAW image files.<br>I will talk about four main topics. First, I will review how RAW image files from cameras, are supported by Core Image on Apple platforms. And show you significant RAW quality improvements that are coming in iOS, iPadOS, macOS, and visionOS 27.<br>After that, I will discuss how to get optimal performance when rendering RAWs.<br>And lastly, I will describe features, that Apple added to the CIImageProcessor class for RAW.<br>To start, here is a quick summary of how Core Image supports RAW. RAW files come from a diverse set of camera makes and models, but unlike HEIFs and JPEGs, they need special handling before they can be displayed.<br>The first step, is to parse the file's metadata, and unpack the RAW sensor values. At this stage, each pixel location only has a red, green, or blue value arranged in a mosaic pattern. This image is cropped way in, so that this pattern is visible.<br>The next step, is to demosaic the sensor data, so that each pixel contains red, green, and blue values.<br>The following stage is to denoise the pixels, so that the image is free of photon noise, read noise and thermal noise. After this, convolutions are applied to sharpen edges and add local contrast.<br>The last major step is to adjust white balance, exposure, color and tone to make a pleasing final image.<br>The algorithms for all these steps are built in to iOS, iPadOS, macOS, and visionOS.<br>This means that people can view RAW files in system apps like Finder, Preview and Freeform.<br>In fact, any app or framework that uses the Image IO API gets RAW support automatically.<br>And beyond basic viewing, apps that use the CIRAWFilter API, can present advanced editing controls.<br>This API is used by apps like Photos, Pixelmator Pro, and others like Nitro, Acorn, and many more.<br>Starting back in 2006, the system included hand-tuned calibrations for just 21 camera models. In the years since, that number has grown to 784 models, across all major camera vendors. This even includes support for Apple Pro RAW files for iPhone cameras.<br>But the key feature of RAW, is that a beloved photo you took years ago, can be reprocessed using the latest state-of-the-art algorithms. Apple's RAW processing pipeline has been updated eight times, each improving demosaic, denoise and color. Some older versions have been maintained, so that people can still use them if desired.<br>Now, Apple has made its biggest update yet, RAW 9! This version dramatically improves the rendering of RAW files. It is built atop a tiled CoreML model, that combines demosaic with denoise for best quality. And the model is run on device using the Apple Neural Engine cores, for optimal performance.<br>Let me demonstrate the improvements up close.<br>This is a zoomed-in crop of a low noise image using RAW 8. This Sony Alpha 7 II image of a vintage dial indicator actually looks quite good. However, when you explore that same image under RAW 9, the image is sharper, clearer, and the fine text is easier to read.<br>The differences are even more dramatic, when you view high noise images. First, observe the actual RAW data that is contained in this very noisy ISO 51,200 image.<br>In this example from a Canon 5D Mark III, the image is a 10x Crop of a box of crayons. There is so much luma and chroma noise in the RAW data, that it's impossible to discern the unique color of each crayon. Using our previous algorithms, this is the result! RAW 8 did an acceptable job of recovering the actual colors in the scene. But if you examine the results under RAW 9, the output is significantly better. The colors are accurate and well defined. Even the shiny specular...