What’s new in the Foundation Models framework - WWDC26 - Videos - Apple Developer

View in English

-->

More Videos

About

Summary

Transcript

Code

What’s new in the Foundation Models framework

Explore what's new in the Foundation Models framework. Learn how to access Private Cloud Compute, integrate third-party and open source models, and work with vision capabilities. Discover context management APIs, built-in semantic search, and powerful primitives for creating agentic experiences in your apps.

Chapters

0:00 - Introduction

2:34 - New on-device model

3:21 - Vision: image understanding

4:20 - Private Cloud Compute

6:46 - Model abstraction layer

7:32 - Partner model integrations

9:40 - System tools: Vision and Spotlight

10:57 - Dynamic Profiles for agentic apps

13:46 - Composing models and configurations

15:30 - Evaluations framework

16:02 - The fm command line tool

17:13 - Foundation Models Python SDK

17:55 - Open source and framework utilities

19:24 - Next steps

Resources

Expanding generation with tool calling

Analyzing images with multimodal prompting

Composing dynamic sessions with instructions and profiles

Adding server-side intelligence with Private Cloud Compute

HD Video

SD Video

Search this video…

Hello everyone, and welcome! My name is Erik! And my name is Zhen! Last year we introduced the Foundation Models framework with features like guided generation, snapshot streaming, and the powerful tool protocol. We were blown away by your excitement for the Foundation Models framework in year one, and we think you're going to love what we've lined up for you this year, even more! We're going to walk through everything that's new in the framework this year. And let me tell you, this release is packed! Our 2027 release is all about integrations into and beyond the OS, a wider variety of models, and new primitives for building agentic experiences. Let's kick it off with one of our most exciting updates. The Foundation Models framework, including many of the brand new APIs that we're announcing today, is going open source! And doing it in style! In addition to the core framework, we're also releasing a new package, Foundation Models framework utilities, that will be updated between OS releases to give you access to emerging and experimental building blocks. And during the course of this session, you'll be hearing about multiple other packages all joining the Foundation Models framework ecosystem. So, are you ready to jump in with us? I'm going to cover everything related to the new models, new modalities, and new tools we're adding to the framework. And I'm going to show you the brand new APIs we've made to help you get the most out of them! We've got a full schedule! First up, we've got a whole pile of model updates for you, including updates to the on-device model and access to server models.<br>We've also added new system tools that supercharge your session with help from Spotlight and the Vision framework. Afterwards, Zhen will rejoin us to tell you about a powerful new API called dynamic profiles, our new primitive for creating agentic experiences. Zhen will also dig into the brand new Evaluations framework and its tight integration with Foundation Models.<br>And to wrap up, we have some exciting news to share about Mac-specific productivity tools, so make sure to stick around! Let's jump right in with our new model!<br>This release comes with a new on-device model, rebuilt from the ground up, and better across the board. It's more intelligent; better at logic and tool calling. In iOS 26.4, we released new APIs for inspecting the model's context size and counting the tokens in instructions, prompts, and transcripts. You'll want to use these going forward to adapt your app to the hardware it's running on. We've also been hard at work on refining our guardrails. You may have noticed adjustments in iOS 26.4 to reduce the number of false positives, and we're continuing to make even more improvements in iOS 27.<br>In addition the on-device model is also gaining Vision capabilities, which unlocks entire new categories of applications.<br>The API is simple, a natural extension of the existing prompt builders.<br>Here we've created a session, and we want to ask about the photo of the origami on the right.<br>Simply insert an image attachment into your prompt, together with text.<br>Now, the model can answer questions about the image.<br>Image attachments can be created from a variety of types including; UIImage, NSImage, CGImage Core Image types, CoreVideo Pixel Buffers, and file URLs.<br>The model supports images in any size and aspect ratio, so you don't need to crop or pad to any particular shape.<br>Arbitrary image sizes are allowed, but bear in mind that larger images will consume more tokens and incur more latency. Thanks to all these upgrades, the on-device system language model is more capable than ever. But if you need even more horsepower, we're giving you access to a brand new...