What's New in Apple Intelligence: Foundation Models, App Intents, and More

A clear breakdown of the latest Apple Intelligence updates for developers, covering the Foundation Models framework, Private Cloud Compute, App Intents, Image Playground, and Visual Intelligence.

If you have ever wished Apple's on-device AI could do more in your app without jumping through hoops, this year's updates are for you. Apple Intelligence has expanded in ways that give developers real tools to build smarter, faster, and more integrated experiences.

From native Swift APIs that plug directly into Apple's language model, to Siri understanding your app's actions without writing custom phrases, the gap between "AI-powered" and "actually useful" just got a lot smaller. And the best part: most of it works on-device, with no API keys and no cloud bills.

Here is a practical look at what changed and what it means for your app.

---

Foundation Models Framework

The Foundation Models framework gives you a native Swift API to interact directly with the same on-device model that powers Apple Intelligence.

You are not locked into Apple's model either. The LanguageModel protocol lets you swap in cloud models like Claude or Gemini, or any provider you build yourself.

Key features include:

• Multimodal prompts: Pass images alongside text so your model can reason about visual content.
• Vision framework tools: OCR and barcode readers are available as callable tools, all on-device.
• Dynamic Profiles: Swap models, tools, and instructions mid-session without restarting.
• Evaluations framework: Test your AI features across dynamic conditions, beyond what unit tests cover.

If your app is enrolled in the App Store Small Business Program and has fewer than 2 million first-time downloads, you can access the next-generation Apple Foundation Model on Private Cloud Compute at no cloud API cost.

---

Private Cloud Compute: When On-Device Is Not Enough

The on-device model is great for always-available tasks, but some features need more power. Private Cloud Compute (PCC) routes your session to a server-side model with a larger context window and stronger reasoning.

Capability	On-Device (SystemLanguageModel)	Server-Side (PrivateCloudComputeLanguageModel)
Privacy	Yes	Yes
Works offline	Yes	No
Usage limits	Unlimited	Daily limit
Reasoning	Not supported	Multiple levels
Context size	4K tokens	32K tokens

Switching to PCC takes one line:

// Route the session through Private Cloud Compute
let session = LanguageModelSession(model: PrivateCloudComputeLanguageModel())

Both models conform to the same LanguageModel protocol, so your tools, instructions, and respond() calls work without any other changes.

Always gate PCC usage behind an availability check and fall back gracefully:

if #available(iOS 27.0, macOS 27.0, watchOS 27.0, visionOS 27.0, *) {
    // Use the server-based model
} else {
    // Fall back to the on-device model
}

Before making a request, check whether PCC is actually available on the device:

let model = PrivateCloudComputeLanguageModel()

switch model.availability {
case .available:
    // Show your AI feature
case .unavailable(.deviceNotEligible):
    // Show an alternative UI
case .unavailable(.systemNotReady):
    // PCC is not ready yet
case .unavailable(let other):
    // Unknown reason
}

Handling Daily Usage Limits

PCC has a daily request quota per user. When someone gets close to or hits their limit, you should show them a clear status message instead of a dismissible alert:

if model.quotaUsage.isLimitReached {
    Text("Usage limit exceeded")
        .foregroundStyle(Color.red)
} else if case .belowLimit(let info) = model.quotaUsage.status {
    if info.isApproachingLimit {
        Text("Nearing usage limit")
            .foregroundStyle(Color.orange)
    }
}

// Offer an upgrade path if one is available
if let suggestion = model.quotaUsage.limitIncreaseSuggestion {
    Button("Show options") {
        suggestion.show()
    }
}

To test these quota states in Xcode: go to Product > Scheme > Edit Scheme > Run > Options, and select a simulated quota scenario from the dropdown.

---

Extended Reasoning

PCC supports configurable reasoning levels, giving you control over the tradeoff between speed and quality:

Level	When to use
.light	Fast tasks, simple completions
.moderate	Starting point for most features
.deep	Complex decisions, multi-step reasoning

let response = try await session.respond(
    to: "What are the tradeoffs in this architecture?",
    contextOptions: ContextOptions(reasoningLevel: .deep)
)

Deeper reasoning uses more of the context window and takes longer, but catches things lighter levels miss. Reasoning segments (the model's intermediate "thinking") do not appear in the final response, but you can review them when debugging complex prompts.

---

App Intents Framework

App Intents is how your app connects to Siri, Spotlight, Shortcuts, and Apple Intelligence. The big idea: instead of defining specific phrases, you define what your app can do using schemas, and Siri figures out the language.

Three core building blocks:

App Intents define actions. Each intent has a perform() method, required or optional parameters, and a localized title.

// Use the @AppIntent(schema:) macro for common actions
// Xcode generates the boilerplate including expected parameters

App Entities represent your app's data in a lightweight form. The system uses them to resolve natural language requests like "play this song" based on what is currently on screen.

App Enums wrap finite sets of options (like clothing sizes or RSVP status) so the system can present them as choices in Siri or Shortcuts.

New: View Annotations API

The new View Annotations API lets you map your SwiftUI views directly to entities. This gives Siri on-screen awareness so people can say things like "share this" or "add this to my list" referring to whatever is currently visible.

App Intents Testing Framework

You can now validate your Siri, Shortcuts, and Spotlight integrations through real system pathways without UI automation, making it easier to catch regressions early.

---

Image Playground

Image Playground has been rebuilt with a new generative model running on PCC. It now supports photorealistic output, landscape and portrait dimensions, and natural language or touch-based refinement.

You surface it in your app through the ImagePlayground framework:

// Present the system sheet from SwiftUI
// Specify a text description, optional source image, and style
// The system handles all user interaction and returns the final image

There is also a programmatic API if you want to generate images without showing the system UI.

---

Visual Intelligence

Visual Intelligence lets users point their camera at something and find matching content in your app. You integrate using the Visual Intelligence framework plus App Intents.

The framework sends your app the captured image or object data, your entities describe what you have, and your intents define what actions can be taken (like opening or playing the matched item) directly from the Visual Intelligence results panel.

VisionKit

VisionKit analyzes images and live camera feeds to extract structured data: text, URLs, addresses, phone numbers, barcodes, flight numbers, dates, and more.

Three main interfaces:

• DataScannerViewController: Live camera view for scanning real-world content.
• ImageAnalysisInteraction / ImageAnalysisOverlayView: Overlay on a static image with Live Text support.
• VNDocumentCameraViewController: Multi-page document scanner that outputs images by page.

From iOS 17 and macOS 14, VisionKit also supports subject extraction, letting your app lift objects out of images with backgrounds removed.

Q&A

1. Do I need an API key to use Apple Foundation Models?

No. Neither the on-device model nor PCC requires API key management. Users just need a device with Apple Intelligence enabled.

2. What is the difference between the on-device model and PCC?

The on-device model works offline with a 4K token context and no daily limit. PCC requires a network connection but offers a 32K token context, stronger reasoning, and support for extended reasoning levels.

3. How do I qualify for free PCC access?

Your app must be enrolled in the App Store Small Business Program and have fewer than 2 million total first-time App Store downloads.

4. Can I use Claude or Gemini with the Foundation Models framework?

Yes. Any model that conforms to the LanguageModel protocol can be used, including cloud-hosted models like Claude and Gemini.

5. What happens if the user has no network connection and my app is using PCC?

PCC requests will fail without a network. You should catch the error and retry using the on-device model as a fallback.

6. Do I have to rewrite my session logic when switching between on-device and PCC?

No. Both SystemLanguageModel and PrivateCloudComputeLanguageModel conform to the same LanguageModel protocol, so you only change the model passed to LanguageModelSession.

7. How do I test usage quota scenarios during development?

In Xcode, go to Product > Scheme > Edit Scheme > Run > Options and select a simulated quota state from the "Simulated Apple Foundation Models Availability" dropdown.

8. Do I need to write specific phrases for Siri to understand my app's actions?

No. By adopting App Intents schemas, Siri understands your actions through its language model training. You do not define phrases, and your intents automatically benefit from future Siri improvements.

9. What is the View Annotations API used for?

It maps your SwiftUI views to App Entities so Siri knows what content is currently on screen. This lets users reference visible content conversationally without having to name it explicitly.

10. Is extended reasoning available on the on-device model?

No. Reasoning levels (light, moderate, deep) are only available through the PCC server-side model.

My SaaS

Acluebox

Build modular and reusable system prompts with my SaaS,

Acluebox

. Also, free prompt template generators there.

References

• What’s new in Apple Intelligence

• Adding server-side intelligence with Private Cloud Compute

• Getting started with the App Intents framework

• Image Playground

• VisionKit

• Visual Intelligence