I tried image analysis with the multimodal capabilities of Foundation Models

The Foundation Models framework has long been capable of text generation. However, the use case of "inputting an image for analysis" — commonly used with cloud LLMs — was not supported.

At WWDC26, multimodal capabilities were newly added to Foundation Models, making it possible to combine images with prompts. Curious about what kind of image analysis could be performed, I decided to try it out.

This article introduces the steps to perform image analysis using the multimodal features of Foundation Models. I hope it serves as a useful reference for those who want to try similar experiments.

Test Environment

• MacBook Pro (Apple M2 Pro)
• macOS Tahoe 26.4.1
• Xcode 27.0 Beta
• iPhone 17 Pro Simulator (iOS 27.0 Beta)
• iPhone 16e physical device (iOS 27.0 Beta)

About the Multimodal Features of Foundation Models

The Foundation Models framework is a framework that enables on-device inference on devices equipped with Apple Intelligence, which was introduced at WWDC25. Recently, Reruossa introduced a method using Foundation Models to replace diary content with emoji at "try! Swift Tokyo 2026."

https://dev.classmethod.jp/articles/please-save-genmoji/

In addition to text generation, it also supports multimodal prompts that include images. The main use cases available for image analysis include the following:

• Caption generation describing the content of an image
• Identification of objects shown in an image
• Answering questions about images (Visual Q&A)

However, operation requires a device that supports Apple Intelligence. Please refer to the Apple official page for supported devices.

Implementation Steps

Step 1: Project Setup

Create a new iOS project in Xcode and use the FoundationModels framework. No additional SPM dependencies are required, as it is available as a system framework.

No special settings are required in Info.plist, but you need to use a device with Apple Intelligence enabled.

First, add a simple screen that executes a process and displays the result as text when a button is tapped to run the sample code. It is assumed that the processing described below will be added to the action1() section.

import SwiftUI
import FoundationModels

struct ContentView: View {
    @State private var text: String = ""

    var body: some View {
        VStack {
            Text(text)
            Button("Run", action: action1)
        }
    }

    func action1() {
        // Add Foundation Models processing here
    }
}

Also, don't forget to add the target image to .xcassets. Here, we use a photo of the Wada family's beloved dog, "Marony."

It was taken during a walk in the park, and it's a photo of me wearing a green T-shirt while holding Marony. I wonder what kind of response will be generated when this photo is analyzed.

Step 2: Creating a Session

Obtain the device's default model using SystemLanguageModel.default. Make sure to check isAvailable before using it. All subsequent code will be added inside action1().

// Check if the device supports Apple Intelligence
guard SystemLanguageModel.default.isAvailable else {
    text = "Apple Intelligence is not available"
    return
}
let session = LanguageModelSession()

Step 3: Building a Prompt with an Image

Send a request using the session created in Step 2. Pass text and Attachment together using result builder syntax. Result builder syntax is a Swift notation that allows you to compose multiple elements into a single prompt simply by listing them inside a closure.

In the current beta, only two types of initializers are implemented for Attachment: CGImage and file URL. When using UIImage, convert it using the .cgImage property before passing it.

Task {
    // Image to be analyzed
    let uiImage = UIImage(named: "SampleImage")

    // Convert UIImage → CGImage (UIImage cannot be passed directly to Attachment)
    guard let cgImage = uiImage?.cgImage else {
        text = "Failed to load image"
        return
    }

    do {
        let response = try await session.respond {
            "What is shown in this image? Please describe it in Japanese."
            Attachment(cgImage)
        }
        text = response.content
        print(response.content)
    } catch {
        text = "Error: \(error.localizedDescription)"
        print("Error: \(error)\n\(String(reflecting: error))")
    }
}

The following analysis results were returned. To check for variation in output, the same prompt was run 4 times. The number of seconds in parentheses is the processing time measured as the difference in Date() before and after execution, and there was a tendency for longer outputs to take more time.

Even with the same image and prompt, the expression changes every time, and there are instances where the breed is identified as "Chihuahua" and instances where it is not. This is typical probabilistic behavior of LLMs, and it was confirmed that on-device models exhibit the same behavior.

struct ContentView: View {
    @State private var text: String = ""

    var body: some View {
        VStack {
            Text(text)
            Button("Run", action: action2)
        }
    }

    func action1() {
        // Check if the device supports Apple Intelligence
        guard SystemLanguageModel.default.isAvailable else {
            text = "Apple Intelligence is not available"
            return
        }
        let session = LanguageModelSession()

        Task {
            // Image to be analyzed
            let uiImage = UIImage(named: "SampleImage")

            // Convert UIImage → CGImage (UIImage cannot be passed directly to Attachment)
            guard let cgImage = uiImage?.cgImage else {
                text = "Failed to load image"
                return
            }

            do {
                let response = try await session.respond {
                    "What is shown in this image? Please describe it in Japanese."
                    Attachment(cgImage)
                }
                text = response.content
                print(response.content)
            } catch {
                text = "Error: \(error.localizedDescription)"
                print("Error: \(error)\n\(String(reflecting: error))")
            }
        }
    }
}

Step 4: Analysis Using Structured Output

By attaching the @Generable macro to a Swift struct or enum, you can receive the model's output as an instance of that type. The framework converts the type information into a JSON schema and passes it to the model.

The @Guide macro is used to convey the meaning of a property to the model in natural language, and it is not required. If property names are sufficiently clear, the model may be able to understand the intent on its own. However, it is useful when you want to improve output quality or control the range of generated values.

The description of @Guide is written in English, following the official documentation samples. This is because English is considered to convey the intent more accurately as instructions to the model.

@Generable
struct ImageAnalysisResult {
    @Guide(description: "A description of the image content")
    var description: String
    @Guide(description: "A list of detected objects in the image")
    var detectedObjects: [String]
    @Guide(description: "The dominant colors visible in the image")
    var dominantColors: [String]
}

Note that @Generable type information consumes the context window. The more properties there are and the longer the @Guide descriptions, the greater the consumption, so it is effective to omit unnecessary properties and keep property names concise.

Define ImageAnalysisResult at the top level of the file (outside ContentView).

The results of analysis using this ImageAnalysisResult are as follows. Add action2() to ContentView and switch the button action from action1 to action2 to verify.

func action2() {
    // Check if the device supports Apple Intelligence
    guard SystemLanguageModel.default.isAvailable else {
        text = "Apple Intelligence is not available"
        return
    }
    let session = LanguageModelSession()

    Task {
        // Image to be analyzed
        let uiImage = UIImage(named: "SampleImage")

        // Convert UIImage → CGImage (UIImage cannot be passed directly to Attachment)
        guard let cgImage = uiImage?.cgImage else {
            text = "Failed to load image"
            return
        }

        do {
            let response = try await session.respond(
                generating: ImageAnalysisResult.self
            ) {
                "Please analyze this image. Please describe it in Japanese."
                Attachment(cgImage)
            }
            print(response.content.description)
            print(response.content.detectedObjects)
            print(response.content.dominantColors)
        } catch {
            text = "Error: \(error.localizedDescription)"
            print("Error: \(error)\n\(String(reflecting: error))")
        }
    }
}

The following analysis results were obtained.

A person holding a Chihuahua dog.
["dog", "human"]
["black", "brown", "green"]

Verifying Operation

In the current beta, image analysis does not work in the simulator, so verify on a physical device (see Troubleshooting for details).

Confirm in advance that Apple Intelligence is enabled on the physical device.

1. Settings app → "Apple Intelligence & Siri" → Turn on "Apple Intelligence"
2. Confirm that the language and region is set to a supported language such as English (US)
3. Wait for the model download to complete

Once the above preparations are complete, tapping the button will return a response after a few seconds. Since processing is done on-device, no communication to external networks occurs.

Notes

Input Image Size

Since the framework automatically scales and converts colors before sending to the model, no prior conversion is necessary. However, the larger the image, the more tokens are consumed, so be careful with large images from the perspective of response speed and context window.

Japanese Prompts

Writing prompts in Japanese works, but the language of the response depends on the instructions in the prompt. If you want responses in Japanese, it is best to explicitly state "Please answer in Japanese."

Troubleshooting

ModelManagerError 1001 Occurs

When running on the iOS simulator, the following error occurred.

Error Domain=FoundationModels.LanguageModelError Code=-1
  └─ ModelManagerServices.ModelManagerError Code=1001

This occurs when the Vision model component does not exist. Since SystemLanguageModel.default.isAvailable only checks the readiness state of the text generation model, this error can still occur with Vision features even if this check passes.

In the current beta, Vision features including image analysis do not work in the simulator. This is resolved by testing on a physical device.

Summary

By using the multimodal features of Foundation Models, on-device image analysis could be implemented with a simple API. Since no server upload is required, I feel it can be utilized for privacy-conscious app development.

On the other hand, there are constraints in that an Apple Intelligence-compatible device and a beta version of Xcode are required, and at this point, setting up the development environment takes some effort. After the official release, the number of supported devices is expected to expand, so I look forward to what lies ahead.

I hope this serves as a useful reference for those who want to try out the multimodal API.

References

• Analyzing images with multimodal prompting | Apple Developer Documentation
• Foundation Models | Apple Developer Documentation

Job Listings: Classmethod is Hiring iOS Engineers

The Starbucks Digital Technology Division is looking for engineers who can develop iOS apps. We are waiting for applications from those who want to work with us while sharing thoughts on new Xcode and iOS features in the misc-ios channel!

https://careers.classmethod.jp/requirements/sbj-nativeapp-ios/

We are also hiring iOS/Android engineers in other areas. Let's talk about mobile app development together!

https://careers.classmethod.jp/requirements/category/development/