Exploring Apple Intelligence: Image Generation
Understand how image generation powered by Apple Intelligence is taking shape as system features.
Powered by Apple Intelligence, Image Playground is the latest tool that seamlessly translates textual and visual prompts into images. It was brought to developers with macOS 15.2 beta, iOS 18.2 beta, and iPadOS 18.2 beta, and it unlocks creative possibilities for developers and users alike.
Image Playground is integrated across various applications, enhancing functionality and user experience. One of its standout features is its incorporation into the PencilKit palette, elevating creativity with the Image Wand tool, which users can use to transform hand-drawn sketches into fully realized images.
Another exciting image generation integration comes with Genmoji, a feature that enables the generation of emojis based on natural language prompts, adding a playful and highly personalized dimension to digital communication.
Developers can now natively integrate image generation technology into their applications, offering users interactive and intuitive ways to create illustrations tailored to their needs.
To learn more, check the full list here.
Image Playground
Image Playground is an image generation app that creates unique content using textual and visual prompts sourced directly from the camera or the photo library. For users seeking inspiration, the app also offers a curated list of suggestions in its main view, enabling effortless experimentation and the possibility of choosing the illustrated or animated style of the outcome.
Apple has clearly prioritized fairness and ethical considerations in its approach to image generation, setting Image Playground apart as a thoughtful and responsible tool.
Inclusion is a core principle. When generating images involving people, the app prompts users to specify appearance settings, such as skin tone. Instead of relying solely on descriptive text, the app provides a selection of sample images representing the chosen settings, empowering users to make informed choices.
Beyond inclusivity, Image Playground addresses property rights concerns.
If users attempt to create images mimicking styles from specific franchises or brands out of textual prompts, the app ensures compliance by preventing references to copyrighted material, reinforcing its commitment to respecting intellectual property.
Once the images are generated, users can scroll among the generated options, selecting the one that best meets their needs; images can be copied, shared, or saved directly from the app. When saving the image, users can revisit the prompting information behind the generated image, this means that from the user’s image collection, it is always possible to go back and start editing the image again just by tapping on it.
Image Playground is also exposed to developers as a framework enabling the integration of image generation capability into apps by embedding components that invoke and present the Image Playground interface, available both in UIKit and SwifTUI.
In SwiftUI a set of four different methods has been released, they present a sheet with the image playground interface.
imagePlaygroundSheet(isPresented:concept:sourceImage:onCompletion:onCancellation:)imagePlaygroundSheet(isPresented:concept:sourceImageURL:onCompletion:onCancellation:)imagePlaygroundSheet(isPresented:concepts:sourceImage:onCompletion:onCancellation:)imagePlaygroundSheet(isPresented:concepts:sourceImageURL:onCompletion:onCancellation:)
The interface is enabled only if the device matches the operating system and device features requirements, which can be checked through the @Environment(\.supportsImagePlayground) property value.
In UIKit, ImagePlaygroundViewController is responsible for presenting the image generation interface and ImagePlaygroundViewController.isAvailable is the attribute to check for understanding whether the feature is available or not on the device where the app is running.
These classes will allow shaping experiences where users can create custom artwork or illustrations on the go, like in storytelling or writing apps, where it can be used to generate personalized book covers, character illustrations, or thematic images, bringing users’ narratives to life, for example.
Image Wand
The Notes app has Image Playground capabilities integrated through Image Wand, a new tool in the PencilKit palette that lets users generate images based on their own sketches by circling them.
Users can sketch directly in the app and then provide a textual description to refine the image. For those who prefer to start from scratch, rotating on blank space within the tool allows users to input textual prompts, leveraging Apple Intelligence models for image generation.
Users can provide up to six textual prompts. As they add new prompts, the oldest ones will be replaced, keeping the process dynamic and responsive.
Image Wand exclusively supports sketches made within the Notes app, this means that it does not allow users to include visual prompts from their photo library or camera.
A unique addition to Image Wand is the ability to apply specific sketch styles to the generated images. This feature, currently exclusive to Image Wand, enhances the creative possibilities by allowing users to match the output with their artistic intent, making it a standout tool for sketch-based image creation.
Unlike the standalone Image Playground, the Pencil Kit framework does not present APIs to implement support for the Image Wand tool at the moment. Furthermore, Apple has not provided clarity on whether developers will have the ability to integrate the Image Wand tool into their applications in the future.
Genmoji
Another way by which Apple leverages natural language processing to generate visual personalized content is Genmoji, the generation of customized emojis based on textual descriptions.
Unlike traditional emojis, Genmoji provides the ability to design unique emojis, such as animals, characters, or objects, that are customized through text prompts. In fact, users can describe an emotion, action, or object, and Genmoji transforms these prompts into custom, expressive emojis that can be used to communicate and convey thoughts and feelings in unique and creative ways, enriching messaging experiences.
Genmoji creation is accessible from the keyboard in apps like Messages and Notes, right from the emoji section, just by tapping on the animated icon, the user is able to start describing their own unique emoji.
Genmoji are brought to developers via NSAdaptiveImageGlyph , the API behind it, which allows these custom emojis - that are essentially image glyphs - to be treated like regular text characters, making them adaptable and compatible with existing rich text systems, and embedded within text as part of the string, rather than being separate image files. This allows them to be formatted, styled, and integrated with other text seamlessly.
To start taking advantage of Genmoji is very easy wherever rich TextViews are supported and accessible just by setting the supportsAdaptiveImageGlyph property to true.
Genmoji are stored in the adaptiveImageGlyph attribute of the NSAttributedString as an NSAdaptiveImageGlyph object.
What is leveraged is the ability of this type to work like a container that stores image data and metadata describing how to adapt those images correctly to different fonts and font attributes maintaining a square aspect ratio, as well as alternative textual description of the image contents. Processing all this information enables the adaptability to different sizes and resolutions of its content.
Apple is expanding its ecosystem with image generation capabilities showing a strong commitment to fostering digital creativity while prioritizing inclusivity, personalization, and ethical considerations.
The launch of Image Playground and Genmoji has reimagined personalized communication, They allow users to create unique, expressive visuals that seamlessly enhance their interactions. These tools enrich messaging experiences and provide developers with tools to foster deeper user engagement through customization.
The Image Wand stands out as a transformative feature, enabling users to turn sketches into polished images. However, its exclusivity to the Notes app at the moment limits its full potential. Extending this capability to developers could pave the way for groundbreaking applications in creative industries, from storytelling to design, for instance.
These advancements reflect Apple's dedication to a creative, inclusive, and responsible ecosystem, setting new benchmarks for how technology can elevate personal and collective expression.