Exploring Apple Intelligence: Talking with Siri
Understand how Siri is evolving, being powered by Apple Intelligence.
The Siri assistant was introduced with the iPhone 4S in 2011. Its role was to streamline user interaction by executing some tasks with vocal commands. Over the years, Siri has evolved from a basic voice assistant into an integral part of the Apple ecosystem, becoming more useful and versatile. With the introduction of Apple Intelligence, Siri is receiving one of its biggest updates, enhancing its capabilities, performance, and user experience.
The new Siri, powered by Apple Intelligence
Thanks to the Natural Language Processing (NLP) capabilities embedded in Apple Intelligence Siri will better understand user queries, whether it’s varied sentence structures, idiomatic expressions, or follow-up questions, without the need for rigid, predefined commands. Additionally, to make the experience more seamless, Siri will help users understand what they are viewing on the screen with the new On-Screen Awareness capability.
To be even more effective in understanding and anticipating users' needs, Siri will also be able to have access to what Apple calls Personal Context to remember your preferences, past interactions, and routines, giving it the ability to offer personalized and proactive assistance, with the promise of maintaining maximum privacy thanks to the Private Cloud Compute architecture.
These technologies’ improvements have a huge impact on the Voice User Interface (VUI), which focuses on creating natural interactions through voice alone. Users will speak to Siri as they would to a human assistant, using everyday language and expecting intelligent, context-aware responses.
For example, instead of saying, “Remind me to reply to the message that I received from…” users can now simply say, “Remind me to reply in 10 minutes” and Siri will intuitively set a timer based on the context of what the user is doing, seeing the last message received.
An important milestone in Siri's evolution is the collaboration between Apple and OpenAI to integrate ChatGPT into the assistant. This partnership aims to enhance Siri's capabilities, allowing it to better handle tasks like analyzing photos, interpreting document contents, and explaining complex topics in greater depth.
Beyond the conversation: reshaping the user experience
While conversation is the primary way users interact with virtual assistants, Apple Intelligence goes beyond that to enhance daily interactions with devices, helping to focus on what truly matters. For instance, notifications are a major source of distraction and thanks to features like Focus Mode we can filter the amount of notifications we receive daily.
Siri now enhances Focus Mode through the new Reduce Interruptions mode, intelligently analyzing the content of notifications, prioritizing them based on context and importance creating a more balanced and efficient experience, allowing users to stay productive, informed, and in control—without feeling overwhelmed.
If you receive multiple notifications from the same app Siri is also able to condense notifications into a concise overview, summarizing the whole content, dramatically reducing the notification overload and allowing users to focus on what matters most without constant interruptions.
However, what will significantly impact the user experience of our devices will be the capability of executing action not only within a specific app but also across different apps, streamlining the process of completing complex actions in just one command.
Extend your app features to Siri
When the user makes a request to Siri, it needs to comprehend the request to execute it. To achieve this, Siri relies on Apple Intelligence and its language models that receive and process the request. These models are trained to anticipate a specific format known as a schema, which dictates the structure in which intents or functionalities must be presented to be recognizable and allow the model to reason over them. Consequently, this enables Siri to develop new capabilities.
By understanding this mechanism, we have a better understanding of how Apple Intelligence works and what developers must do to expose their app features to Siri and Apple Intelligence in the future. Apple has empowered the existing App Intents framework by introducing new macros that improve Apple Intelligence’s ability to categorize and understand your app’s data and functionality.
The biggest change when designing apps is that as developers we should create app intents for all the actions that can be performed in the app, prioritizing anyway the most common ones.
Tiago Gomes Pereira
To learn more on how to implement App Intents, check this tutorial.
To take this integration a step further and ensure Siri can fully understand and execute app-related tasks, you’ll need to define the appropriate AssistantSchema for AppIntent, AppEntity, and AppEnum. These schemas help Siri recognize and categorize the specific functions and data within your app, enabling Siri to respond with contextually relevant actions.
The responsibility of an AssistantSchemas is to validate if the schema on which the AppIntent is built matches the schema the AI models are trained on and are expecting, to then start to process and reason over the request prompted by the user.
To enable onscreen awareness, the app’s screen content must:
- Be identified with an
EntityIdentifierand exposed to Siri and Apple Intelligence using theAppIntents - Associate the identifier to the current
NSUserActivity
Conclusion
Siri’s evolution powered by Apple Intelligence is moving toward a future where the assistant will predict user needs, act seamlessly across multiple apps, and help users with increasingly complex, personal tasks. Apple’s current progress (including schema alignment, App Intents, and the integration of onscreen content awareness) shows a clear path forward, though some features and capabilities are still in beta and being refined.
This direction promises groundbreaking benefits for accessibility.
For visually impaired users, for example, the improvements in natural language comprehension, on-screen awareness, and cross-app control mean that Siri could provide a more reliable, hands-free experience; users will be able to rely on voice commands to navigate interfaces, access information, and manage daily tasks without directly interact with their screens, making technology more accessible to a wider audience.
As Siri continues to evolve, it is shaping up to be not just a tool, but an integral part of a connected ecosystem that adapts to and simplifies users' daily lives.