Embracing Extended Reality (XR): diving deep into Spatial Design for visionOS

Spatial Design for Extended Reality (XR) is not just the future, but the present.

As an iOS developer, pioneering this new frontier is an opportunity to take your skills to the next level and create experiences that captivate and astonish users; you can start with confidence and creativity by blending your iOS expertise with Spatial Design.

Achieving this balance requires a profound understanding of the Spatial Design domain, a new canvas where the laws of physics can be bent to create exceptional memories.

However, what is Spatial Design, and what are its origins?

A definition for XR Spatial Design

According to Tschumi (Tschumi, Bernard. Architecture and Disjunction. UK: MIT Press, 1996. 96), Space goes beyond static forms; it is connected to dynamic events and movements in and around architecture. Understanding Architecture requires recognizing the importance of human activities, interactions, and emotional incidents within a space.

Space is not empty but dynamic human content, playing a vital role in unfolding life. Architecture, Urbanism, Interior Design, and Extended Reality (XR) are separate fields specializing in designing and organizing spaces in various contexts. However, they all converge around the central theme of Space.

Architecture is basically a container of something. I hope they will enjoy not so much the teacup, but the tea.

Yoshio Taniguchi, Coles, J., House, N. (2007). "The Fundamentals of Interior Architecture"

In practical terms, Architecture is undoubtedly a discipline with a scope that revolves around the design and construction of physical buildings and structures, spanning diverse categories such as residential, commercial, institutional, and more. Architects use tangible materials, construction methodologies, and real-world site conditions to realize functional, safe, and aesthetically pleasing edifices.

Simplifying but not diminishing, Urbanism (Urban Design) is a field dedicated to transforming urban areas, encompassing their physical and functional aspects. This practice, known as urban design or city planning, revolves around deliberately shaping streets, public spaces, transportation networks, and infrastructure within urban environments. Urban designers engage with the existing urban fabric, landscapes, and infrastructure to create interconnected, sustainable, and harmonious cityscapes.

Like every urban process, industrialisation and suburbanisation in Detroit was facilitated by the street network, an interface that connects every building to a plot of land, to a street, to other buildings, other blocks and streets proceeding hand in hand with railways, freeways, commercial exchanges, corporate enterprises, municipal taxes, regulations, services and the ideologies that motivate the spatial distribution of resources and people. The argument that we develop through the Detroit study is that while capitalism drove the city, it was the infrastructural ecology of its streets that provided the foundation for changes to occur throughout the phases of Fordist growth and post-Fordist decline.

"Engaged Urbanism: Cities and Methodologies" (2016). Regno Unito: Bloomsbury Publishing.

While Spatial Design is integral to their work, urban designers focus on organizing and enriching the public realm, including streetscapes, plazas, parks, and transportation systems, with less emphasis on individual building design.

Interiors (Interior Design) is a specialized domain that centers on meticulously organizing and embellishing interior spaces within pre-existing structures, encompassing various settings such as homes, offices, hotels, and retail spaces. Interior designers are chiefly concerned with the meticulous selection of materials, colors, furniture, fixtures, and decorative elements. The overarching goal of crafting functional interiors is imbued with comfort and aesthetic allure. While interior designers may encounter local building and safety codes, they generally enjoy greater design flexibility than architects, allowing them to bring a nuanced focus on interior spaces’ layout and aesthetics.

Under the Extended Reality (XR) umbrella, Spatial Design is a new multifaceted discipline focused on creating immersive digital environments that blur the boundaries between the physical and virtual worlds. Often resembling the structure of interactive storytelling, this domain transcends conventional notions of Spatial Design to forge innovative and boundary-pushing journeys. (Hillmann, C. (2021). UX for XR: User Experience Design and Strategies for Immersive Technologies. Singapore: Apress. 10)

Rather than Architecture, it operates as a meta-architecture, weaving the fabric of virtual and mixed realities and crafting interactive experiences beyond the constraints of physical structures. Objects such as walls, chairs, and squares exhibit non-Euclidean properties, akin to the concept famously illustrated in The Matrix, where everything can be altered because there is no spoon or, instead, no absolute reality.

Design Disciplines

XR combines architecture, content creation, and technology to craft spaces that bend reality, seamlessly integrate with information design, and are enriched by narrative and hyper-content. Accessing an XR spatial zone can be equated to entering a liminal daydreaming room.

Spatial Design introduces several new dimensions to development:

• Information Design. Enhances clarity and user-friendliness of digital information within virtual space.
• Wayfinding. Offers cues for smooth navigation in Extended Reality (XR), ensuring users can easily find their way.
• Hyper-content. Integrates interactive elements such as 3D models and audio to boost user engagement within the virtual environment.
• Physical World Integration. Seamlessly blends digital content with the real world, creating a convincing and immersive experience in augmented and mixed reality scenarios.

Traditionally, graphic artists artfully manipulated typography, color, and layouts within the confines of a 2D dimension. Technological advancements breathed new life into this space, introducing innovations like parallax scrolling and shadow effects, injecting depth and motion into the once-static designs. Navigating this 2D environment demanded a keen understanding of visual hierarchy, usability, and the challenges posed by limited-screen real estate.

User experience in this realm revolved around screens, clicks, and gestures, with the iPhone launch in 2007 marking a milestone in crafting seamless interfaces for effortless navigation.

However, the advent of augmented and virtual reality has ushered in a dynamic world of 3D, representing a paradigm shift for creative minds. Users can now immerse themselves in captivating environments, interact with lifelike objects, and traverse spatial depths. Nevertheless, this shift brings forth a peculiar human ergonomy bottleneck: Extended Reality (XR) obliges users to a first-person perspective view, challenging conventional notions of interaction design.

The New Extended Reality (XR) Challenges

In the traditional user experience model, users engage seamlessly with iOS, relying on human sensory-motor interaction with digital screens, aiming to achieve specific outcomes that fulfill their needs; metaphorically, the user is like the archer but not the arrow itself.

Like Lewis Carroll’s Alice in Through the Looking-Glass, the first-person view moves the user on the screen, accessing a non-space, a non-setting, or an improvised theater without spectators, where the user becomes an unwitting actor surrounded by phantasmal elements.

In immersive environments, users seamlessly transition into the central character, storyteller, and protagonist. This first-person vantage point not only grants users the power to mold the narrative from their unique viewpoint intricately but also cultivates experiences that are profoundly personalized and inherently engaging.

However, this embrace of the first-person perspective has its inevitable consequences. As the renowned Carl Jung astutely remarked, “There is no light without shadow and no psychic wholeness without imperfection”.

This profound insight, sourced from C.G. Jung's psychological anthology, Reflections: A New Anthology of His Writings 1905-1961, gains poignant relevance in the context of Spatial Design. Much like Jung’s philosophical reflections, the immersive realms of Augmented and Virtual Realities introduce a shadow of their own: Cybersickness.

When traveling in a vehicle, passengers can encounter a range of sickness symptoms, from discomfort to nausea through dizziness or vomiting and more (Reason and Brand 1975), which is commonly referred to as car sickness, air sickness, sea sickness, or, generally, vehicle sickness. Cybersickness or also called Virtual Reality Induced Sickness Effects (VRISE) and simulator sickness are reported to produce similar sickness effects as motion sickness (Kemeny 2014; Mazloumi Gavgani et al. 2018). Even if the situations that cause it are slightly different, the underlying mech- anisms may be explained the same way. Cybersickness is a phenomenon involving nausea and discomfort that can last for hours after experimenting VR applications, linked to the discrepancies of perceived motion between real and virtual worlds.

Kemeny, Andras & Chardonnet, Jean-Remy & Colombet, Florent. (2020). "Getting Rid of Cybersickness: In Virtual Reality, Augmented Reality, and Simulators"

This phenomenon, stemming from integrating XR devices with human eyes, adds a nuanced layer to our understanding of the delicate equilibrium between technological innovation and its potential ramifications on human well-being.

Cybersickness is a motion sickness experienced in Extended Reality (XR); it causes physical discomfort like Nausea, dizziness, sweating, and vomiting. It is a significant deterrent for adopting users, as it can lead to a hostile and aversive experience. Dramatically, Cybersickness, together with Fatigue and Emotional Triggers, are a significant concern and a potential turnoff for users for several reasons:

1. Limited Duration. Users may be reluctant to keep engaging with applications or content for extended periods due to Cybersickness; this can limit the potential for various use cases, such as training, education, and entertainment.
2. Ineffective Content Delivery. Even worse than Limited Duration, content creators must know Cybersickness and design their experiences to minimize its effects. Failing to do so can result in ineffective content and failing to deliver the desired message or experience.
3. Exclusion. Cybersickness affects individuals differently. Some people may be more prone to it than others. As a result, experiences may exclude users susceptible to motion sickness, leading to a loss of potential audience.
4. Adoption Bias. Cybersickness is a barrier to the widespread adoption of technologies. Potential users who hear about or experience motion sickness may hesitate to invest in XR devices or content.
5. Hazard! Physical locomotion in immersive apps can cause user fatigue and reduce users’ motor quality, causing instability and leading them to fall to the ground or collide with furniture or walls. Labyrinthitis and Vection phenomena can happen so that users who are disoriented or dizzy while using AR/VR may accidentally run into walls or objects, causing injuries.
6. Psychological Risk! There is a risk of Phobic Response, where virtual or Extended Reality (XR) experiences can trigger or worsen phobias in individuals. Additionally, there is an Emotional Distress Risk, acknowledging the potential for heightened emotional responses, such as panic, anxiety, or distress, when exposed to specific content. Furthermore, there is a Trauma-Inducing Content Risk, recognizing that exposure to intense or traumatic virtual experiences may lead to psychological distress similar to trauma.

Remember that, just like the bad user experience of iOS apps, Cybersickness is an induced discomfort. Ignoring is an absolute killing of your app or game.

Biologically, the phenomenon is ignited due to conflicting signals in our brains. This contradiction between what the eyes see and what the body feels can lead to Vision-Vestibular discordance and, subsequently, induce feelings of discomfort and sickness from the brain to prevent danger due to the instinct to prevent intoxication causing vomiting in the user.

[…] the nausea and malaise, may occur because they contribute to this process of aversive conditioning. A common symptom of motion sickness is unusual sensitivity to repulsive sights or odors, suggesting sensitization of previous aversive conditioning. The marked depression sometimes experienced may be part of the same phenomenon, but it might also be a precautionary adaptation in view of the excitatory effects sometimes produced by ingested toxins. Finally, one may note that other types of anomalous vomiting, such as that consequent on a head injury, raised intra-ocular pressure, or severe pain, may have analogous explanations: these conditions may activate the same or other toxin early warning system.

Treisman, Michel. “Motion Sickness: An Evolutionary Hypothesis.” Science, vol. 197, no. 4302, 1977, pp. 493–95

To fight these adverse effects, we should quantify and, if feasible, predict them to minimize their impact. Delving into more advanced concepts of Cybersickness, Neuroscience examines human cognition and consciousness. Let's cover the following senses: the sense of Self-location, the sense of Agency, the Sight sense, the Vestibular sense, and Proprioception.

A primary benefit of both VR and AR/MR is that they allow users to have virtual embodiments, which enhances both ease of user interaction and presence, mainly by providing an anchor for visuomotor tasks (Borrego et al., 2019). They do so in different ways; thus, one must carefully determine which form of embodiment meets the needs of a given XR application (Matamala-Gomez et al., 2019). Generally, three specific subcomponents can create a sense of image: self-location, a sense of agency, and a sense of body ownership (Kilteni et al., 2012).

Sense of location (i.e., a sense of residing inside a body) can be fostered by providing a first-person visual perspective.

Sense of agency (i.e., a sense of being able to initiate and control actions) is fostered when users can interact with and control their movements in active XR scenarios […]

Gavriel Salvendy and Waldemar Karwowski (2021). "Handbook of Human Factors and Ergonomics". Wiley

The Self-location sense pertains to the perception of being within a particular body or entity in the immersive environment; achieving this sensation is facilitated by providing users with a first-person visual perspective. This perspective, which initially forms a 3D space that captures depth and dimension, is the foundation for their virtual embodiment.

The sense of Agency is closely linked to users’ ability to initiate and control actions within the XR scenario. This feeling of control over one’s actions is particularly pronounced when users can actively interact with and direct their movements, especially in pursuit of specific goals. The potential for goal attainment further reinforces the sense of agency, contributing to an overall sense of immersion and engagement in the XR experience.

Proprioception is the body’s remarkable ability to sense the position and movement of its limbs in space, providing us with an innate understanding of where our body parts are, even when we cannot see them. This sense becomes crucial when engaging in activities like walking, dancing, or playing sports, where precise control and coordination of our limbs are essential.

On the other hand, the Vestibular sense, residing within our inner ears, acts as a silent guardian of balance and spatial orientation. It continuously feeds us information about the position and movement of our head, allowing us to maintain equilibrium and navigate our surroundings effectively. This sense is especially vital for activities that involve action through three-dimensional space, such as walking on uneven terrain or piloting a spacecraft.

While Cybersickness is a risk, Fatigue is a health condition that can arise with head-mouthed displays (HMD). Suppose the immersive experience generates visual discrepancies between Vergence (the movement of the eyes to focus on an object) and Eye Accommodation (adjusting the lens to focus on that object). When we gaze at an item, our eyes converge (move towards each other) to focus on it, and our lenses change (accommodate) to bring the thing into sharp focus. This natural synchronization between Vergence and Eye Accommodation might be broken by errors in XR settings, where virtual objects are projected on displays fixed at a particular distance.

The human visual system is deceived into viewing augmented or virtual things as if they were physically there at the set distance of the screen, resulting in a disparity. While our eyes converge as if the item were there at that distance, our lenses may not need to adjust similarly since the virtual object is not physically present. This mismatch, known as the vergence-accommodation conflict, can cause visual discomfort and tiredness since our eyes and lenses are not operating together as they should. Additionally, Target Fixation contributes to cognitive load and fatigue in immersive environments. This dissonance can cause pain, eyestrain, and mental weariness, mainly when used for an extended period. Reading in virtual settings increases cognitive strain and Target Fixation problems. The constant screen distance clashes with the dynamic nature of reading, in which the eyes must travel lines of text at various depths. Users may mistakenly focus on a particular location, interfering with the normal flow of eye movements essential for good reading.

Conclusion

The transformative role of Spatial Design for Extended Reality (XR) in shaping immersive digital environments underscores the multidisciplinary nature of XR Spatial Design, converging Architecture, Content Creation, and technology to redefine traditional notions of spatial experiences. The challenges posed by XR technologies, particularly Cybersickness and Fatigue, are highlighted as significant considerations for developers and users alike.

The new paradigm shift from conventional 2D User Experiences to the first-person perspective in XR necessitates a deep understanding of user spatial models. Addressing visual system conflicts and balancing innovation and user comfort is emphasized. While the provided text lacks a specific conclusion section, it anticipates summarizing key takeaways and insights into the future of Spatial Design for Extended Reality. Developers should be encouraged to navigate the complexities of XR spatial design, focusing on user experience and technological advancements.

Having absorbed these concepts, we now know the delicate balance between innovation and discomfort, joy and pain, light and shadow. With our heightened scientific awareness, how can we safely model the spatial layout in visionOS?

Now, to start delving into the new user spatial model and to understand the unique context better and more deeply, the next step is to analyze the phenomenon of the relationships between space and human Neuroscience.