Revolutionizing Surgery and Medical Treatments with visionOS

This article has been crafted as the theoretical companion to the Speculative Spatial Design Canvas, providing an in-depth exploration of the principles and concepts that underpin the canvas’s practical application. It aims to enrich your understanding and enhance your use of the canvas, ensuring you can apply it more effectively to create innovative and user-friendly spatial experiences.

One of the oldest healthcare documents in history begins with the words:

I swear by Apollo the Physician and by Asclepius and by Health the god Hygieia and Panacea and by all the gods as well as goddesses, making them judges witnesses, to bring the following oath and written covenant to fulfillment, by my power and my judgment;…

Miles, S. H. (2005). The Hippocratic Oath and the Ethics of Medicine. Regno Unito: Oxford University Press, USA.

The Hippocratic Oath is attributed to Hippocrates, often called the Father of Medicine, in ancient Greece around the 5th century BCE. As developers embark on creating healthcare applications for Apple Vision Pro, the insight of the Hippocratic Oath resonates more than ever: embrace ethical responsibility.

For decades, medical institutions and developers have used Apple’s products to enhance patient care, streamline operations, and push the boundaries of medical research. For example, iPads have been used to reduce NICU stays, enable newborns to reunite with their families sooner and leverage iPhones for quicker nurse responses and better medication management. What is next?

In February 2024, after the first release of Apple Vision Pro, Stanford Medicine integrated mixed reality into surgical practice, using the device to enhance real-time data visualization during procedures. This innovation aligns with Stanford Medicine’s commitment to combining digital technology with patient-centered care in which surgeons can interact with virtual monitors directly within the operating room, streamlining access to critical data and improving procedural efficiency. Indeed, Dr Alexander Perino, a cardiac electrophysiologist, successfully used this technology to perform ablation for atrial fibrillation, highlighting its potential to transform surgical practices.

Erin Laviola discusses the transformative potential of Apple Vision Pro in healthcare, highlighting its capacity to improve operational efficiencies and patient care in the article Spatial Computing: What Do Healthcare IT Leaders Need to Know?.

Medical Education benefits from immersive simulations using tools like the Apple Vision Pro, which offer realistic training environments for practicing procedures and experiencing critical situations. Incorporating emotional elements enhances learning by improving retention and understanding.

In Mental and Behavioral Health, AI-driven platforms like Xaia on the Apple Vision Pro provide personalized support in virtual settings, particularly in underserved areas. These platforms also monitor patient well-being and alert therapists to potential risks, augmenting traditional therapy.

Advanced Imaging is enhanced by Spatial Computing, offering detailed 3D anatomical views for better diagnosis and surgical planning. Clinicians can interact with all necessary data virtually, improving focus and efficiency. Epic’s EHR app for the Apple Vision Pro further allows interaction with patient data in a virtual environment, with future potential for mixed reality. For healthcare organizations, Dr Andrew Gostine advises carefully selecting spatial computing use cases to avoid costly mistakes and advocates for a strategic, long-term approach to gradual adoption.

Before exploring spatial computing’s role in healthcare fields, let us discuss the most surprising and perhaps intentional innovation in the context of Surgery and Medical Treatments, which Apple introduced with Apple Vision Pro.

By default, visionOS users do not need physical controllers to interact with the system—only their hands and eyes. In the operating room, surgeons cannot handle markers, flip through books, or unfold MRI cells over the patient. Apple Vision Pro changes this, allowing surgeons to interact with previously off-limits and now endless resources; this newfound freedom might seem obvious to those outside the medical field, but it is revolutionary for surgeons.

As of August 2024, here are some of the available visionOS apps focusing on enhancing medical and health-related experiences through spatial computing:

1. Medical Education and Training

• INSIGHT HEART: Immersive exploration of the human heart for educational experiences.
• Fundamental Surgery: Immersive surgical training simulations.
• Osso Health: Procedural education through advanced simulations.
• Complete HeartX: Realistic 3D renderings of the human heart for education.

2. Surgical Planning and Visualization

• SurgicalAR Vision: Augmented reality for surgical planning and execution.
• Holoeyes Body: 3D visualizations aiding diagnostics and treatment planning.
• OrthoKit: Tools for orthodontists to enhance patient outcomes through advanced visualizations.

3. Patient Care and Monitoring

• Visage Ease VP: Enhancing visualization and improving patient care.
• Monitor My BP: Tracking blood pressure with spatial capabilities.
• uMetaCareLink: Streamlining patient care and clinical workflows.

4. Medical and Educational Visualization

• Cinematic Reality: Cinematic, immersive medical visuals for educational purposes.
• Epic Spectacles: Exploring spatial computing in visually stunning formats.

To truly understand how Apple Vision Pro fits into this landscape, we must go deeper into this world and see how it positions itself as a powerful tool.

The Promise of Medical Spatial Computing

Before the advent of Spatial Computing, medical technology advancements primarily focused on enhancing imaging techniques, surgical tools, and electronic health records (EHRs).

The journey began in the pre-3D imaging era (before the late 1970s), where surgical planning relied heavily on physical examinations and essential 2D X-rays. This era posed substantial challenges due to the absence of detailed imaging, leading to a higher risk of exploratory surgeries and increased cognitive load on surgeons, who had to reconstruct anatomical structures with minimal data mentally.

The advent of 3D imaging technologies like CT and MRI scans marked a significant leap forward by providing cross-sectional views of the human body, which allowed for more precise planning and better surgical outcomes. However, the limitation of interacting with 3D images on 2D screens persisted, requiring surgeons to exert considerable cognitive effort to interpret spatial relationships and potentially leading to misinterpretations.

The study titled Spatial Computing for Preoperative Planning and Postoperative Evaluation of Single-Position Lateral Approaches in Spinal Revision Surgery, published in the Journal of Craniovertebral Junction & Spine, provides an in-depth analysis of how advanced technology was applied in complex spinal revision surgery. The research highlights significant benefits in improving the accuracy of preoperative planning, precision during surgery, and thoroughness in postoperative evaluations. This groundbreaking work contributes to the growing evidence supporting the integration of advanced technologies in neurosurgery.

The authors described how the technology implementation integrates CT and MRI scan data, creating a detailed, interactive 4D patient anatomy model. This model allowed for an immersive visualization of spinal structures, enabling the surgical team to assess the patient’s specific anatomical challenges with exceptional clarity following:

1. Enhanced Preoperative Planning:
   1. Detailed Anatomical Assessment: The 3D examination allowed the team to identify issues like pseudoarthrosis and cord compression.
   2. Surgical Trajectory Planning: Surgeons could map out precise surgical approaches, considering critical structures.
   3. Customized Tools: Preoperative sizing of grafts and implants reduced operative time and risks.
2. Intraoperative Advantages:
   1. Real-time Visualization: Live 3D views during surgery aided in accurate implant placement.
   2. Minimally Invasive Techniques: The technology supported less invasive methods, leading to quicker patient recovery.
3. Postoperative Evaluation:
   1. Outcome Assessment: Comparing pre- and postoperative spine configurations ensured surgical success.
   2. Enhanced Follow-Up: Detailed monitoring of recovery helped identify potential issues early.
4. Long-term Surgical Innovation:
   1. Standardization: The technology could standardize complex spinal surgeries and enhance training.
   2. Future Research: Further studies are needed to validate this approach in spinal surgery and develop widespread protocols.

In their article, titled Virtual Reality in Neurosurgery: Beyond Neurosurgical Planning, Mishra et al. (2022) delve into the expansive role of virtual reality (VR) and augmented reality (AR) in neurosurgery. While these technologies are well-known for their contributions to surgical planning, the authors explore their broader applications, including complex vascular interventions, spine deformity correction, and neurosurgeons and ancillary staff training. Furthermore, the article highlights the potential of VR and AR in improving patient comfort during procedures, enhancing rehabilitation, and assisting with pain management.

The study emphasizes the significant potential of these technologies to revolutionize neurosurgical practices, particularly in economically disadvantaged settings, and calls for further research to fully realize their benefits (Mishra et al., 2022).

“Virtual-reality technology is giving surgeons the equivalent of X-ray vision, helping them to remove tumors more effectively, to minimize surgical wounds and to avoid damaging critical tissues.”

The publication The Growing Value of XR in Healthcare in the United Kingdom explores different scenarios, such as how, in brain surgery, where tumor removal and aneurysm repair demand extreme precision, these technologies offer significant advantages. Similarly, complex coordination and accuracy are essential in organ transplantation, such as heart and liver transplants, and these technologies can significantly improve outcomes. Spinal surgeries, including spinal fusion and tumor removal, require high precision to avoid nerve damage, a challenge that these tools help address.

Cardiovascular surgeries, like aortic dissections and heart valve repairs, benefit from real-time spatial data for optimized procedures. Oncological surgeries, such as pancreatic resections, pelvic exenterations, fetal surgeries, and in-utero spina bifida corrections, also gain from enhanced visualization and meticulous planning provided by these advanced technologies.

Realize Medical Inc. collected a growing body of evidence that supports the benefits of Spatial Computing planning in surgery:

• Reduced operative times.
• Reduced blood loss through more precise planning.
• Shorter hospital stays due to quicker recoveries.
• Fewer fluoroscopic images, minimizing radiation exposure.
• Improved assessment and planning for complex cases.
• Better clinical outcomes from more precise surgical planning.
• Alteration of surgical plans based on enhanced understanding of anatomy and 3D relationships.

However, how has the medical scientific community embraced this technology, and what are the current schools of thought and perspectives beyond Apple’s visionOS?

A Comprehensive Framework for Medical Extended Reality

In response to the evolving scientific landscape, the American Medical Extended Reality Association (AMXRA) and the Journal of Medical Extended Reality editorial board have created a detailed classification system for Medical Extended Reality (MXR).

This taxonomy helps provide a clear perspective and understanding of the Medical horizon in Spatial Computing: the following graph presents a comprehensive framework for understanding the role and application of Medical Extended Reality (MXR), a form of spatial computing used for medical purposes. The diagram is a hexagon with interconnected segments, illustrating different domains and relationships.

Medical simulation and education platforms are at the forefront of this transformation. For instance, Simulation in Healthcare (Simulcast) provides a rich resource of podcasts, blogs, and educational content on healthcare simulation. In contrast, the Society for Simulation in Healthcare (SSH) supports research and education in simulation-based healthcare, offering certifications and various resources. The International Nursing Association for Clinical Simulation and Learning (INACSL) focuses on advancing nursing simulation education, providing best practice guidelines and professional development tools. In Canada, Simulation Canada (SIM-one) offers courses and events tailored to healthcare simulation.

On the technological frontier, several companies are pioneering spatial computing in surgical and medical practices, featuring Medivis, Osso VR, Zebra Medical Vision and FundamentalVR.

Alongside the scientific community and over 30 years of research into spatial computing applied to medicine and surgery before 2024, there is another major topic: Robotic Surgery. How does Apple Vision Pro compare with these giants?

The article titled Robotic Surgery: A Comprehensive Review of the Literature and Current Trends by Yeisson Rivero-Moreno and colleagues offers an extensive review of robotic surgery (RS), highlighting its evolution, current applications, and potential future directions. RS, which combines medical science, robotics, and engineering, significantly advances minimally invasive surgery. The article traces the history of robotic surgery from its origins in the 1970s, influenced by NASA and DARPA research, to the development of the first surgical robots like the PUMA 560 and the da Vinci Surgical System.

Robotic surgery (RS) is an evolution of minimally invasive surgery that combines medical science, robotics, and engineering. RS has revolutionized urology, enabling surgeons to perform complex procedures with greater precision and accuracy, and many other surgical specialties, such as gynecology, general surgery, otolaryngology, cardiothoracic surgery, and neurosurgery.

Robotic surgery has transformed various medical fields, particularly urology, enabling complex procedures with enhanced precision. Beyond urology, it has seen adoption in neurosurgery, gynecology, orthopedics, cardiology, and otolaryngology, offering benefits such as reduced postoperative complications, more minor scars, and shorter recovery times. Despite its advantages, RS faces challenges related to high costs and technical limitations. The authors discuss the ongoing innovations in RS, including the integration of machine learning for surgical training and decision-making. They also review various robotic systems, such as the da Vinci, Versius, and Hugo RAS, detailing their components, uses, and differences.

Surgical planning using SRS compensates for the quantitative problem of surgical planning to some extent and allows surgeons to analyse personalised patient data quantitatively.

In their review titled Evolution of Surgical Robot Systems Enhanced by Artificial Intelligence, Liu et al. (2024) discuss the transformative impact of artificial intelligence (AI) on Surgical Robot Systems (SRS). The authors note that:

AI offers a new direction and impetus for SRS by enhancing performance in areas such as perception, navigation, surgical planning, and control strategies.

Despite the significant progress, they highlight ongoing challenges, including algorithm development, surgical data science, human-robot coordination, and trust building between humans and robots. Surgical Robot Systems (SRS) help mitigate these challenges by allowing for more precise, data-driven planning. Computer-assisted manual planning is becoming more common, especially in surgical CAD/CAM systems.

Several prominent companies are notable for their surgical robots.

• Intuitive Surgical leads with the da Vinci Surgical System, which offers precise and controlled minimally invasive procedures.
• Medtronic provides the Hugo™ Robotic Assisted Surgery system, designed to advance surgical capabilities and increase accessibility.
• Stryker features the MAKO™ Robotic-Arm Assisted Surgery system, primarily used to enhance precision and outcomes for orthopedic procedures.
• Johnson & Johnson introduces the Ottava™, a newer system to improve surgical precision and flexibility.
• With enhanced planning capabilities, Zimmer Biomet offers the ROSA™ Robotics system, tailored for orthopedic procedures such as spine and knee surgeries.
• Lastly, CMR Surgical presents the Versius®, a flexible modular system for minimally invasive surgeries across various specialties.

While surgical robots provide unparalleled precision and advanced capabilities, their high cost and immobility are significant drawbacks. Apple Vision Pro offers an affordable, portable solution for enhancing surgical support and training in various contexts.

Obviously, it is not an alternative to robots. Spatial computing is a complementary technology that does not replace surgical robots but offers unique advantages, such as providing emergency support outside the hospital, which robots currently cannot.

Nevertheless, how can we push these advancements even further? The key lies in the collaboration between medical professionals, engineers, technology developers, and service designers. How can this teamwork be enhanced?

Bridging Disciplines: The Collaborative Canvas for Surgical Innovation

The convergence of medical expertise and technological innovation is essential for unlocking new possibilities and driving an even more significant impact in healthcare. The Speculative Spatial Design Canvas perfectly illustrates how this collaborative spirit can be harnessed to drive meaningful advancements in surgery.

The Speculative Spatial Design Canvas is a comprehensive framework that guides developers in creating immersive, user-friendly spatial computing experiences across various applications, from gaming to medical fields. Grounded in data-driven principles and Neuroergonomics, the canvas addresses critical challenges such as cybersickness and fatigue. By helping developers outline spatial relationships and optimize interaction dynamics, the canvas facilitates the creation of environments that prioritize user comfort, engagement, and awareness.

Designed to be printed and sketched. By the end of the canvas, the goal is to analyze and refine ideas, document insights, prototype concepts, and communicate results to stakeholders, fostering ongoing collaboration.

A seamless third-party resource. Crafted to align with Apple’s Human Interface Guidelines and WWDC events, which offer essential guidance and best practices for designing exceptional experiences across Apple platforms.

This article is based on the Spatial App for surgical use-case three scenario, specifically dedicated to the medical and surgical field, this flow facilitates a dialogue between the surgeon and the software developer to brainstorm application ideas for Vision Pro. This experimental flow aims to be pioneering in the field of Health.

The Speculative Spatial Design Canvas is not just a tool but a conversation starter. This framework bridges the gap between two worlds: the meticulous, hands-on practice of surgery and the innovative, solution-oriented domain of digital development. This canvas facilitates a focused dialogue between surgeons and developers, ensuring that the solutions created are medically accurate and technologically feasible. Indeed, innovative tools, such as spatial surgical apps, can be developed through this structured collaboration with aims to:

1. Clear Emphasis on Collaboration
2. Detailed Procedural Breakdown
3. Surgeon-Centric Design
4. Forward-Thinking Application

The collaboration begins with the surgeon, who brings a wealth of medical knowledge and experience. In the example canvas context, the surgeon might be dealing with a complex case, such as supraorbital osteosarcoma—a type of tumor located above the eye sockets.

Procedure surgical from Supraorbital Osteotomy P113-120 Quinones-Hinojosa, A. (2016). Video Atlas of Neurosurge- ry: Video Atlas of Neurosurgery E-Book. United Kingdom: Elsevier Health Sciences.

This condition requires a delicate surgical approach, known as supraorbital osteotomy, to remove the tumor while preserving critical structures like the brain, nerves, and major blood vessels. The surgeon uses the canvas to explain the condition, sketch the surgical incision, and outline the critical steps of the procedure. This detailed input is crucial for the developer, who needs to translate this medical information into a solution that can be used in a spatial surgical app.

For example, suppose the developer identifies a particular challenge, such as the risk of bleeding or difficulty navigating around critical nerves: this might include real-time visual and auditory cues to guide the surgeon through the procedure, alerts for potential complications, or simulations that help the surgeon practice complex maneuvers before performing them on a patient. Building on the collaborative framework introduced in the Speculative Spatial Design Canvas, the potential for innovation in surgery becomes even more profound when we explore integrating mixed reality (MR) into the operating room.

The subsequent artboards, Artboard B: Mixed Reality Layout Surgery Operation and Artboard C: Mixed Reality Layout Surgery Room, extend the conversation by providing a visual and procedural guide to how mixed reality can revolutionize surgical planning and execution. These artboards further emphasize the critical collaboration between surgeons and developers, as they must work in tandem to design and implement these advanced technologies effectively.

The Canvas prompts developers to collaborate closely with surgeons by first detailing their envisioned Spatially Assisted Procedure in response to a guiding question. After this, the procedure should be broken down into a comprehensive six-step storyboard with a visual incision model. This storyboard should depict each stage of the surgical process as it unfolds in a mixed-reality setting. Upon completion, the storyboard should be reviewed by experienced surgeons to ensure it aligns with medical standards and gain their validation.

The Canvas prompts to design a Mixed Reality layout tailored for the surgical room, focusing on supporting surgeons in action. While the Apple Vision Pro is primarily designed for indoor environments like living rooms or studios, adapting it for a surgical setting requires customization. In this scenario, the layout should enable at least two surgeons to simultaneously use Apple Vision Pro, collaborating in real-time through a shared Spatial App.

Together, surgeons and developers can create a new paradigm in surgery where technology enhances human skill and precision. This would improve patient outcomes and pave the way for the next generation of surgical innovations. The Speculative Spatial Design Canvas and these artboards offer a roadmap for this journey, highlighting the importance of collaboration in achieving the full potential of mixed reality in surgery.

Conclusion

The dawn of Medical Spatial Computing heralds a transformative era in healthcare, where the boundaries between the physical and digital worlds blur to create unprecedented possibilities in medical practice. From the precision of surgical robots to the immersive capabilities of mixed reality, we are witnessing the emergence of tools that enhance the skills of healthcare professionals and reimagine the nature of patient care.

As explored in this article, the synergy between advanced technology and medical expertise is not just a fleeting trend but a revolution in how we approach surgery, education, and patient outcomes. The Speculative Spatial Design Canvas and the integration of visionOS into surgical practices are just the beginning of this journey. They exemplify the power of collaboration between surgeons and developers, showing us that the future of medicine is one where creativity and technical innovation converge to overcome even the most complex challenges.

However, the real power of this transformation extends beyond individual advancements in surgery. It touches on a broader political and ethical context, where the distribution of technological benefits is a critical concern. With their high costs and intricate infrastructure needs, surgical robots are predominantly accessible to well-funded, affluent hospitals. This creates a disparity in healthcare access, reinforcing existing inequalities between wealthy and less-resourced regions.

This raises important questions about equity and justice in healthcare. We are to embrace the potential of medical spatial computing fully. In that case, we must also address the challenge of ensuring that these innovations are not confined to the elite but are accessible to all who need them. This is not just a technological issue but a moral imperative. We must consider how to democratize access to these life-saving tools so that their benefits are felt far beyond the walls of the world’s most privileged institutions. The urgency and responsibility of this task cannot be overstated.

This is a provocative question on my part. Is it worth investing in a single Da Vinci surgical robot for $1.5 to $2.5 million when that exact cost could provide a hospital with 428 to 714 Apple Vision Pro units?

Such a comparison challenges us to rethink how we allocate resources in healthcare. Should we prioritize the most cutting-edge technology available to a few or seek a more distributed approach that can significantly benefit many? If given the choice, I would opt for both. It would be ideal to have the precision of a top-tier surgical robot and the flexibility and resilience of the Vision Pro in a hospital. It is essential to strike a balance between technological horizontality and verticality.

Even if medical app development is not your expertise, sharing this article can make a meaningful impact: let us inspire students and professionals to create spatial apps that drive significant social change together.