Improvisation in Spinal Surgery Using AR (Augmented Reality), MR (Mixed Reality), and VR (Virtual Reality) ^†

Abstract

The day-by-day advancement of extended reality and its subset technologies, along with effective hardware, is increasing their utilization in various sectors like education, training, sports, and healthcare. Healthcare is one domain of concern. Considering this, the main focus of this paper is on spine surgery. In orthopedic surgery, the main uses of virtual reality (VR) are for education, preoperative planning, and intraoperative use. Yet the training imparted still lags. Orthopedic training committees in North America and Europe have endorsed the use of virtual reality for educational purposes. Spinal surgery is one of the main focuses where virtual reality (VR) is applied. In the past, open techniques and instruments that could be seen in real time were used to perform spine surgery. Significant advancements in minimally invasive spine (MIS) surgery have been made. Virtual reality (VR) has been used in preoperative contexts for spine surgery. This paper delves into the applications of augmented reality (AR), virtual reality (VR), and mixed reality (MR) in spinal surgery, emphasizing their potential in education, training, and surgical settings. Specifically, we focus on procedures like pedicle screw placement, cervical spine, and deformity correction, where AR augments surgical precision and information accessibility. The primary objective is to provide a comprehensive framework for evaluating the clinical benefits of AR–VR-enabled spinal surgery technology and propose a viable business model catering to diverse stakeholders, including patients, hospitals, research centers, and technology adopters.

1. Introduction

A lot has changed in the healthcare profession, yet many clinics are still unable to overcome the same challenges they have had for years. In addition to the high cost and difficulty of educating employees, there is also the issue of patient safety and well-being when undergoing major surgery. Some forms of treatment have a negligible impact on patient outcomes. On the other hand, Google, Microsoft, and Facebook are on the verge of completing the preparation of the gadgets which are worth around USD 3 trillion [1,2,3]. As a result, medicine will become more precise, effective, and tailored. There are numerous issues in the healthcare industry that can be resolved using modern technologies. Surgery, medical student instruction, and the treatment of post-traumatic stress disorder can all be performed using virtual reality. Medicine in North America and Europe is already following all of these trends. Prior to the integration of virtual reality into medicine, there was no such thing. As a surgeon, one must be able to see the human body clearly. Surgeons can practice virtual, augmented, and mixed reality technologies for even the most unpredictable surgeries by creating VR reconstructions of patients’ scans (CT, MRI) from 2D data. In other words, surgeons will be able to design the entire procedure in advance using virtual, augmented, and mixed reality. Patients and doctors can benefit from the use of AR/MR/VR in medicine [4,5,6].

It is important to consider patient satisfaction when evaluating healthcare services. Patients can switch doctors based on their expectations and overall satisfaction. Anxiety about the procedure itself is expected in patients who have been admitted to the hospital for surgery, and patients may experience severe perioperative stress and vulnerability [5]. Pulmonary embolism (PE) is often employed in total joint replacement surgery in the orthopedic area. As such, there is no evidence that supports the use of PE in patients who undergo spine surgery. Patients undergoing elective spinal surgery were asked to participate in this study to see how well it worked for them. Figure 1 presents a visual representation of extended reality (XR) and its subcategories. The fundamental hardware components necessary for the current consumer-oriented virtual reality experience include a computer with 3D graphics capability, a head-mounted display (HMD), and controllers equipped with position-tracking capabilities. The incorporation of haptic feedback in virtual reality (VR) to replicate sensations like touch, vibration, and motion is growing in popularity.

Virtual reality implementation in clinical care lags behind its counterparts in consumer electronics [6,7,8,9], and there is ambiguity regarding its applications. Recently, the term “computer-assisted orthopaedic surgery” (CAOS) has gained prominence in the literature. A CAOS system improves both preoperative and intraoperative settings. Immersive virtual reality (IVR) utilizes a head-mounted display (HMD) equipped with visual and auditory cues, along with controllers that provide haptic feedback and various movement capabilities.

Figure 1. XR and its subset concept [9].

Figure 1. XR and its subset concept [9].

As a result, VR strives to achieve realism through high multimodal fidelity, which includes visual, psychomotor, and cognitive capabilities. AR/MR/VR will become more prevalent in the hospital as the future of medicine becomes virtual, mixed, and augmented. It has nothing to do with fashion. It is impossible to deny or overlook the benefits of virtual reality and augmented reality (VR/MR/AR) for doctors, nurses, and patients. By incorporating VR/AR/MR technologies, hospitals may be able to save even more money on medical training. Many companies have emerged with efficient concepts and devices for healthcare [10,11,12,13,14].

Novelty

• This study presents various spinal surgery treatments through AR with substantial supportive data.
• AR, VR, and MR devices for spinal treatment are compared and presented with their efficacy.
• A rigorous survey was conducted, showing the growth of XR in healthcare regarding treatment and training.

This paper’s structure is as follows: Section 2 presents the background study, followed by Section 3, which outlines the application of AR/VR concepts in spinal surgery. Section 4 addresses the existing research voids, and Section 5 summarizes the paper and outlines future directions. Figure 2 visually presents the paper’s outline, adhering to the structured literature review (SLR) approach.

2. Background Study

Virtual reality (VR) publications in orthopedics have grown significantly since the early 1990s. Most of these studies have focused on surgical education, specifically arthroscopy education, due to the difficulty of learning arthroscopy and the complexity of the skill set required. General and urologic surgeries are two fields where this has occurred. In the early days of VR, publications were underpowered and lacked fidelity documentation. They also struggled with unambiguous, consistent outcome measurements and inconsistent reporting. The Oxford Centre for Evidence-Based Medicine established a system for grading evidence and making recommendations, drawing from 32 studies and 5 commercially available general surgery simulators. In 2010, Van Nortwick et al. noted a lack of rigor in standardized reporting in surgical simulation. They suggested prioritizing the establishment of concept, concurrent, and predictive validity [11,13,14,15]. As VR progressed, papers began to include face, content, concept, and transfer validity studies.

Table 1. Various application domains for demonstrating the utility of AR, VR, and MR.

Application Areas	Usage
Healthcare	Navigating features during surgery, providing experience (psychological) to patients, risk assessment, and medical education/training simulation.
Human Resources	Three-dimensional (3D) video recruiting; potential applicants are vetted by simulating and testing their talents, conducting meetings and webinars in VR mode to minimize the barrier of face-to-face communication, and training employees through VR simulators.
Travel Sector	Augmented tourist destinations, immersive 360-degree VR video of the destination, and 3D navigation for tourists.
Government	VR simulators to train government officials, seamless communication from remote locations; creating visual models of government projects.
Infrastructure	Flooring selection using AR applications, interior design, and 3D designs of buildings.

indicates the various application domains for demonstrating the utility of AR, VR, and MR.

2.1. Discussion on Virtual Reality

Preoperative Steps

Surgical procedures of any kind are dangerous. The physical and mental health of the patient is considered during a surgical risk assessment. To limit surgical risk, the medical staff evaluates the patient’s health (e.g., pre-existing disorders like diabetes) during the preoperative evaluation. This assessment includes a review of X-rays, CT scans, MRI studies, and other diagnostic testing. During a physical and neurological examination, the patient’s general health is checked. Physical problems (existing and unknown) could lead to surgical complications during the preoperative examination process (e.g., cardiac or breathing difficulties). The patient may consult a medical professional prior to surgery in some instances. Anatomical two-dimensional photographs are frequently used to assist with preoperative planning in spine surgery, which is a critical phase in the surgical process. This time-consuming method’s simulations and outcomes might overestimate the three-dimensional (3D) nature of spinal anatomy. An interactive 3D anatomical model in virtual reality (VR) can be freely explored and altered. Virtual reality (VR) is an emerging technology [16]. Aaron Cohen-Gado’s 3D neuroanatomy atlas demonstrates how virtual reality is gaining acceptance in anatomy education. This training method’s main advantage is its ability to improve surgical techniques while reducing the risk of making costly mistakes on live patients [17,18,19].

2.2. Discussion on Augmented Reality

It is possible to capture real-world objects in real time and then overlay virtual objects on top of the image. As a pilot, one has the opportunity to see relevant flight safety information projected into their cockpit vision using a heads-up display. Although it is expensive and new, AR has been used in spine surgery to speed up the healing process and reduce radiation and fluoroscopy exposure for both the patient and the medical staff [20].

2.2.1. Rod Bending

Wanivenhaus et al. have developed an augmented reality (AR) application for accurately bending rods during spinal surgeries. This application utilizes fiducials to register the pedicle screw heads in three-dimensional space within the Microsoft HoloLens head-mounted display (HMD), which serves as the focal point for the operator. Additionally, the operator is presented with a holographic representation of the ideal rod, serving as a visual reference for making bends.

2.2.2. Quality Control in the Operating Room

Uneri et al. opted for intraoperative quality assurance in order to carry out pedicle screw placement. Pedicle ports can be assessed in real-time using a 3D model of recognized surgical equipment, such as pedicle screws, paired with an intraoperative 3D–2D registration approach. For example, 3D imaging may simplify detecting breaches or misplacements compared to lateral and anteroposterior X-rays taken after the procedure.

2.2.3. Keyhole Spine Surgery

For keyhole spine surgery, AR can be used. For trans-vertebral anterior cervical and posterior foraminotomies, O-arm AR registration can be used. In both cases, the AR imaging model was visible in the surgical microscope image. This form of HUD has a lot of proponents because it is been employed in brain surgery. Keyhole spine surgery does not attempt full anatomic exposure to identify critical landmarks, allowing this procedure to work without the limitations of brain displacement, which can lead to registration errors [20].

2.2.4. Facet Joint Injections

Inflammation of the facet joints can cause back pain, and a facet injection is a non-invasive therapy option. An injection may be performed to determine if facet joint inflammation is the cause of a patient’s pain. There are many possible causes of facet joint discomfort, including spinal stenosis and arthritis. Facet joints connect each vertebra to the one above and below it. Injuries to these joints can cause pain in the groin, buttocks, hips, shoulders, or neck. Facet disease can be caused by aging, overuse, or injury. This illness affects the lumbar (lower) spine. Nonetheless, it has the potential for widespread application. Agten et al. [14] evaluated the precision of lumbar facet joint injections performed by radiologists using an augmented reality head-mounted display (AR HMD) like the Microsoft HoloLens, which projected a 3D spine phantom onto an actual model. They observed improved timing and needle placement compared to conventional CT-guided procedures. This is especially appealing for procedures that may require repetition, as it offers the prospect of decreased radiation exposure and procedure duration.

2.3. Discussion on Mixed Reality

Simulation training provides students with a controlled environment where they can make mistakes without endangering the patient’s well-being. As mentioned earlier, virtual reality (VR) offers visual feedback but lacks mechanical feedback. On the other hand, mixed reality (MR) provides a virtual setting that typically replicates a physical environment through haptic feedback. In the context of muscle memory training, this can be particularly beneficial for surgical education, especially in spine surgery, where the precise placement of corrective screws and percutaneous procedures is crucial. By integrating virtual and physical models, the traditional apprenticeship-based teaching model for surgical simulation can be enhanced. This allows residents to practice critical procedures and develop muscle memory before performing them on actual patients. Cadaveric dissection has several limitations, including anatomical disparities between deceased and living tissue, which can complicate replicating intricate surgical maneuvers. Additionally, there are ethical and economic concerns associated with the preservation and dissection of cadaveric tissue [21,22,23].

Our comprehensive systematic literature review (SLR) aims to demonstrate the applicability of AR, VR, and MR concepts in spinal surgery through a thorough examination of the methodologies employed by medical professionals. Our SLR offers a thorough assessment of the effectiveness of VR, AR, and MR in different stages of spinal surgery, while also highlighting areas for future research by emphasizing existing research gaps.

• Motivation

AR enables new ways to interact with the actual world and can generate experiences that are completely actually realized. However, there are limited reviews on the application of immersive technologies (AR/VR/MR) in medical practice and education. Here, in this paper, a systematic literature review (SLR) of the AR/VR/MR is presented, which helps the reader grasp concrete prerequisites to start work in this area.

2.

Research Goals

In the context of the specified research topic, our SLR attempts to discover and critically examine current studies and their conclusions on the basis of research questions and objectives, as mentioned in

Table 2. Research questions and objectives.

Number	Research Questions	Objectives
RQ-1	How and where can AR/VR be introduced in healthcare?	Healthcare is a very sensitive issue because, here, someone’s life is affected; it is important to precisely decide what sections of treatment we can use AR/VR/MR.
RQ-2	How can patients be convinced to use the AR/VR?	Prepare advantages and successful case studies for patients, so that patients do not have any issues while adopting the technology for their treatment.
RQ-3	How can doctors be convinced to use AR/VR in healthcare to treat patients?	Doctors and surgeons are not technology-driven; they also require convincing evidence of success for adopting the concept in treatments.
RQ-4	What are the devices required for patients and doctors to utilize AR/VR?	AR/VR/MR is fully device-driven technology. As uses change, the need for devices also changes. So, to adopt the concept in healthcare, it is important to know the devices that are required, along with their efficiency and error rates.
RQ-5	What will be the impact of the system’s failure?	There should be a backup plan or no effect of technology failure during treatment; otherwise, there will be a barrier to using the technology from both patients’ and doctors’ perspectives.

.

3. Utility of AR/VR Concept in the Surgery of Spine

AR is defined as the overlaying of a virtual environment on reality, providing a more realistic view of reality with computer-generated image data. Spine surgeons may find a use for image projection on real-world surroundings and wearable heads-up displays inside operating rooms. The placement of pedicle screws, the cervical spine, and deformity are promising areas where AR can offer the spine surgeon helpful intraoperative assistance. The VR concept does not directly apply to operation theater because it is completely based on an artificial environment and automation that does not instill confidence in doctors and patients [24]. AR has been more proactively utilized for the purpose of surgery because it provides augmented information to the doctor during surgery, hel** the doctor perform accurate and efficient surgery. The xvision system is the only headset that has obtained clearance from the FDA (Food and Drug Administration, USA) for spine surgery. The xvision system, as shown in Figure 3, is a headset that uses AR to transform a patient’s CT scan into a 3D visualization, aiding a spinal surgeon during their operation, where every millimeter matters.

Accurate pedicle screw placement is crucial for spine stability, especially in minimally invasive procedures. Augmented reality (AR) navigation systems, like VIPAR, have improved precision. VIPAR used O-arm imaging for successful trans-vertebral and posterior cervical surgeries, with patients remaining symptom-free for an average of 20 months. It employed a head-mounted virtual protractor to monitor needle trajectory during vertebroplasty, achieving a minimal angular variation of 0.96 degrees in phantom trials. VIPAR was also used in clinical examinations for five patients with osteoporotic vertebral fractures.

Table 3. Utility of VR, AR, and MR in the various processes of spinal surgery.

Characteristics	VR	AR	MR
Imaging data of patients, e.g., CT and MRI scans	✓	✓	✓
User (surgeon) manipulation of real and computer-augmented images	✖	✓	✓
Spinal surgery training without patients (all parts of the spine can be computer-generated and arranged completely like in a real patient)	✓	✖	✖
Live help from an expert for the surgery (receive an expert’s suggestions during the surgery without their presence)	✓	✓	✓
Pedicle screw placement in spinal surgery	✖	✓	✖
Cervical spine treatment	✓	✓	✖
Surgery of an abnormal curve in the spine (deformity)	✖	✓	✖

illustrates the application of VR, AR, and MR in various stages of spinal surgery. Augmented reality enhances interaction with the actual environment. A recent example is xvision, an image-guidance system that provides real-time detection of surgical instrument placement and overlays a virtual pathway onto the patient’s CT data [25,26,27,28,29]. Mixed reality allows for physical interaction with virtual elements. For instance, specialized goggles can project a 3D representation of the patient’s spine, enabling surgeons to practice or plan procedures by manipulating virtual objects like screws. Two commercially available options are Microsoft HoloLens and Magic Leap. HoloLens offers greater processing power, while Magic Leap is lighter as its visual processing module is worn on the hip.

4. Research Gaps

The main challenges to adopting AR–VR in spinal surgery are cost-effectiveness, effective integration, and evidence of improved surgical outcomes. This section describes these challenges in detail:

4.1. Cost-Effectiveness

The adoption of AR–VR in spinal surgery has yet to be proven as cost-effective because the gadgets required for surgery are now available on the open market. When we refer to an “open market”, we imply that the approving agency should be aware of different AR–VR gadget manufacturers. Various manufacturers have the ability to obtain their products approved by the agency, which then reduces the cost of hardware.

4.2. Effective Integration

An integrated technique like robotics-based spinal surgery—in which the surgeon provides instructions from a remote location, and the robot performs the surgical process—has been proposed as an effective means of spinal surgery. Integrating AR–VR and machine control is a major challenge, as it must be seamless and not display any lags in the process so that robots and machines can perform surgery on the surgeon’s behalf.

4.3. Evidence of Improved Surgical Outcome

There is no quantitative or qualitative data evidence of patients recovering from spinal treatment while adopting AR–VR. Surgeons who have used AR–VR technology during surgery have provided feedback, and they all expressed gratitude for its benefits. However, patients’ perspectives must also be considered to understand how they utilize the technology. To address this, conclusive data can be prepared about patients treated while adopting these technologies.

In the field of healthcare, virtual reality (VR) and augmented reality (AR) can be used to educate doctors, surgeons, students, trainees, and patients, as well as connect the surgeon to an operative module, such as robots. AR and VR can be used in conjunction with a variety of other healthcare sectors, and advancements in each of the technologies like artificial intelligence, AR, VR, and surgical robots can open up new possibilities and applications for AR in the healthcare industry.

5. Conclusions and Future Work

Presently, AR and VR show good potential in the field of spine surgery; however, these computation systems are still in their early stages. The most prominent projected consequences of AR/VR in intraoperative navigation on the financial front are the reduction in surgical treatment time, infection rates, review rates, length of stay, and the potential conversion of open to minimally invasive spinal surgery. The headsets that surgeons like to utilize as hardware can cause headaches, blurred vision, discomfort, and constant worry about malfunctioning systems. Overall, the application of AR/VR in spinal surgery is still in its very early stages but is moving on a path that will produce extremely positive outcomes soon, which will undoubtedly be appreciated by surgeons and adopted by patients. If technology develops further, surgeons may soon use augmented reality (AR) glasses for patient consultations, rehabilitation, training, and even surgery. Virtual and augmented reality will be heavily used in training the next generation of spine surgeons. The field of spine surgery, a significant orthopedic specialty, should be prepared to adopt this new technology. All aspects of spine surgery will see significant improvements in the coming decade as clinicians, engineers, and game designers work together.