Extended Reality for Healthcare Systems: Recent Advances in Contemporary Research

Extended Reality for Healthcare Systems: Recent Advances in Contemporary Research focuses on real world applications in medicine, also providing an overview of emerging technologies. The book includes case studies that break down the ways in which this technology has and can be used, while also taking readers through evidence, best practices and obstacles. Sections emphasize evidence, research-based practices and work. Content coverage includes Enhancing Medical Education with AR/VR, and XR: The Future of Surgery and Building Systems for Enhanced Health, and more. Readers will learn how to use this technology to improve existing systems by enhancing precision and reducing costs.

Other sections cover extended reality in elderly care and remote monitoring of patients, building systems for enhanced health, including telehealth and telepsychiatry, using AR and VR in medical education, and designing technology for use in telesurgery.

• Offers advice on the development of state-of-the-art tech-driven healthcare systems and technologies for improving the quality of healthcare
• Focuses on healthcare solutions that are inclusive and cost-effective
• Discusses the future, limitations and challenges associated with the use and adoption of XR for healthcare

• Language: English
• Publisher: Academic Press
• Release date: Oct 18, 2022
• ISBN: 9780323985390

Book preview

Chapter One: Extended reality

bringing the 3Rs together

Samiya Khan     Faculty of Science & Engineering, University of Wolverhampton, Wolverhampton, United Kingdom

Abstract

Extended reality (XR) is an umbrella term that includes augmented (AR), virtual (VR), and mixed realities (MR). Widespread applications of these technologies in improving user interaction and experience led to its popularity in the recent past. Moreover, the availability of equipment that can be commercially accessed by users for complex VR and MR application has also popularized this technology and driven research into fields that explore the use of XR in specific application domains. However, multiple views on the definitions and concepts in this regard are presented in existing literature. This chapter explores the three types of realities and their contextual meaning with respect to XR. In addition, this chapter also explores the differences between these technologies. Additionally, this chapter explores other related concepts in literature that expand the idea of XR to ∗R or All-R, to provide a comprehensive view on existing and related literature in this field. Finally, it explores the different technologies and platforms available for XR development along with an introduction to the potential use of XR in healthcare.

Keywords

Augmented reality; Extended reality; Immersive technologies; Mixed reality; Virtual reality

1.1. Introduction

Extended reality (XR) is an umbrella term that jointly includes augmented (AR), virtual (VR), and mixed realities (MR). The past few decades have witnessed remarkable advancements in research and technology adoption for these fields. The multifaceted benefits of using XR technologies ranging from developing user-friendly applications to streamlining business processes to improve productivity and efficiency have led to extensive industry investments and research efforts in this domain (Du et al., 2018; Miettinen and Paavola, 2014; Stanton et al., 2020; Radianti et al., 2020; Wang et al., 2018).

XR technology has lately garnered immense attention with sale of XR devices expected to rise to USD $9.1 billion by 2021 from USD $1.5 billion in 2017 (Flavián et al., 2019). The fact that this technology enables a unique user experience makes it not just useful, but it also makes this technology extremely fascinating. Conventionally, this technology has most extensively been used in the gaming industry. However, recent advances have broadened the realm to education, businesses, and healthcare sectors, in addition to many others (Arnaldi et al., 2018). The growing market value and use-case-based applications in diverse domains make XR a technology for the future.

The XR technology in the most basic way can support use cases for remote collaboration, experience, and intervention. Therefore, any use case that requires human intervention can greatly benefit from this technology with the added benefit that the individual need not be physically present onsite. This inevitably reduces costs and saves time. In addition to mediating the constraints of space and location, the use of these technologies has additional advantages such as safety and resource optimization.

Existing literature on XR research (Radianti et al., 2020; Zahabi and Abdul Razak, 2020; Seymour et al., 2018; Yan et al., 2011; Nikolić et al., 2019) focuses on one of the component technologies (AR, VR, or MR) or the use of XR for specific use cases. This chapter aims to provide the theoretical and methodological overview of the three component technologies, also providing insights on their evolution and applications. This chapter provides a literature review on XR and the three constituent technologies elaborating on the foundations and applications of these technologies.

The rest of the chapter is organized in the following manner: Section 1.2 provides history of XR, while Section 1.3 introduces the three main Rs including AR, VR, and MR. Section 1.4 gives a theoretical background on definitions of XR and its component technologies, and Section 1.5 elaborates on the new Rs introduced in research. Finally, Section 1.6 discusses the different technologies and platforms associated with XR, and Section 1.7 concludes with insights on scope for future research.

1.2. History

Interestingly, the history and evolution of XR has its roots in art. The first immersive system was developed by Giovani Fontana, an Italian engineer in the 15th century. The system was called CAVE, and it used light from lanterns to make wall projections of images (Arnaldi et al., 2018). Many engineers and artists contributed to the concept of illusionary world in the years to follow. The modern concept of VR is known to have been defined by Antonin, a theater artist, in 1958 whose work majorly focused on characters, images and objects for theatrical purposes.

One of the most significant contributors to popularity and development of VR concept is science fiction. The era of the 80 and 90s witnessed many groundbreaking works in this domain; some of which include True Names and Snow Crash (Arnaldi et al., 2018). Historically, science has evolved through science fiction and what was imaginary and impossible in one age become the reality of another. Recent works of fiction in this domain such as Minority Report have open doors for commercial adoption of XR (Dick, 2002).

On the technology front, the first machine created to play videos with immersive experience was Sensorama. This machine was developed by Heilig and used stereographic audio and images to enable the experience (Heilig, 1962) and is known to have been inspired by Antonin's concept of a virtual theater. It is for these reasons that this machine is considered one of the first VR systems developed. The limitation of this machine was its size, which was overcome by a device called The Ultimate Display, which was developed in 1965 by Ivan Sutherland and Robert Sproull (Sutherland, 1965). The merger of portability to VR gave birth to the HMD concept, which was used by many companies and scientists to develop advanced VR devices.

The next milestone in the evolution of XR was the launch of The Virtual Environment Workstation in 1984 by NASA (McGreevy, 1991). This machine improved its predecessor by incorporating functionality such as tracking of gestures made using fingers and position of head. In addition, the graphics of this machine were the most powerful in its time. In comprehension of the limitations of HMD concept, Milgram and Kishino (Milgram, 1994) proposed the Reality to Virtual concept in 1994.

The widespread adoption of XR is known to have occurred after 2010, more so after the 2013 launch of the first popular VR device called Oculus Rift (Arnaldi et al., 2018). Many other devices such as Valve Index and HTC Vive were launched later (Huang et al., 2019). HoloLens, an MR device, was launched in the commercial space in 2015 (Arnaldi et al., 2018). Moreover, the availability of AR development platforms such as ARKit and ARCore has facilitated development of AR-backed mobile applications.

1.3. The 3Rs

XR is a combination of 3Rs, namely, AR, VR, and MR.

1.3.1. Virtual reality

VR allows users to interact with a digitally created world in a completely immersed mode. There are three main characteristics of VR that includes interactivity, real-time experience, and a full 360-degree view of the space. The 360-degree view of the virtually created world is typically enabled by the device (Speicher et al., 2019). To ensure that users have a pseudonatural experience, the virtually created objects and entities must have a real-time behavior to fulfill the real-time criteria (Arnaldi et al., 2018). Another significant aspect of providing pseudonatural experience is interactivity. Users must be able to position themselves and have an active interaction with the virtual world (Arnaldi et al., 2018). An image of an individual experiencing VR using a headset can be seen in Fig. 1.1A.

VR has found numerous applications in fields such as manufacturing, construction, education, healthcare, and designing. The core of VR benefits lies in the fact that it allows high communication efficiency. With that said, the use of VR is also known to have an impact on humans with conditions such as headaches and stomach awareness reported (Weiβker et al., 2018). Additionally, the cost of devices required for VR is still high in comparison with conventional entertainment equipment even though there has been gradual and considerable reduction in costs over time (Avila and Bailey, 2014).

1.3.2. Augmented reality

AR is a technology that extends real environment in such a manner that knowledge and perception of the real world are reinforced (Arnaldi et al., 2018). One of the simplest examples of AR in action is applications that allow a user to open the camera and view additional information about an object in the field of view. The three main characteristics of AR include interactivity, combined operation, and real-time responsiveness.

AR does not create a digital or virtual world. Instead, it simply appends a layer on top of the real world to provide additional information to the user (Ro et al., 2018). Therefore, AR enabled combination of real and virtual world in such a manner that one supplements the other to provide enhanced user experience. An image of a girl using AR app on the handheld device can be seen in Fig. 1.1B. For seamless user experience, AR needs to work in real time and allows interaction between user and system to the level that cooperating elements to support interaction are provided.

Figure 1.1  XR user experience. (A) Virtual Reality; (B) Augmented Reality; (C) Mixed Reality.

Uses of AR are found in specialized domains such as driving assistance, gaming, professional gesture assistance, and tourism, to name a few (Kimura et al., 2017; Mgbemena et al., 2016). Similar to VR, one of the biggest strengths of AR is that it enhances communication efficiency by providing visual information and enhanced user experience. In comparison with VR, the health-related concerns with VR are not there in AR, but then AR experience is not as immersive and fascinating as VR.

1.3.3. Mixed reality

MR can be best described as a concept that lies on the intersection of VR and AR (Speicher et al., 2019). While AR presents a combination of the real and virtual world, VR completely replaces the real world with a digitally created world. However, MR creates a digital or virtual world under and working along with the real world. Like VR, a head-mounted display is required for MR experience. The limitation of AR that it cannot simulate the virtual world without the assistance of practical support from the environment is mitigated by MR. Thus, MR is more inclined toward VR than AR.

Therefore, in a way, MR inherits the advantages of AR and VR solving their disadvantages to effect. However, MR has its own disadvantages and limitations, which include high cost of equipment (Kun et al., 2017) and limited capabilities of the available MR devices. Any form of information consumption in this digital age can be impacted greatly by MR. One of the most popular products in this domain is HoloLens (Flavián et al., 2019). An image of an individual experiencing MR using HoloLens can be seen in Fig. 1.1C. MR also happens to be one of the best candidate technologies for the healthcare sector in view of the multiple available scenarios that require merging of real and virtual world to communicate and comprehend in the most effective manner.

1.3.4. Extended reality

The concept of XR was given by Charles Wyckoff in 1961 (Wyckoff, 1961, 1962). The introduction of this concept was a result of the patent filed for XR film that could enable observance of phenomena beyond the capabilities of normal human vision. The modern trademark for this concept is registered in the name of Sony, who extensively use this term to refer to mobile AR. Several definitions of XR have come into existence, which are described in the following:

• Type 1 XR

This type of XR defines X mathematically as a variable that can take any number of the real number line, which is an axis for one of the following:

o Type 1a XR

This type of reality augments human sensory capabilities with the help of wearable devices and thus is seen as an extrapolation (Wyckoff, 1961, 1962). Therefore, this form of XR is a superset of MR.

o Type 1b XR

This type of reality blends human sensory capabilities with virtual elements being seen as an interpolation (Mann et al., 2018). Therefore, this form of XR is similar to MR.

• Type 2 XR

In this type of XR, the real and virtual aspects of reality are provisioned by sensors/actuators and an online platform, respectively. Therefore, this form of XR is a subset of MR.

1.4. Concepts and definitions

As mentioned previously, XR is a term that typically refers to the technological foundation for AR, VR as well as MR (Fast-Berglund et al., 2018). This makes XR a rather loosely defined term with some references also using the term cross reality to refer to the same concept (Sherman and Craig, 2019). One of the first milestones in the technological chronology of XR includes conceptualization of VR in the 1960s (Aukstakalnis, 2017; Berg and Vance, 2017; Whyte and Nikolić, 2018). However, it was not until the 1990s when multiple solutions in the VR space came about (see, e.g. Moore, 1995, 1998).

The categorization of VR is classically performed using the virtuality continuum (Milgram and Kishino's, 1994). The virtuality continuum states that there are two environments namely the virtual and real environments, and MR lies between these two environments. Therefore, when the observer or participant is completely immersed in a synthetic, digitally created environment, the type of reality is referred to as VR.

The concept of AR was introduced in the 1990s (Aukstakalnis, 2017). When the real environment is augmented or expanded with objects that are virtual and digitally produced, the type of reality is called AR. Another definition of AR refers to it as augmentation of virtual, digitally produced environment with elements from the real environment (Milgram and Kishino, 1994).

In conclusion, the key difference between AR and VR is the fact that AR supplements reality, whereas VR totally replaces it. MR is a term that cumulatively refers to both forms of augmentation: virtuality and reality (Fast-Berglund et al., 2018). There is literature-level contradiction on whether AR and VR are sibling technologies (Wang et al., 2018) or AR is a subset technology of VR (Sherman and Craig, 2019). Key differences between the three constituent technologies and their place in the XR tech domain are provided in Table 1.1.

Two of the main features of XR include immersion and interactivity (Radianti et al., 2020; Sherman and Craig, 2019). The definition of these two features differs on the basis of XR component and the devices being used. For instance, if we consider VR, then VR is actually a medium and the content being mediated is the VR world (Sherman and Craig, 2019). Thus, in this case, the medium, VR, will have two characteristics, namely, immersion and interactivity. In addition to medium and content, the space will also constitute of content creators and participants. For the other two kinds of realities (Rs), the definitions can be extended from the VR context because all the Rs have an aspect of VR in them whether the whole reality is virtualized or augmented.

Table 1.1

The sense of being within an environment is referred to as immersion. This can be accomplished via physical or mental means (Sherman and Craig, 2019, p.10). There have been many perspectives on how presence is related to immersion. According to Slater and Wilbur (1997), presence is immersion-related state of consciousness, while Nykänen et al. (2020) considered presence as a state that is consequential to immersion. These aspects make immersion a significant facet of XR. Since VR is a medium of entering and interacting with digitally created world/content, the physically approached immersion excludes the sense of any interaction with the real (Wang et al., 2018). Thus, VR is usually associated with and referred to as immersive technology (Whyte and Nikolić, 2018) where immersion can also be understood as sense of presence.

The extent of immersion may vary from nonimmersive to completely immersive (Jamei et al., 2017). The most widely used classification criterion for deciding if a technology is immersive or not is based on the VR hardware used for the same. For instance, if VR experience involves the use of headsets, it is immersive. On the other hand, desktop-based VR is classified as nonimmersive. In fact, existing literature also question if monitor or 3D glasses-based experience can even be called VR because it does not involve immersion (Whyte and Nikolić, 2018). In contrast to VR, AR, and its relatedness to immersion is confined to usability (Aukstakalnis, 2017). From a generic point of view, high-level usability is considered to have been achieved if a technology can be accessed in an effortless manner without any scope for unreasonable