mixed reality (hybrid reality, extended reality)
Mixed reality (MR) is a user environment in which physical reality and digital content are combined in a way that enables interaction with and among real-world and virtual objects. Unlike virtual reality (VR) which immerses the end user in a completely digital environment, or augmented reality (AR) which layers digital content on top of a physical environment, mixed reality blends digital and real world settings. Mixed reality is sometimes considered a type of augmented reality (AR), but its capacity for interactivity between real-world and digital elements places it further along the virtuality continuum, which has physical reality at one extreme and immersive virtual reality at the other.
Mixed reality is sometimes also referred to as hybrid reality or extended reality (XR). A headset form factor follows the user's gaze and maps the user's physical surroundings and software then uses deep learning algorithms to align digital content with specific areas of the map. MR programming allows digital objects to interact with physical objects and people to interact with digital objects as if they are physical. The result is that an ordinary desktop can turn into an interactive computer touch screen or an MR-generated film character can sit on the homeowner's couch.
Although mixed reality is still in the early stages, it is already being used in many industries for educational purposes. For example, aircraft manufacturers are using MR as a cost-effective way to train repair technicians. Instead of pulling an engine out of an aircraft to conduct a training session, technicians wearing special headsets can view a holographic image of an engine and use gesture, gaze and voice user interface (VUI) commands to interact with the hologram, changing perspectives and extracting meaningful information, layer by layer.
Other use cases for mixed reality
Construction - supervisors can use MR to cue workers about what work needs to be completed. For example, a supervisor might drop a virtual pin to signal that a specific piece of equipment is not working correctly. Workers will be alerted and use their headsets to view schematic diagrams and diagnose what needs to be repaired.
Manufacturing - inspectors can use MR headsets during plant tours and view real-time data for specific mechanical components. Information about the component's performance and energy consumption can be transmitted with Near Field Communication (NFC) technology and will update when the inspector's gaze turns toward a different component.
Design - designers can use MR to interact with new products before they are built. For example, MR could allow spacecraft engineers to view and interact with full-scale digital prototypes of a new Mars rover in order to identify potential design issues, reducing the overall development time and cost.
B2B communication - international corporations can use MR to provide simulated face-to-face meetings with geographically dispersed product teams. Participants can view a life-size, 3-D version of the product being discussed, while translation services can be integrated, eliminating language barriers.
Healthcare - surgeons can use MR to practice inserting an extremely small medical implant into a specific patient’s body. It's expected that in the near future, medical students will also be using MR imaging, instead of cadavers, in anatomy classes.
Supply chain management - a supply chain manager can view a pallet through his smartphone's camera or smart glasses, verbally ask a question about the pallet's shipping history and view an immediate answer on the display.
History of virtual reality, augmented reality and mixed reality
Virtual reality (VR) can trace its roots back centuries with the popularity of the stereoscope, a hand-held device invented by Sir Charles Wheatstone in the early 1800's. Stereoscopes use mirrors to blend twin photographs into a single image, creating the illusion of depth. Morton Heilig, sometimes referred to as the father of virtual reality, took stereoscopic imaging a step further when he patented the Sensorama Simulator in 1962. Heilig's immersive sensory environment was essentially a giant View-Master that enabled the user to experience stereoscopic images accompanied by stereo sound and aromas. The label virtual reality gained popularity in the 1980’s and is usually credited to computer scientist Jaron Lanier, the inventor of the data glove.
Boeing researcher Thomas Caudell coined the term augmented reality in 1990 to describe how the head-mounted displays worn by Boeing's electricians worked. Instead of relying on a wiring map that was physically printed on plywood, electricians were given headsets that could project a digital version of the map onto the plywood. Using AR not only saved the company money by making plywood substrates reusable, it also allowed electricians to implement digital modifications to the wiring maps in real time. On the consumer side, one of the first commercial applications of AR technology was the yellow "first down" line that began appearing in televised football games in 1998. The smartphone app Pokémon Go also introduced a wide audience to how augmented reality works by layering digital characters on the phone camera's field of vision.
The term mixed reality was first introduced in a 1994 by Paul Milgram and Fumio Kishino in a paper entitled "A Taxonomy of Mixed Reality Visual Displays." The paper introduced the idea that mixed reality is a continuum that includes VR and AR, as well as any new technologies that will allow digital content to interact with real world objects and humans to interact with digital objects as if they are real. For example, if Pokémon Go added MR capabilities, the Pokémon character would be able to climb a real set of stairs because the digital character would be aware of its physical surroundings.
Mixed reality products and future trends
Current mixed reality content is generally delivered through headsets, although holographic projections and interactive virtual displays are expected to play an important role in MR. Until recently, creating consumer products to take advantage of the MR continuum wasn't cost effective because the technology requires an immense amount of processing power as well as strong computer vision and image recognition capabilities. Big data, cloud computing and recent advances in artificial intelligence (AI) are making MR capabilities more affordable and as the price comes down, it’s expected that MR will radically change human computer interaction (HCI).
Industry experts predict that mixed reality will become a $1.2 – 6.9 billion global industry by 2024. Although gaming will continue to play an important role in its development, Enterprise MR is expected to dominate mixed reality research efforts in the coming years. Currently, MR headsets are priced anywhere from $300 to $3,000 a set. In the near future, it's expected that camera-equipped smartphones, tablets and smart eyeglasses will replace specialized headsets. Apple is rumored to have mixed reality eyeglasses in development that will connect to the user's iPhone, much like the Apple Watch.
Windows Mixed Reality (WMR) offers a variety of products powered by Windows 10 and the Universal Windows Platform (UWP). Popular gaming products include HoloLens and the Acer MR headset. Microsoft is also experimenting with MRTouch, software that uses HoloLens’ environment mapping and hand tracking technology to allow flat surfaces to be used as input devices.
Microsoft's Edge browser and Mozilla's Firefox Reality browser have each been developed with mixed reality in mind to incorporate features that will make it easier for headset or smart glass wearers to easily shift their gaze between 2D and 3D content within the same display.