What is Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR) Technologies?

Augmented reality (AR) and virtual reality (VR) and mixed reality (MR) — or used collectively, XR— are disruptive technologies that will impact many aspects of our lives in the future. According to research firm IDC, a growing number of companies are turning to VR as a way to drive training, collaboration, design, sales, and numerous other use cases. The firm adds that AR use cases are also growing and will outpace Virtual Reality with a wide variety of companies leveraging next-generation hardware, software, and services to fundamentally change existing business processes and bringing new capabilities to first-line workers who require hands-free technology.  Mixed Reality enables a user to experience objects and set them into the real-world environment virtually.

With 5G technology right around the corner, these technologies are going to explode exponentially.
5G is reaching speeds that are 20 times faster than 4G LTE, which, coupled with new technologies, alliances, and standards that are developing, is going to push AR, VR and MR to all-new horizons.

Worldwide spending on AR, VR and MR is forecast to reach $160 billion in 2023, up significantly from the $16.8 billion forecast for 2019. According to IDC’s Worldwide Semiannual Augmented and Virtual Reality Spending Guide, the five-year compound annual growth rate (CAGR) for AR/VR/MR spending will be 78.3%.

Video: How Virtual Reality Impacts Business and Our Personal Lives.

Why Quality Assurance and Testing Is Important for AR, VR and MR Products.

Technology and media firms are developing new immersive experiences every day, and with these experiences come new rules to ensure quality, efficiency, and effectiveness. Innovative approaches to manual and automation testing are a critical need for firms that are attempting to deliver business and consumer products, services, and content in this sector.

Traditionally, QA is considered to be the last phase in the process of development and focuses on a series of test scripts to evaluate product quality according to functionality, performance, reliability, compatibility, usability, security, maintainability, and portability. But AR, VR and MR introduce a new quality paradigm that can’t be programmatically tested: Immersiveness. Because AR, VR and MR blend the physical and virtual worlds in new ways, it is crucial for the experience to be immersive for the end-user.

The Most Effective Testing on AR, VR and MR Products Begins with Preparation.

The most important approach to effective testing is to identify as many scenarios, or angles of perception, as possible. Most companies will focus on the most obvious factors, which are associated with functionality, but there are many other angles to consider.

Before any testing begins, QA engineers should review and familiarize themselves with the product’s scope of requirements and prepare a list of real-world experiences. This facilitates an understanding of all the potential scenarios for user engagement and provides a holistic view of the product.

Preparation for QA for VR usually includes environmental setup, meaning that a play space needs to be created, and any obstacles must be removed to ensure a clear path. AR has a similar type of setup, however, existing objects (e.g. table, chair, etc.) need to be detected and tagged prior to usage so they can be identified during operation. For MR, you’ll still want to augment a real-world setup like AR, but it will include an interactive layer to the scene that AR doesn’t capture. For example, capturing holograms that overlay on physical objects (I.e.: couch, side table, the kitchen counter), will need to be considered in the field of vision of your MR device setup. 

Measuring performance in the mixed reality world typically revolves around calibration of sensors, IMUs, and checking a series of collective points for frame rates, latency, machine vision, and various optical measurements.

The key to testing both AR, VR and MR technologies can be broken down into two approaches.

1. Manual testing: Emulating the real-world environment, setting up the play space, and utilizing the full 360-degree range of scope, while measuring for performance variables and gameplay. This also involves testing camera tracking and reducing latency.

As digital experiences occupy more virtual worlds and objects, the design, development, and testing process requires an approach focused on the human aspect. Since the user becomes part of a system both mentally and physically, taking the user’s feelings and goals into consideration is key to creating truly immersive digital experiences. Unlike traditional QA that relies on scripts, various test data, boundaries, and equivalence partitioning, AR, VR and MR testing will rely on feelings and sensations to deliver final feedback to developers and designers.

Thinking through multiple considerations with multiple lenses will go a long way toward discovering issues that might not have surfaced otherwise. Not only will the quality of the AR or VR product have improved, but extra, unnecessary cycles of testing and resolution can be avoided.

Challenges and Common Misperceptions about QA and AR, VR and MR Testing.

There are several challenges and common misperceptions about QA and AR, VR and MR testing.

• AR, VR and MR developers can test their own software and hardware effectively.
  While this is certainly true, working with a QA solutions provider with expertise in AR, VR and MR brings an experience level, as well as breadth and depth of knowledge, that can improve quality and add business value.

• Developers already know what they don’t know.
  Because AR, VR and MR developers are deep into their projects, they often believe they know every aspect of the project and what they will need to learn. However, an external QA team can bring new perspectives to test and analyze aspects of the project that may not have been considered by the internal team.

• The same approach to QA for enterprise or consumer products can be used for AR, VR and MR.
  Since QA is based on understanding and meeting user expectations, understanding that enterprise users and consumers have different requirements will impact how the testing is carried out. Certainly, the differences in user environment and conditions will dictate how and what type of testing is done, as well as what the software is used for. Enterprises are likely to have more stringent expectations with regard to security, for example.

• Functional Testing will produce a good product.
  In the world of AR, VR and MR the end-user experience is tightly integrated with the device capabilities. Whether it be a mobile phone or a dedicated device, testing for AR, VR and MR technologies involve testing the full stack of software, hardware and firmware beyond the traditional user experience. Areas to cover in a test plan may include functions such as battery life and power consumption, firmware, Bluetooth and Wi-Fi, overheating, security among others.

• Cutting a QA program will reduce a company’s costs.
  Reducing QA testing in an area that is experiencing such high growth and innovation as AR, VR and MR will almost certainly result in a higher cost to the company, due to compromised product performance. Today more than ever, with social media and online reviews and rating sites such as G2.com, maintaining top performance is crucial for upholding a brand’s credibility and increasing market share.

The Q Analysts Approach.

• Q Analysts’ testing services involve the full end-to-end spectrum of a head-mounted display (HMD), rather than just repeating test cases. We have done system-level testing, which involves a fully automated lab suite of hardware, from robotic automation of full 6DOF (6 degrees of freedom) testing, to sensors and IMU calibrations, to full regression test suites against industry standards and even competitive benchmarking. In short, we do a lot more than just test the headset as a user.

• Our advisors and testers understand how VR, AR and MR platforms work and how to create the right test strategy. They can help develop out-of-the-box solutions to test hardware, systems, content, platform integration, and device compatibility to name a few. We have helped our clients with innovative approaches to test automation. In addition, we work on innovative needs that this area has spawned, including capture and tagging of reference data for all aspects of human gestures as well as inanimate spaces and objects.

• Q Analysts approaches each unique project from the perspective of a consultancy, partnering with our clients to bring significant added value and optimal results.

• We offer customized quality assurance programs with a suite of service solutions, beyond simple “staff augmentation.” Our subject matter experts provide fully managed and tailored programs, acting as a secondary arm of a client’s company, and deploying a highly specialized team built around project scenarios and technologies.

• Q Analysts invests in team members, offering highly trained professionals who bring specific expertise and operate with company-level commitment. We spend significant resources on management overhead to train and review the highest-quality staff and support them for sustained career growth. Training and maintaining a satisfied work team contributes to company loyalty and the continuation of institutional knowledge, so career investment builds through consecutive temporary projects.

• We are driven by initiatives, recognizing how even a temporary project fits within the context of larger organizational strategy. Q Analysts understands all the roles and key responsibilities associated with a project and can determine the best team members to deploy who comprehend the core technology. Partnering with a client to develop a relationship around an initiative enables us to contribute to the organization in meaningful ways while adding real business value.

• Q Analysts offers complete turnkey onshore offsite or offshore services in the form of our Q Testlab, which uses a proven and scalable delivery framework to provide outsourced QA and testing services at a competitive overall ROI. The lab’s services are an extension of our clients’ capabilities and can be utilized for a wide range of applications and technologies. We customize each project to suit client requirements and can replicate the onsite environment with such elements as networking, hardware, electricity, and security in the lab, saving time and resources.

• Q Analysts also specializes in setting up remote offsite testing areas that replicate the required environments in homes, offices, and other facilities. For example, the performance of a home automation device may be evaluated at a specific height, representing a kitchen counter, and a given distance away from furniture such as a couch. Q Analysts provides technical skills, project management processes, communications, and productivity for offsite, onshore or offshore QA and testing services.

Conclusion.

AR, VR, and MR are disruptive technologies that impact, and will continue to impact, many aspects of both business and our personal lives. With new immersive experiences, there are new rules for ensuring quality, efficiency, and effectiveness. Innovative approaches to manual and test automation are a critical need for firms attempting to deliver business and consumer products, services, and content in this sector.

The Q Analysts approach is driven by vast experience in AR, VR and MR giving us the distinct advantage of being able to identify potential issues, alternative scenarios, and outcomes that may not have occurred to our clients. This is why we are the preferred quality assurance and testing partner, having helped shipped some of the top-selling headsets in the world, developed by industry leaders.