Mixed reality comes in focus for maintenance and beyond

Here are some of the more popular use cases where AR is gaining a foot hold:

Operator work instructions. Given the rising complexity of heavy equipment and industrial assets, experts see big potential for leveraging AR as a more intuitive and engaging way to deliver work instructions to shop floor operators and workers in the field. Highly configured products like offshore power generation units or agricultural combines are complex to assemble, typically requiring operators to juggle binders of paper documents and several screens of how-to instructions on proper procedures. “AR is a way to deliver all the right information in one screen in the right context and overlay it on top of the physical unit where there is work in progress,” explains Milan Thakker, an IoT manufacturing solutions specialist for PTC. “The operator doesn’t have to put things down or pick things up and they don’t have to look at a specific set of work instructions to see the information they need in context to what they’re doing.”

Pre-sales support. For highly customized and complex products, AR provides a way for customers to visualize the product in their specific setting well before it’s built. While this application is a natural fit for cars and other industrial products, thyssenkrupp is using Microsoft HoloLens to visualize, configure, price and quote a stair lift in real time. No two stair lifts are alike, with each system custom-designed and tailored to an individual’s requirements; HoloLens ensures customers can visualize what their chair will look like on their staircase. At the same time, the AR tool, combined with a custom-built Configure Price Quote (CPQ) solution, lets the thyssenkrupp team measure, visualize, configure, price and quote a stair lift in one visit, greatly streamlining the process. “This fully digitized engineering-to-order process allows both sales and manufacturing operations to sync up in the digital realm and significantly reduce the time from initial call to final delivery,” a thyssenkrupp spokesman says. “Above all is the impact the process has on enhancing the customer experience.”

Remote expert guidance. It takes highly specialized expertise to maintain and operate complex industrial assets, but that manpower isn’t always available on site and it can be expensive to get it there. AR—supplemented with 3D annotation capabilities and in some cases video calling—connects field technicians and operators with remote experts, who can see what’s happening with a piece of equipment without having to be physically present. Thyssenkrupp’s elevator group is leveraging Microsoft HoloLens to give elevator support repair personnel the ability to reach out to expert advice since they are not always 100 percent familiar with every elevator they are called on to service. Another HoloLens partner, Trimble, is tapping AR technology as part of its Connected Mine offering to allow remote teams to share 3D holographic data to improve communication, save on travel time, and make more informed decisions, says Aviad Almagor, director of the mixed reality program at Trimble. “The digital and physical worlds are merging, and mixed reality plays a vital role,” he says. “Enriching mixed reality with AI [artificial intelligence] and machine learning processes will enable automation of workflows such as quality control and production control.”

HMI replacement. AR/VR also has potential as an alternative to the traditional human-machine interface (HMI), according to Andy Stump, business manager for design software at Rockwell Automation. Instead of manually checking individual HMI screens on equipment, AR allows operators to walk the floor and see relevant information such as tank capacity, temperature or line speed overlaid on the equipment and contextualized in line of sight, Stump explains. Coupled with a digital twin, which serves as a real-time data feed, there’s greater potential for optimization and predictive maintenance. “Using the digital twin and AR, you can optimize in the virtual world before you do something in the physical world,” he says. “It may not always be practical to test something out in the physical world.”

Quality control and inspection. Given the harsh environments in the process industry, quality control is a more effective application for AR than maintenance, notes Youssef Mestari, program director for Honeywell’s Connected Plant program. Field operators are tasked with making regular inspection rounds to check that equipment temperature is right or that there are no leaks or sounds coming from a compressor. Using AR tools and with a 3D model of the asset or plant, operators can receive a step-by-step inspection route with the right data overlaid directly on the asset, he explains. For example, Continental Tire has a pilot application that leverages AR technology with Iconics HMI/SCADA systems to facilitate tire inspection. “It makes a nice inspection quality control system that is hands-free,” explains Russ Agrusa, Iconics’ CEO and president.

Challenges remain

While there have been huge strides made in hardware, some of the AR devices tend to be heavy and bulky, making them uncomfortable to wear for full-day shifts. In addition, there are still safety concerns as operators equipped with AR glasses could get caught up in the virtual world and lose grounding in their physical surroundings. “Creating situational awareness is key for successful deployment,” PTC’s Thakker says.

3D and AR development skills are another barrier because most manufacturers don’t have the on-staff expertise to develop AR apps. In fact, the lack of packaged AR solutions prompted the recent rollout of Daqri’s Worksense productivity suite of software for its AR smart glasses. Worksense offers five common use case apps for AR, including: Show, for hands-free remote expert guidance and annotation; Tag, for attaching critical information like notes or multimedia assets like training videos to objects and assets to improve management and operations; Scan, for mapping out the environment and equipment to create a 3D digital model to aid in maintenance tasks or plant retrofits; Model, initially for creating immersive, full-scale walkthroughs using Autodesk BIM Docs; and Guide, for creating multimedia work instructions.

“Over the last year or two, a lot of companies have done proof of concepts around AR, but it required custom software development in order to make it work,” says Jason Haggar, Daqri’s vice president of global partner and developer programs. “We’re trying to take existing business processes and make them better through AR. It’s less about doing something like a science experiment and more about making it accessible for companies of all sizes to do real AR apps.”