Our Sites

Augmented reality reaches the manufacturing floor

How augmented reality simplifies complicated manufacturing tasks

This conceptual drawing shows how augmented reality systems could work for logistics. AR shows fork truck drivers which direction to go, which components need to be moved, and where. Image courtesy of Index AR Solutions.

Augmented reality (AR) has gotten a fair amount of attention in the consumer arena. Tech company advertisements show all sorts of eye-grabbing applications. Want to hold a video conference where the “screen” appears on a nearby wall and moves around as you move from room to room? With the right AR setup, you can.

While not yet ubiquitous, AR may soon become mainstays in certain areas of manufacturing. The technology won’t work everywhere, but in specific applications it may have profound effects. Applying it in industry really boils down to achieving one goal: Simplify the complex.

Accuracy and Reliability

Industrial applications stand apart from consumer and office AR apps. “For one thing, many [office and consumer apps] are add-ons to existing tools we’re already using,” said Matt Kammerait, vice president of innovation for Los Angeles-based DAQRI, an AR company launched in 2010. The firm offers wearable headsets designed for industrial environments. They not only help people do their jobs, but also meet the requirements for personal protective equipment.

Reliability in office and consumer environments isn’t critical. If you hold a video conference using some type of wearable device, and if it only works nine times out of 10, so be it. You can always pick up the phone and talk the old-fashioned way. But if an AR system shows a technician exactly where and how to put an important piece into a critical assembly, the AR software had better work.

Another big difference is accuracy. “With a lot of commercial and consumer applications, there’s no need for things to be incredibly accurate,” said

Dexter Lilley, executive vice president and chief operating officer of Index AR Solutions, Williamsburg, Va., an AR company launched in 2015. “They just have to look cool. Whereas in manufacturing and fabrication, the accuracy of things really matters.”

AR Versus VR

AR essentially overlays digital data on the real world, while virtual reality immerses a person in the virtual world. VR has potential in certain office, engineering, and training situations, including welding. But VR usually doesn’t apply to actual manufacturing. When people make things, they can’t be in a virtual world; they need to be in the real one.

The most well-known AR applications occur through some type of wearable device—a helmet, glasses, or something similar—but other applications involve different hardware. Some use a camera and a tablet, for instance (see Figure 1). Still others project images or visual aids onto a specific space. In this so-called projected AR, an assembler sees a visual aid projected onto an area that shows how and where to orient, insert, or assemble specific components.

More kinds of AR displays may be on the horizon for manufacturing, including what’s known as dynamic holography. “This is a really exciting next-generation display,” said Kammerait. This year DAQRI purchased Two Trees Photonics, a U.K.-based firm that’s developing dynamic holography for heads-up, below-the-windshield displays in high-end cars like Jaguar and Land Rover.

“This is something we see as a huge opportunity not only for consumer vehicles, but also vehicles for the workplace,” Kammerait said. “For instance, think about a windshield on a fork truck. It could show the driver exactly where the next job is.”

Figure 1
In this augmented reality (AR) application, the 3-D model is overlaid on top of the actual components, showing the inspector if the components are in the right location and orientation. Photo courtesy of Index AR Solutions.

Although AR may be more prevalent on the future manufacturing floor, it won’t work everywhere. “The technology doesn’t make much sense in areas where jobs are very repetitive,” Kammerait said. “Those repetitive jobs will move into the realm of industrial automation. The AR can control the automation, but you won’t use AR to perform repetitive tasks.”

The best applications take a complex act and, with visual aids, make it a lot simpler. Still, what one person considers “complex” another may see as obvious. It really depends on that person’s knowledge.

“It’s all about matching what a person knows to what that person needs to know,” Kammerait said.

AR needs to feed enough information to help, but not so much as to distract, hinder productivity, or, even worse, create an unsafe work environment. For instance, a novice being trained may benefit from a complete 3-D model being overlaid on top of the real workpiece in front of him. As training progresses and the worker gains more experience, he needs less information in front of him to do his job effectively. At this point, instead of the full 3-D model, a simple arrow may suffice.

Conversely, if an assembler or technician tackles jobs that change continually, or works in an environment in which the cost of error is very high, even an experienced worker may benefit from the AR displaying more information.

Building a Ship

In metal manufacturing, you really can’t get much more complex than building a ship. This is Dexter Lilley’s world. He has spent more than three decades in the nuclear shipbuilding industry. He ran fabrication, machining, and assembly departments that supported ship construction, and he spent about four years making large steel structures that served as the building blocks for aircraft carriers and submarines.

Several years ago he worked in business development for Huntington Ingalls Industries, and there he met several engineers who were experimenting with AR. In 2011 they were investigating several areas of application, one being deck plate assembly.

“When I saw [augmented reality], I began to understand that this was the way things will get built, if you’re working with your hands. It really is something to make skilled workers even more efficient and capable.”

Lilley worked with Dan Arczynski in strategy and business development at Huntington Ingalls. Both saw potential in the company’s AR development work and its potential for commercialization in industry. Still, Huntington Ingalls is a shipyard, not a software technology company.

“By 2014 Dan asked me, ‘What if we did this? What if we took it outside [to commercialize this]?’ And I said, ‘If you want to do this, I’m with you.’”

Figure 2
The index—the square component with hexagonal patterns—calibrates the augmented reality system, ensuring that elements on-screen align properly with the real elements in the space. Photo courtesy of Index AR Solutions.

After six months of negotiations and hammering out the details, the two officially retired from the shipyard in December 2014; launched Index AR Solutions; and formed a teaming agreement with Newport News Shipbuilding, a subsidiary of Huntington Ingalls.

Specifically, the agreement allows Index to lease Newport News Shipbuilding (NNS) AR engineers, designers, and developers for projects, sales, and development. NNS creates AR for its own shipyards and the U.S. Navy; Index creates AR for commercial clients; and Index and NNS share other government clients. The agreement also allows the two entities to share software and intellectual property.

In a shipyard application for a deck plate assembly, a camera captures an image of a 20- by 40-foot steel plate with studs welded onto it. Some of these plates can have up to 150 studs on them. Typically, welders worked laboriously from paper drawings, comparing reference lines on paper with what’s actually on the steel plate. After measuring, the welder marked the spot and welded the stud. That process could take 20 to 25 minutes.

Using AR, the procedure changed. The 2-D drawings were converted to a digital file. The technician places on the plate something known as an index (see Figure 2). The inspiration for Index AR’s company name, an index is a 2- by 2-ft. patterned target. The technician moves the index several times to calibrate the system. This takes about 15 minutes, but after this, the AR software shows where all of the studs should go, overlaying marks over the real image of the steel deck plate.

“We’re able to locate and line up each stud individually within about three minutes,” Lilley said, adding that the welders no longer need to look at the work package, read the drawing, or measure each stud location from a reference line.

This application does involve two people, one to hold the tablet and indicate where studs should go with a laser pointer, and another person to weld. But again, the operation takes only three minutes per stud, versus 25 minutes doing it the old way.

Inspection Potential

“The big differentiator [of industrial AR applications] is being able to accurately place 3-D models in situ, to visualize where things will go, and then to inspect these components after they’re completed,” Lilley said.

“Object recognition and tracking, which helps the system understand the position of whatever it is you’re looking at, determines how accurately you can position content over the top of the real world,” Kammerait said. “Today we’re already at submillimeter accuracy.” He added that this depends on certain environmental conditions, though some sensors now are able to adapt to challenging industrial and even outdoor environments (see Figures 3 and 4).

“From a measurement standpoint, we’re not to the micrometer level yet,” he added, “but for quick checks and gross measurements, we’re absolutely there today.”

This level of accuracy has allowed AR to aid tasks like visual inspection. Lilley described one shipyard application in which engineers have to identify temporary steel structures that eventually need to be removed. All the steel structures (both permanent and temporary) get blasted and coated, which makes everything look the same. “So at some point, some poor engineer has to take a drawing and go through and look for the temporary steel to make sure it’s taken out.”

Figure 3
A technician wears an AR helmet to assist him with a complicated setup. Photo courtesy of DAQRI.

He looks at the paper drawing, and refers to the actual space. After he determines he’s in the right place, he looks for the temporary steel to confirm it’s been taken out. If it hasn’t, he writes up a work order to have it removed.

Using AR, the engineer aligns the tablet camera with the actual space. On-screen, the temporary steel is green, the permanent steel is purple, and the foundation is blue. If a green component appears on-screen, but no steel is in the actual space, the temporary steel had been removed. If the green component on-screen matches up with an actual steel plate, that steel is temporary and has to be removed.

“That inspection work previously took 36 hours,” Lilley said. “[With AR], he was able to do it in 90 minutes.”

Right Information, Right Time

The potential for industrial AR abounds. AR devices can hook up to thermal or similar sensors; a worker using that device would be able to look at a machine and see before his eyes which areas of it may be having trouble. In the training arena, sources said that AR has already been integrated with auditory signals, like the sound of pressure being released after turning a knob, that show the trainee what to listen for when he eventually works in the field.

And then there’s the potential for information management—perhaps the most challenging aspect of modern metal fabrication. Software can convert new or revised job information to step-based processes for AR devices and disseminate them automatically to the people who need them on the floor, based on specific business rules: who needs to review and sign off on what and when, for instance.

Theoretically, an assembler or other worker using AR could receive a new set of instructions, updated just moments before. He looks through the AR device and sees virtual models of the new components overlaid onto the real picture. The picture shows exactly how each should be placed and in what order. It all looks so simple—yet, of course, it is anything but.

DAQRI, 213-375-8830, www.daqri.com

Index AR Solutions, 757-707-8046, www.indexarsolutions.com

Figure 4
AR helmets can also be used for field inspection and repair. Photo courtesy of DAQRI.

About the Author
The Fabricator

Tim Heston

Senior Editor

2135 Point Blvd

Elgin, IL 60123

815-381-1314

Tim Heston, The Fabricator's senior editor, has covered the metal fabrication industry since 1998, starting his career at the American Welding Society's Welding Journal. Since then he has covered the full range of metal fabrication processes, from stamping, bending, and cutting to grinding and polishing. He joined The Fabricator's staff in October 2007.