Why manufacturers need to be prepared for a cyberattack

In today’s digitally connected world, manufacturers should do themselves a favor and pay attention to cybersecurity. cifotart/iStock/Getty Images Plus

When Chris Lawless, president of Wilson Tool Intl., got a call on Sunday, March 13, and was notified of an information technology (IT) issue at the company’s White Bear Lake, Minn., headquarters, he didn’t think much about it because in the past those issues were resolved quickly. A follow-up phone call only moments later made him a little more concerned. This wasn’t a typical IT issue.

Mark Haupt, Wilson Tool’s IT manager, originally thought that maybe a server had gone down that Sunday, but as he and his team were poking around the network, they started to wonder the same thing: “Have we been breached?” That’s when the call went out to the company’s IT security consultant, who began walking them through the proper response.

After several hours, Haupt said, they found the evidence of what was going on. It was a text file buried on one of the servers: a ransom note.

“By the time you find that note, there’s a realization you’ve been locked out, at least locally, of most systems,” Haupt said. Further investigation, however, revealed that eight out of the company’s 12 locations outside of the U.S. also were hit by this cyberattack. Wilson Tool did not have access to internal files stored on the network, enterprise software, CAD software, and machine control software. Without the use of a time machine, the company quickly found itself in a manufacturing era before widespread automation.

Wilson Tool reached out to The FABRICATOR in May to speak about the cyberattack because it wanted to warn others that this could happen to them. Haupt said that Wilson Tool thought it was in “pretty good shape” with its IT practices, such as updating systems and stopping phishing attempts from becoming a big issue, but even a small lapse in vigilance can lead to big problems.

As this publication went to press, Lawless said that Wilson Tool is in a “much better place” than it was in March, but it might be the end of summer before the company is back to operating the way it was before the cyberattack. At least, the outside world will see it that way. Wilson Tool is changing the way it goes about protecting itself from another such attack. The company’s experience might be a good reference point for other organizations that are wondering what they would do if they found themselves locked out of their own systems. In today’s digitally connected world, manufacturers should do themselves a favor and pay attention to this tale.

Immediate Actions

In the hours after the cyberattack discovery, Lawless said that his focus went from wondering if the systems could come back online to how Wilson Tool would respond to customer demands without its computer systems and software programs. With a significant portion of the company’s business model set up around the idea of same-day service for much of its press brake and punching tooling, Wilson Tool needed to figure out how to carry out its processes manually and be as responsive as possible to incoming customer requests. While the company might not be able to meet one-day turnaround on customer orders, it was going to work to complete them as quickly and correctly as possible.

“We realized not too far into this that we had really become dependent on automation,” Lawless said. “It’s great to have it, but when you don’t have it, it’s tough. There’s only so much your manual systems can do to get you caught up.”

Wilson Tool did have access to email, and a phone line remained up. The following days were filled with meetings, where IT updates were provided and internal processes were put under the microscope as the company tried to figure out how products would be made, orders would be taken, and activities in support of those two critical areas would be carried out.

In the meantime, Lawless said he gave the IT team room to continue its investigation and recovery efforts. Working with the local cybersecurity firm TCE Strategy, the IT staff prepared a plan to see what was immediately recoverable and what sort of workarounds could be established to help the manual processes that would be relied upon in the coming weeks. It also began the slow and steady process of clearing servers, machines, and computers of any potential malware; installing appropriate safeguards on these devices; and slowly expanding device connectivity again. Haupt said the company used a red, yellow, and green marking system to designate if devices could be used. (Red noted that a device was compromised. Yellow meant that the device could be used for business, but that all of the safeguards and updates were not fully in place. Green signified that the device was clean, updated, and ready for network connections.)

After the cyberattack on March 13, the Wilson Tool information technology team worked long hours to bring the company safely back online.

“We never went dark,” Lawless said. “I’ve heard stories where other companies literally went down and didn’t conduct any business for days, if not weeks.” He added that Wilson Tool was back operating at near 70% capacity only days after they were locked out of their systems.”

“People just jumped in, and in some instances, it was like you didn’t miss a beat,” Haupt said.

Experience Pays Off

Jeremy Edson, marketing manager, Wilson Tool, said you could see how much the company had become reliant on automation when employees tried to piece together a manual process that replicated simple custom orders. Before the cyberattack, a phone call came into the sales desk, and the customer service representative could pull up all the customer information from the company’s SAP ERP software, take the order, and send it to the shop floor for processing and shipping, all with a few mouse clicks. It was about 15 minutes of work, according to Edson. The manual replacement process was a tad more complicated.

Luckily, some long-tenured employees remembered the days before SAP was ever implemented. They helped to create Microsoft Excel documents that called for the collection of the same information that the ERP software provided. It took more time to collect the information from the customer placing the order, but it was all there in the spreadsheet: contact information, shipping method, and product information. This document was kept on Microsoft’s Sharepoint cloud-based collaborative workspace. (Haupt said that Wilson Tool had been using a third-party conferencing and collaborative tool before the pandemic but decided to commit to the Microsoft platform because the company was already paying for it as part of the overall Office package.)

“It only took a couple of days to really iron that process out,” Edson said.

That process was repeated across the company. Machinists confident in their manual skills tackled work much like their predecessors from a couple of generations ago. Front office personnel pulled documents that they had kept on their local hard drives, off the company’s network, and used them to flesh out manual processes that replaced those once done by software. Part designers worked with a few guest licenses provided by SolidWorks, along with additional CAM partners, to do the best they could with incoming custom orders.

Lawless said it’s not uncommon to find employees with 20 years’ experience among the 500 that work in White Bear Lake. In fact, one just celebrated 49 years of service. All of that experience was put to good use in the days after March 13.

“We were able to draw from all of those experiences,” Lawless said. “We had engineers that stayed late and helped to build Visual Basic Scripts [computer code] that alleviated some very manual steps. Retired employees came back to help out. Everyone pitched in from top to bottom. Everyone rallied to the cause.”

In the days after the cyberattack, Wilson Tool held meetings at the beginning of each of the three shifts. Employees were provided updates on systems recovery progress, and a discussion was held about incoming orders and what could be done to address them. That kept orders moving through the facility, even if they didn’t proceed as fast as they did when all computer systems were up and running.

By March 30 Wilson Tool had access to its ERP system. In April most of the shop floor machinery was cleared and running.

Because Wilson Tool was locked out of its software systems, it set up a temporary staging area in its Tech Center where forms could be staged to assist with the manual processing of customer orders.

About Those Consulting Parties …

The actual ransom demands were met quickly. Lawless said that the consultants, the attorneys, the insurance company, and the insurance company’s attorneys handled the negotiations. The cyberattackers are very sophisticated when it comes to handling these situations because it’s a business model for them. If the victims in these situations never get access back to their data and systems, then no one would ever pay the ransoms.

Having said that, it’s not all smooth sailing when a company gets access to their systems again.

“You don’t get all of your data back once it’s been encrypted. Not all of your systems come back,” Lawless said. “We estimate that we got 80% of our data and systems fully recovered, where the other 20% got lost in the de-encryption process.”

As with many things, attorneys and insurance companies advise you to keep a low profile and not to be too public when discussing a cyberattack. While silence tends to be a good strategy in many instances, Lawless said that the company wanted to be as transparent as it could be with its customers. If someone had a question why a piece of tooling would take seven days to be delivered instead of one, the customer deserved to know the real reason for the delay.

“When we told our customers what was going on, what we started to see were customers becoming more supportive and empathetic of our position,” Lawless said. “Before when we weren’t saying too much, they were just getting frustrated.”

Lessons Learned

Of course, the primary goal after the cyberattack was a return to some sort of semblance of business. But even from the beginning, forensic investigators wanted to find out how the infiltration into the Wilson Tool systems was made. The answer would provide an important lesson in preventing such an episode from happening again.

After tracing back how the lockout might have developed, the investigators discovered an unpatched system. A patch is a system upgrade sent out by a company to address a security vulnerability in computer software code. Some people view them with skepticism for various reasons, such as fears of being spied on, but in reality these updates help to keep bad actors from exploiting shortcomings in code design and gaining access to third-party networks.

Haupt said that Wilson Tool was notified about a patch needed for something on its network in the summer of 2021 but did not immediately respond to it because of turnover in the IT department. The infiltrator used a two-week window to exploit that vulnerability and gain access to Wilson Tool.

He added that the forensics investigators weren’t sure if the original hacker was the one that carried out the cyberattack or if that person turned the access point over to more professional hackers. That’s how sophisticated these cyberattacks can be.

Once inside, the hacker took his time, looking for ways to move within the Wilson Tool organization, seeking out passwords to accounts, and working to get administrative rights. That’s one of the reasons that the takeover was a wide-ranging as it was.

Stacks of paper wait to be manually scanned and attached to customer orders.

“So the hackers are trying to find these vulnerabilities. That’s their goal,” Lawless said. “Once they find the vulnerabilities, they exploit that to get to your data.”

Haupt said that Wilson Tool is looking to be more aggressive with updating systems and installing security patches. Moving more software programs to the web will help. More use of extended detection and response tools, which collect and automatically correlate data across a system, will help detect threats faster and quicken response times.

But there are some challenges that come with the more aggressive policing. Wilson Tool is an international organization, and it behooves the company to have similar computing tools across the enterprise. That way, scans can be run with the confidence that they are covering all assets in the company, no matter where they might be located, and that patching is complete with no outliers. It’ll take some time to have that sort of consistency across the entire organization, Haupt said.

Another concern is that older machines have their own older embedded controllers with their own software. Sometimes the software on this older equipment is not updated regularly, and if these devices are connected to a company’s network, they present a potential entry point for hackers. Sometimes these older control software systems are patchable, and other times they are just too antiquated, Haupt said.

An organization can create a computing environment where these machines can operate and remain independent from the main network, but is that really what a modern manufacturing company wants as it tries to leverage manufacturing data to become a more efficient manufacturer? Lawless said this has led to some discussions about what sort of equipment Wilson Tool should have on the shop floor. Perhaps keeping that older piece of machinery around because it’s paid off and functional is not the best idea if that equipment is a potential access point for a cyberattacker.

Those that access Wilson Tool systems also have new requirements to gain entry. They must have at least 10-digit passwords with the mandatory use of numbers and symbols. There is also multifactor authentication.

Wilson Tool also is going to move its backups offline. That was on the to-do list before the cyberattack and obviously has been prioritized since then. Now if such a thing were to happen again, the company could just start over using the previous day’s data.

What about the actual ransom? Lawless didn’t reveal the ransom paid but acknowledged insurance didn’t cover the entire amount. The reality of the situation now is that Wilson Tool needs to find a new policy after having enacted its old policy.

Considering all that happened, Lawless is optimistic about the future of cybersecurity. He said insurance companies are going to be better educated on the subject and demand more from their clients, such as the need for firewalls and established backup procedures, before they offer cyberattack coverage. As the industry grows more confident, more companies will step forward and offer coverage.

He’s also confident that Wilson Tool is in a better place than it was before the cyberattack. Despite the difficulties of rebounding from being locked out of their systems, he saw the company rally together and keep the doors open for business.

Even locked out of its systems in the days after the cyberattack, Wilson Tool was still able to process customer orders and manufacture products, such as press brake tooling.

“Along with the workforce stepping up in every manner, we’d like to thank our customers for their understanding and unwavering support,” Lawless added.

Wilson Tool can’t change the past, but Lawless said he hopes he can help others avoid an unpleasant future. If they haven’t looked at the vulnerability of their own computing systems, they should probably do so sooner rather than later.

Manually processed paperwork needed to be scanned and saved to the company’s ERP system once it was safe, secure, and accessible.