One year ago, Packaging World published a two-part series on the topic of plant floor cybersecurity for the consumer packaged goods industry. Part one explored the nature of the cyber threat and why it should be on a packager’s agenda. Part two suggested resources and strategies that could be applied to turn discussion into action to address the cyber threat on the plant floor. (Both articles may be downloaded as a single reprint at http://pwgo.to/cybersecurity_reprint.)
As a follow-up, this article provides a high-level update to the cybersecurity landscape and explores if and how packagers and packaging machinery suppliers are actually addressing the plant floor cyber threat. For this update, we have circled back to the resources identified in the original series and are using the results of an August 2016 online survey of packaging machinery suppliers and packaging machinery buyers. This survey queried respondents on their experience with plant floor cybersecurity and on their use of the strategies identified in last year’s articles.
What’s the big picture?
We all know that in the big picture, cybersecurity remains a problem on the national scene. Since our original series in the Fall of 2015, Congress, the President, and numerous government agencies have been at work on the problem.
In 2015, the Food and Drug Administration, the Department of Agriculture, and the Department of Homeland Security updated the Food and Agriculture sector-specific plan for critical infrastructure. This plan cites as a key accomplishment: “The Food and Agriculture (FA) Sector conducted the Cybersecurity Assessment & Risk Management Approach (CARMA) to critically examined cyber threats, consequences, and vulnerabilities from farm-to-fork to better identify and manage cyber risks.” DHS points out that there are more than 200,000 registered food manufacturing, processing (many with packaging), and storage facilities in the US and related to these, the FA Plan says: “One area of interest for the FA Sector is the use of Industrial Control Systems (ICS), such as Supervisory Control and Data Acquisition (SCADA), by many food production and processing facilities. With the vast majority of ICS developing to enhance connectivity and remote access, the vulnerability of these systems to cyber threats needs to be better understood. As the FA Sector becomes increasingly reliant on technology, the sector will continually revisit the issue of cybersecurity.” The plan establishes the Food and Agriculture Sector Coordinating Council whose membership includes many CPG/food companies and associations.
Last December, Congress passed the Cyber Security Act of 2015 after two years of effort by the National Association of Manufacturers (NAM) and others to encourage legislation that would enable real-time sharing of cyber threat information on a voluntary basis without creating new and burdensome regulatory restraints. Early in 2016, President Obama signed two new executive orders on cybersecurity, one creating a Commission on Enhancing National Cybersecurity consisting of business, technology, national security, and law enforcement leaders; and a second creating a Federal Privacy Council to focus on issues of privacy in government databases. In April, he issued a Presidential Policy Directive to strengthen the security and resilience of critical infrastructure against both physical and cyber threats. Again on July 26th, he issued PPD-41, a Directive on coordination of responses to cyber incidents in the public and private sectors, including the food industry.
On August 10th, the Commission on Enhancing National Cybersecurity announced, in the Federal Register, a 25-day period of public input on topics of relevance to packagers including the Internet of Things (IoT). According to NAM, the IoT (also frequently referred to as IIoT, or the Industrial Internet of Things) is the term that they use to describe the systems and devices subject to cyber attack on the manufacturing floor. This public input period is part of a process leading to a commission report to be issued before the end of 2016.
Also in August, the Information Sharing and Analysis Organizations Standards Organization (ISAO SO), funded by the Department of Homeland Security (DHS), closed its public comment period for two documents: one setting out guidelines for the formation and operation of an ISAO and the other dealing with policies, principles, and the establishment of trust between government and private ISAOs for the sharing of information.
Government is clearly trying to get a handle on this problem, and is establishing the legal framework, policies, guidelines, and standards within which both government and private enterprise may act together and separately to protect the digital infrastructure and intellectual property upon which our safety, security, and economy depend. However, frameworks, policies, guidelines, and standards are not solutions. Solutions will come through the efforts of the private sector.
Who is thinking about it?
In the private sector, ISA published a whitepaper in January of 2016 written by Joseph Weiss and entitled “What Executives Need to Know About Industrial Control Systems Cybersecurity.” The executive summary of this paper echoed the advice of this magazine’s series, saying: “A company’s Board of Directors and executive management must continuously and meticulously identify, categorize, and mitigate risks to the organization’s success resulting from cyber attacks. In many cases the largest risk to the well-being of your company, your people, your processes, and your profits may be the compromise of your Industrial Control System—not a data breach.” The idea that manufacturing cyber security is a Board-level issue was echoed by individuals interviewed for this update and in the comments provided by some responding to our survey.
While our survey probably did not reach any CPG board members, it did show that cybersecurity is on the minds of some, but not all of Packaging World’s readers. Forty three percent of packagers reported having a plant floor cyber security program in place. For 64% of the companies with a program, it is managed by the IT department, for 22% by engineering, and for 14% by other executives. While we can’t tell from the responses we received how diverse the cybersecurity teams that these packagers depend upon may be, the conclusion of the same ISA whitepaper highlights the need for IT, engineering, and operations management to be united in the cybersecurity effort. Mr. Weiss concludes: “Culture, knowledge, and experience gaps exist between IT and Operations personnel in most companies, and the coordination of these functions with guidance from a team of Industrial Control Systems Cybersecurity Experts is critical to the success of a comprehensive cybersecurity program.” I would add that cybersecurity is the responsibility of every employee at every level in the organization, and therefore needs strong leadership from line management.
In the broader manufacturing space, the National Defense Industries Association posted this in a July 18th blog: “IT Professionals Still Not Taking Cybersecurity Seriously.” This conclusion was based upon a survey sponsored by Dell of over 600 global IT decision makers in companies with more than 1000 employees who were asked to think about digital transformation technologies such as the Internet of Things when responding to the questions posed to them. Security is considered too late in the process according to 76% of the respondents.
Who is being affected?
Six percent of the packagers responding to our survey reported some sort of known cyber attack against their plant during the last two years, while nearly 30% responded that they didn’t know if their plant had been the victim of cyber crime or not. A similar number of packaging machine suppliers were aware of cyber attacks against their customers, with half of those involving equipment supplied by them. One responder commented: “Unfortunately, we often don’t hear much from our customers unless they need our help.” That is probably a very common situation, and would explain a lower level of awareness among machine builders than among packagers. It would also seem reasonable to conclude that among the 57% of packagers who report that they have no cyber program, it would be difficult to know if an intruder had breached the system, unless the breach led to some confirmed damage. The Verizon 2016 Data Breach Investigations Report shows that often no harm is done. Of 171 confirmed manufacturing incidents reported by Verizon where integrity, confidentiality, or availability was compromised, only 37 of these, or 22%, resulted in confirmed disclosure or disruption. Attackers may simply probe a system to verify that access may be obtained for a later attack. If attackers are involved in cyber espionage, such as looking for product formulas, they may be careful to not leave any trail. Plants that are not actively monitoring internal and external traffic may have no way of knowing they were breached. A cyber infection may lie dormant without detection for a long time. In August, both Kaspersky Lab and Symantec announced finding malware on dozens of major systems around the world that has probably been in place but gone undetected since 2011 or earlier.
Some have commented that 6% seems like a small number compared to the impression of attacks covered in the media. But there are thousands of small to medium manufacturing facilities in North America, and if 6% are being attacked, that represents a lot of plants, a lot of jobs, and a lot of product potentially affected. Utility systems, power plants, pipelines and other critical infrastructure are no doubt suffering a higher frequency of attack and these attacks are more widely known and, so far at least, have greater impact. The McAfee Labs Threat Report for June 2016 reported that in December of 2015 the Ukrainian power grid was attacked (details provided in Feb 25 ICS-CERT alert), leading to the loss of electricity for 225,000 people. Also, in the first quarter of the year, three hospital systems were held for ransom by ransomware. One packager responding to our survey was also the victim of a ransomware attack. McAfee screens 50 billion queries per day for its customer base, and of those, more than 12 million potentially hazardous events are identified every hour. Mobile devices are becoming an increasing concern among these events, with new methods of infection and an increase of more than 50% in the amount of mobile malware detected in the past 6 months. This is of particular concern as it affects mobile HMIs on the plant floor and the use of “bring your own devices” (BYOD) by employees, vendors, and contractors.
One survey result that stood out as potentially alarming is that only 34% of packaging machine supplier respondents was aware of a formal company policy to protect the security of their customers’ data and network. We did not ask how many suppliers have internal cybersecurity programs to protect their own operations, nor did we ask how many packagers have programs to protect the data up and down their supply chains. But if packaging machinery builders as an industry want to be seen as part of the plant floor solution and not as a potential source of the cyber problem, they are going to have to address their customers’ data security with policy and employee communications in much higher percentages.
What are packagers doing?
We asked about both strategies and practices that are being employed to address the cyber threat in packaging plants. Remembering that only 43% claimed to have a cybersecurity program, 52% do have a policy on authorizing additions of equipment to their networks. But many companies had that long before the word cybersecurity was ever uttered inside a manufacturing site. After that, the most frequently used strategies are maintaining an up-to-date inventory of all digital equipment; maintaining mitigation, response, and recovery plans; using a secure-by-design approach; and treating cybersecurity like a personnel safety or sexual harassment issue with frequent reminders and communications. Among the least-used strategies were those involving sharing and building upon industry standard approaches. Working with local FBI InfraGard chapters, sharing data on cyber-attacks, using the resources of DHS’s ICS-CERT, or following ISA99 / IEC62443 cybersecurity standards were each cited by less than 10% of responders. There is either a lot of duplicated effort going into recreating solutions or people are ill-informed about what resources their companies are actually using.
The Packaging World survey did not ask directly about the NIST Cybersecurity Framework, but it did ask about major recommendations from the framework. Across industry this framework is acknowledged as the most commonly implemented strategy for addressing cybersecurity; but it is also recognized as expensive and time consuming to implement. In June, NIST announced plans to issue a draft update to the Cybersecurity Framework in early 2017 based upon feedback obtained from a request for public input and a workshop held this past April.
According to Brian Raymond, Director, Innovation Policy for the National Association of Manufacturers (NAM), the industry’s response was largely “to keep doing what you are doing.” But Raymond commented further that the challenge that DHS (Department of Homeland Security), NIST (National Institute of Standards and Technology) and others face is bringing cyber solutions to smaller manufacturing companies. Several survey participants commented on their small size being an obstacle. So, among other actions that NIST plans to undertake, they are committed to working with small and medium size businesses by leveraging other programs such as the Hollings Manufacturing Extension Partnerships (MEPs) that many packagers or machine builders should have access to in their local areas. The NIST report also states that dialog with regulators will continue to emphasize the importance of the voluntary nature of the Framework. NIST is suggesting that stakeholders should customize the framework to their particular sector, publish sector-specific profiles or crosswalks, advocate for the Framework within the sector, publish sector-specific case studies, and share experiences with NIST. This leaves the question open about how granular sectors should be and what sort of organizations representing a sector should take on this task. Is the task the same for the corporate front end as it is for the shop floor? Does the front office of a manufacturer look like a financial institution from a cyber perspective? And will the same strategies and practices work for the shop floor? As mentioned earlier, some experts think not.
Interestingly, manufacturing stands out from other industries in a couple of ways. It has a relatively low number of cyber incidents compared with government or the finance industry. But manufacturing stands in number one position among all sectors in the type and nature of its breaches, with 47% of breaches being aimed at cyber espionage and with 33% of incidents falling into an undefined category of “everything else” for the method of attack. Both of these statistics point to the need for manufacturers to take an independent look at cyber security, because their vulnerabilities and attack vectors are unique. The technologies that are subject to attack are no doubt broader in scope and in age than those we hear about in finance and general business systems, and largely fall under the purview of engineering departments rather than IT departments. With large investments in product data management and quality systems, many manufacturing executives would be both surprised and alarmed to learn the extent of intellectual property knowledge contained within shop floor systems and with the dangers that shop floor systems may expose to product quality and safety.
NIST Engineering Lab continues to work on profiles to help implement the framework for various manufacturing operations. Their goal is to make implementation actionable for small and medium sized manufacturers without becoming overwhelming and without producing negative performance implications. A draft profile was published for manufacturing in April 2016. The NIST document separates manufacturing processes into process-based and discrete-based, separates process-based into continuous and batch, and acknowledges that some facilities are hybrids of discrete and process. It remains to be seen if these distinctions affect cyber plans as they affect other issues involving people, processes, and technology within manufacturing.
Sector-specific profiles require sector-specific domain knowledge that resides within particular sectors, and which is often expressed through an industry trade association. We asked both packagers and packaging machine builders if they thought that the industry associations that they belong to are doing enough to assist companies in dealing with cybersecurity issues. Overall, the results were split with fifty-six percent of the suppliers and forty-five percent of the users saying that associations were doing enough, but the question elicited quite a few comments. Some indicated that the survey was the first time they had heard of this issue in their industry. One said no one was discussing it, some felt that it is not a trade association issue but should be handled internally, and others figured that advice from industry associations would be beneficial.
PMMI is in the process of establishing an OpX Leadership Network team called the Secured Vendor Access Solutions Group whose purpose is to develop a framework for methodologies for secured vendor access followed by the creation of guidelines, protocols, and tools to enable 3rd party remote access to manufacturing equipment in a secured manner. The purpose of this effort is to improve the operational excellence of CPGs through added supplier value by providing remote diagnostics and assistance. The task force is just beginning to gather thoughts about how it will proceed. Perhaps this will lead to very specific stakeholder profile that will be a suitable complement to the work being done by NIST.
Who is sharing?
It was mentioned above that government has taken several steps to increase sharing of information on cyber attacks, including the passage of the Cybersecurity Act of 2015 and the work that Department of Homeland Security is doing through ISAO SO on developing standards for organizations who share information. Both of these are an acknowledgement that we are not yet sharing or collaborating enough.
McAfee reported in March that of the companies that have cyber security professionals, 91% would like to receive industry-specific cyber threat intelligence, but only 16% actually share their intelligence. In our survey of packagers, sharing of information was not high on the list of steps being taken to address cybersecurity, with less than 8% of respondents admitting to sharing data with others in the industry through a trade association and less than 3% participating in the FBI’s local InfraGard programs.
Last year we mentioned cybersecurity as a potential threat to be dealt with in regard to the Food Safety & Modernization Act (FSMA) Food Safety Defense Plans. In 2016, new cybersecurity concerns emerged around FSMA when two US House Subcommittee chairs questioned if the FDA has adequate cybersecurity practices in place to protect the trade secrets and confidential commercial information that must now be shared with the FDA as a result of FSMA requirements. In January of this year, the FDA issued draft guidance on cybersecurity for medical device manufacturers, but, at least in the public record, and despite the establishment of the Food & Agriculture Sector Coordinating Council, FDA and USDA seem very quiet on cybersecurity practices with regard to potential threats in the food or drug supply chain. Perhaps we should ask if any real sharing or collaboration on cybersecurity solutions is taking place among members of this Council, many of whom are packagers.
Why are packagers connecting (or not)?
Packagers and their equipment suppliers have more than a casual mutual interest in cybersecurity. As our survey identified, both groups see benefits to external data connections to equipment on the factory floor. Sixty-three percent of equipment suppliers cite economic benefits to both themselves and their customers as the result of being interconnected, with only 7% seeing no benefit at all. Seventy-five percent of suppliers already offer services that require an external data collection. The vast majority of these services relate to remote troubleshooting, maintenance, changeover, and optimization. Remote troubleshooting by the vendor is the #1 potential benefit of plant floor remote access as seen by the packagers, but with slightly fewer than half citing this as one of the top 2 reasons for allowing remote access. Other benefits from remote access are for engineering or management to monitor operations from outside the plant, for corporate engineering or IT to assist with plant issues, or to improve overall equipment effectiveness (OEE). Each of these was viewed as a top reason to allow external data connections (although for internal use only) by more than 25% of respondents.
Packagers and packaging machinery supplies have slightly differing views on the obstacles to allowing remote data access by vendors. Security is the number one obstacle cited by 44% of users with lack of communications capability in machines number two at 30% and cost and workforce skills shortage tied for 3rd place at 28% each. From the suppliers’ perspective, the number one obstacle is lack of plant network infrastructure in the customer site at 44%, strict customer IT policies number 2 at 39%, cybersecurity fears number three at 37%, and customer reluctance to share data number 4 at 34%. It is interesting that customers’ focus is on security, not IT policy, while suppliers tend to focus more on the IT policy than on the security concerns that likely underlie that policy. One packaging machinery supplier responding to our survey commented: “It’s going to streamline their processes and benefit their bottom line, so it’s really a no-brainer to me. There are security measures in place to safely transmit data .... It’s mind-boggling that more customers aren’t jumping on the remote access train.”
How are packagers connecting?
While it is true that there are security measures available that would be quite adequate if everyone in the world were honest and moral, whether or not these measures are adequate in a world of espionage and cyber-terrorism is another question. So the jury is still out on whether it is mind boggling that packagers do or do not jump on the remote access train. So, how many are getting connected and how are they doing it?
Our survey shows that 21% of end users allow NO remote access connections, 8% allow remote access only for their own staff, and 1% only for staff and systems integrators. By policy, a total of 30% of packagers prohibit ANY remote access by equipment suppliers. Twenty-two percent of equipment suppliers confirm that they have customers that permit no access. Twenty percent of the responders say that their company has no policy with regard to remote access, 13% didn’t know if there was a policy, and 7% allow remote access under the control of the suppliers. The remaining 30% who do allow remote access give temporary access under supervision of internal staff (17%), or they provide a monitored and dedicated portal (20%) through which the vendor may connect. Fifty-one percent of suppliers confirm that they may not make dedicated connections to the customer. The fact that 28% of users report having a single point of access to all equipment in their facility and 25% monitor all data traffic into or out of the facility hints that there may be some (less than 10%) preparing for or considering additional remote access.
Drilling down a bit deeper, companies also have prohibitions on local access that affect the digital relationships between customer and supplier. USB ports on installed equipment are disabled by 9% of responding companies and 15% of suppliers report seeing this policy in use. Making the life of field service more challenging, 25% of users either prohibit service personnel from bringing in their own data equipment such as laptops, jump drives, and smart phones or they require a security scan of that equipment. Again, the supplier survey confirms the use of these practices. A minority of suppliers (44%) report experiencing no cyber-security difficulties when working in customer’s plants. And putting the shoe on the other foot, 15% feel that the customer’s lack of a cyber policy endangers the equipment that they take with them into the plant. This seems like an opportunity for customers and suppliers to work together to benchmark how defense, energy, and nuclear facilities handle these issues and to develop some industry-wide standard practices.
Some concluding thoughts
Packagers and packaging machine suppliers agree that there are good and valid reasons to have remote data access to plant equipment. Packagers view it more as a benefit for their own internal use and suppliers see it primarily as a means of providing faster and more cost-effective customer service. But concerns over cyber security are holding many back from allowing remote access and creating obstacles even for local access by suppliers.
Government is taking the issue seriously, although not as seriously for CPGs and machine builders as for utilities, defense contractors, and the financial sector. Legislation, executive orders, and agency activity have contributed much to the discussion over the past year as our legal and physical infrastructures adjust to new challenges and demands. Those who have thought about this issue see it not just as a technical matter for IT and engineering, but as a board-level issue involving all employees. Those who have analyzed the issue for manufacturing realize that perhaps the greatest threat lies within the factory floor, not in the business systems.
Even so, a majority of both packagers and machinery suppliers have not implemented cyber security policies for the shop floor. Those that have appear to be taking limited advantage of the resources being made available by government, trade associations, and the private sector, and they have been very reluctant to share experiences, both good and bad. A few CPG manufacturers and a very few equipment suppliers have embarked upon significant plant floor cyber security programs. Perhaps PMMI’s OpX Leadership Network will be an initial small step in creating collaborative solutions for the problem.
My conclusions are very similar to those from a year ago. CPGs and the machine industry have been given the legislative and bureaucratic latitude that they requested to act under voluntary guidelines to solve the issues of cyber security for their manufacturing sites. They need to take more individual and collaborative steps to do so and share the results with all of us.
