Developing the Framework for Enterprise-Wide Cybersecurity Standards
As an organization grows and its distributed production facilities become more connected with Internet-based functionality, ensuring the security of both the industrial and enterprise networks becomes increasingly difficult.
Working together with this organization’s corporate IT department, Cybertrol helped a Fortune 500 company develop a phased approach to analyzing, assessing, and securing the industrial networks across a number of their distributed production plants while also making the plant floor data easily remotely accessible.
As a manufacturing organization grows, it is likely that growth is not centralized and production facilities are added throughout the United States, or even around the world. As production facilities are acquired, or new facilities are built, it can become increasingly difficult for the corporate headquarters to have a full understanding of all the assets and network procedures in use at each facility. Additionally, since expansion usually happens incrementally, it is unlikely that a standardized approach was taken to develop plant floor assets like industrial networks.
Recently, issues around industrial network security and remote accessibility arose for one of America’s premier agribusiness and food companies as it had grown over the last 100 years to roughly 65 manufacturing and process operations facilities spanning the US. In addition to growing substantially, the company also started to adopt more modern procedures that were leaving their older operations open to possible cyber vulnerabilities. For example, in the past, corporate production managers would call each plant to provide daily production instructions and the plants would schedule production accordingly. However, as the ability to send this information over the Internet emerged, the organization hesitantly brought its manufacturing facilities online with their corporate network to streamline this process.
Today, all the company’s production facilities are connected to the enterprise network in some fashion. However, these connections are not standardized. While this transition to automating daily production information was much needed and provided numerous efficiency gains, some new issues arose.
First, because there were not standardized security measures implemented as production facilities were brought online, corporate IT realized that Internet connectivity could pose security threats to the industrial networks, and by extension, production targets. Second, there was generally a lack of documentation available for each facility, which meant there was not a good picture available of the assets deployed in each plant and how each network was structured. As a result of the lack of asset standardization, network structure, and documentation, they also found it difficult to support their distributed facilities when issues arose.
Working with a Third-Party Expert to Identify and Understand Potential Industrial Cybersecurity Threats
As the need to bring production facilities online and connect with corporate grew, it was clear that the company needed to better understand the industrial networks in place throughout each of its facilities. They also knew they needed to get a holistic picture of how their production facilities were communicating with their corporate office. However, they knew they did not have the resources internally to start this project in a timely and cost-effective manner.
Since the company was already engaged with Cybertrol’s automation department on a variety of projects at their facilities, they decided it would be a good fit to expand their working relationship Cybertrol’s dedicated industrial IT department to investigate the industrial network infrastructure at each of their production facilities across the US. To do this, Cybertrol worked with corporate IT and onsite operators and management staff to develop a phased approach to analyzing, understanding, and securing the company’s industrial networks.
The first step in this process was to establish a baseline, which included an OT network assessment at each facility to determine the equipment in use and how that equipment was connected around the plant floor, to external Internet points and back to the corporate network. This was a necessary first step to understand the extent of their network and also so they could know exactly what they needed to protect and what equipment may be necessary to do so.
Next, since most production sites had, at best, dated materials showing what was in use on the plant floor, after each site visit Cybertrol engineers created appropriate documentation, starting with a site visit summary report. Cybertrol walked through these reports with the organization’s corporate IT security team to discuss the issues they found at each plant and make recommendations for remediation.
The final step was to perform remediation on the network to resolve the connectivity and security issues that were identified, which resulted in working with corporate IT to develop a long-term enterprise-wide cybersecurity strategy. To do this, we worked with the company’s corporate IT department to learn about their corporate standards to see what could translate to the industrial networks versus what new procedures needed to be put into place.
Implementing a Cybersecurity Strategy: It’s All About Risk Mitigation
When it comes to any connected facility, there are a variety of risks that could potentially exist within the industrial network. These risks can range from external threats to the network to unintentional errors made by employees to possible prolonged downtime resulting from the inability to quickly fix a network issue. During site visits, we identified risks across all these areas, which required collaboration with the client to develop solutions to mitigate these risks.
Step 1: Securing Industrial-Enterprise Network Connections
One commonality we found across the distributed facilities was that there was always some type of established connection between the industrial and enterprise networks used to send and receive production information. However, one of the biggest issues we found with this was that there was not a standardized method for securing this connection, and some facilities did not have any security measures in place at all.
Thus, one of the first steps we took was to develop a standard method for securing these industrial-enterprise connections. To start, we developed a demilitarized zone (DMZ) between the networks at each facility. We had a main point of contact within the organization’s corporate IT office that we worked with to establish a control system security standard for the DMZ that specifies the traffic that can pass between the enterprise and industrial zones. On each side of the DMZ, firewalls were put into place to serve as additional points of protection (Figure 1).
Figure 1. This diagram shows a representation of the structure we used to secure the network at these facilities using a DMZ and firewalls we installed.
At each production facility we also installed honeypot software that corporate IT was already using. This software serves as a discoverable device on the network that can get an indication of who is accessing the network or show if someone is probing at the network to try to see what they may have access to.
Step 2: Implementing Strategies to Secure Connections Within the Industrial Network
Since an industrial network environment is much different than what corporate IT is used to working with, their IT managers did not have a good idea of what issues they should be looking for on the plant floor. For example, during our site visits, we discovered that employees at some sites were connecting to wireless access points from unsecured personal devices. At other sites, we saw that network switches appeared to have been rapidly deployed and were either unprotected or using default passwords, making them easily discoverable. These security breaches on the plant floor posed risks for both the industrial and enterprise networks.
Upon making these discoveries, the company’s corporate IT managers leaned on our experts to provide guidance from a production network standpoint. We started by working to bridge the gap between process knowledge and the knowledge of how production networks function, and then worked together to align our solutions with their current corporate security standards and initiatives.
In addition to the changes we made to prevent external security threats, we also recommended a variety of changes within each facility. These recommendations sometimes included small changes, such as incorporating redundant switches to prevent networks from going down due to a simple error like someone connecting to the wrong port or pulling the wrong cable. In other cases, we made recommendations that involved updating old equipment with more modern equipment that had more security features. And, in other instances, we suggested they leverage the security features on equipment they already had in place.
Step 3: Improving Secure Remote Support Capabilities
When we began working with this company, we realized there was not a defined support point or corporate resource dedicated to working on network issues at production facilities. Therefore, we deployed a common industrial PC onto their networks that can be used for remote support purposes (Figure 2). However, because we also added in the DMZ for security, we had to ensure corporate users and vendors responsible for supporting equipment on the production floor could get through the DMZ to connect to the network for remote support.
Figure 2. This industrial PC serves as an outside entry point into the network for remote support purposes.
Thus, another software solution we worked with was Apache Guacamole, a remote desktop gateway solution. With Apache Guacamole, remote vendors and corporate users can connect to the necessary production resources with a termination in the DMZ, which is a requirement for the corporate security standards we developed. In the event of a network issue at a remote facility, now corporate IT can respond quickly and limit downtime.
Apache Guacamole can also be used to limit the functionality any user has when remotely accessing the industrial network for any reason. For example, if someone needs to view reports remotely, that person can be limited to just performing that function. This makes it easy for that person to get the data they need without having privileges to do things they don’t need to.
Step 4: Refining and Expanding Over Time
In the future, we will continue to work with this company to assess the remaining distributed production sites we have not had access to yet. We will deploy the security structure we already established with their corporate IT and will adjust as needed for current risks within each of these unique environments. Additionally, because of the flexibility of the solution we developed, adjustments can be made over time as processes evolve and risks change.
Benefits of a Modernized Approach to Cybersecurity
While there were numerous benefits to this phased approach to developing a cohesive cybersecurity plan for this organization, one of the biggest benefits is that the company now has a complete understanding of the equipment in use on the plant floor at the majority of its facilities. This makes troubleshooting network issues more efficient and limits the duration of downtime events, which has a direct business impact. Additionally, corporate IT and management can now easily and securely access networks remotely to view necessary data when needed. Vendors that need to support their equipment remotely and plant floor employees that need to see production reports at home also have a secure method for access as well.
Overall, Cybertrol worked with this organization to provide the resources and expertise needed to gain insight into a critical aspect of their business that they did not have the resources to properly focus on at this time. We gathered data from each production facility and turned it into a deliverable that maps out the industrial networks and assets within each production facility and how those assets are communicating with each other as well as the enterprise network. Based on what we found, we also provided guidance on what we considered to be the most pressing concerns that should be given the highest priority. As a result, we worked with the company to implement a variety of standards and tactics to start securing their network right away.
Contact us to see how our dedicated industrial IT/OT department can help you implement the most effective cybersecurity strategy, whether you have 1 or 100 facilities.
