Hall of Fame – Cybersecurity Industry Veteran Eric Byres

In the second part of Industrial Cyber’s Hall of Fame series of in-depth interviews with veterans from the industrial cybersecurity industry, we are proud to highlight the contribution by Eric Byres, founder and chief technical officer at aDolus Technology to the field of industrial control system (ICS), SCADA and industrial Internet of Things (IIoT) cybersecurity. Backed by several years of experience in controls engineering, security research, and corporate management, Byres has over his vast profession in the industry integrated deep technical knowledge and practical field experience with business expertise. 

Inventor of the Tofino Security technology, a widely deployed ICS-specific firewall, Byres founded the British Columbia Institute of Technology (BCIT) Critical Infrastructure Security Centre. He has directed government security agencies and energy companies on protection for critical infrastructures, headed the ISA SP-99 Security Technologies Working Group, represented Canada for the IEC TC65/WG10 standards effort, and testified to the U.S. Congress on the security of ICS in national critical infrastructures. 

Q&A:

Industrial Cyber: Who have been the strongest influences in your career?

Eric Byres: At the personal level, my strongest influences came from working with senior business executives like Paul Dorey at BP, Peter Maxwell at Coopers/Eaton, and Wolfgang Schenk at Belden. They led very large and effective teams and did it with warmth, compassion, and vision. It taught me that if you treat your team with respect, the team will do amazing things to help you achieve the goals you define.

At the more philosophical level, Thomas Kuhn and his landmark book on science, “The Structure of Scientific Revolutions,” impacted how I look at engineering and technology problems. Kuhn introduced the then-revolutionary concept of the ‘paradigm shift’ in science. He also pointed out that many of the major scientific breakthroughs in history were initiated by outsiders to a field because they brought new ways of looking at a problem. So when I am faced with a security challenge that is resisting an easy solution, I try to imagine how someone from a completely different field, like biology or psychology, might approach the problem. It was this sort of thinking that was the genesis of the Tofino Firewall.

Finally, “Everyday Zen” by Charlotte Joko Beck has helped keep me sane when everything seems to be falling apart around me. I still have a ton to learn. 

IC: How have you been able to blend the roles of a security researcher, controls engineer and business executive? Did they all just fall in place, or did you have to make some compromises?

EB: Certainly holding multiple roles and responsibilities involves compromises, but I find that taking advantage of the unexpected opportunities that pop up in a multi-discipline career is the rewarding part. I learned early on that I could leverage the knowledge obtained in one role to benefit the other roles. And because of my blend of knowledge and interests, I could see answers to problems that dedicated experts missed. It has been a fun and exhilarating journey.

Now to be clear, I never started out wanting to be an ICS cybersecurity expert. I graduated from university as a geological engineer, soon migrated to computer programming in mining, and then moved on to control system design in pulp and paper. The security of industrial systems only piqued my interest around 1996 when a ‘hack’ of a Foxboro DCS by an employee caused a lot of problems at a pulp and paper company where I was performing serial and LAN communications design tasks. I subsequently wrote up that incident as part of an article for ISA’s InTech magazine in 1998.

After that, I started getting calls from pulp and paper and chemical companies about security and decided to write a peer-reviewed paper on the subject. Unfortunately, it was rejected by several conferences as ‘not relevant’ until the IEEE Pulp and Paper Technical Conference let me present it in the summer of 1999 (E. J. Byres; “Protect that Network: Designing Secure Networks for Process Control”, Proceedings of the IEEE Pulp and Paper Technical Conference, Institute of Electrical and Electronics Engineers, Seattle, June 1999). Much to my amazement, it was awarded Best Overall Conference Paper for 1999. A little while later the IEEE Industry Applications Magazine asked if they could publish it. To my even greater amazement, it eventually won IEEE’s Outstanding Industry Application Article of 2000.

Despite the success of the paper, I didn’t think that ICS security was a career possibility. That didn’t happen until 9/11 occurred in 2001.

IC: What role would you describe the Stuxnet attack as having played in your career? Do you think your career would have been crafted differently if Stuxnet had not happened?

EB: It was the 9/11 attacks that really launched my ICS security career; Stuxnet just turbo-charged its trajectory. My partner Joann and I were trying to get Tofino Security off the ground when Stuxnet was discovered. Without Stuxnet, I suspect Tofino would have been another failed startup, and I would have moved on to another topic that interested me. But when Stuxnet appeared, suddenly company executives, government officials, and mainstream newspapers were asking about ‘SCADA security.’ I had both answers and a solution, which established both the Tofino Firewall and myself as ‘leaders’ in the cybersecurity field.

IC: What advice would you give a young Eric Byres? Which is your most treasured accomplishment so far?

EB: My most treasured accomplishments have been creating the teams at BCIT, Tofino Security, and now at aDolus Technology. I worked to bring the right people together, set the vision, and let smart people do their job. Then amazing things happened — things often beyond my wildest dreams. Take the Tofino Firewall: 25 incredibly clever and dedicated staff built a device that defined an entire product space — the OT/ICS firewall. And they made sure it was a quality product designed to last; there are not many technology products from the early 2000s that are still on the market today, but thousands of Tofino Firewalls still are purchased each year by engineers from around the world.

The advice I would give a young Eric Byres? Be a mentor, not a boss. I micromanaged far too much as a young engineer. Now I try to set a vision, give guidance, and get out of the way. That doesn’t always result in success on the first attempt, but in the end, amazing outcomes are regular occurrences and everyone learns a lot, including me.

IC: From our readers’ input, one of your many strengths is spotting and developing talent. If we are unable to identify and develop talented individuals, the skills gap will keep growing. What advice would you give to other leaders regarding how to mentor/develop talent to build a strong foundation for future success?

EB: As I noted earlier, my first piece of advice is: ‘Be a mentor, not a boss.’ I love my job, I love coming to work, and I want everyone on my team to feel the same way. A few months ago, one of the engineers on the aDolus team (who was in his mid-30s, so not new to the job market) told me, “Yesterday was the best work day of my life.” Hearing that means more to me than a good sales report because I know I’m building a team of new cybersecurity leaders when I retire.

I also want to hire motivated people who can solve problems rather than hire people with long resumes in OT security. Most of what we do in OT security is cutting-edge, so I can’t expect new hires to have done it all before. Energy, enthusiasm, and integrity are more important than experience in this field. If you can work hard and think critically the day you start, you can learn about security or OT on the job. That said, if someone has great OT or cybersecurity experience and is willing to mentor the team, I’m interested in talking.

IC: What advice would you offer younger professionals as they enter the industrial cybersecurity and supply chain security sectors today, and why?

EB: My advice to anyone entering any technical field is to follow that ancient Latin expression carpe diem. For example, when I started at the British Columbia Institute of Technology (BCIT), I didn’t have industrial security research in mind. Instead, I was planning to do network latency research for ICS communications. But that all changed on September 9, 2001, when the World Trade Center towers fell. Suddenly I was getting calls from government agencies and major oil companies interested in security research for ICS.

Our little team of 2 researchers grew to 14 by the time I left BCIT in 2006. The reason was simple: if you did a literature search for ‘security’ and ‘SCADA’ or ‘automation’ or ‘process control’ in 2001, you would find papers by Joe Weiss and myself — that was all there was. It really bothered me at the time as academics are supposed to reference other academic work, and I struggled to find much. One company even paid me to do a formal literature review. I eventually convinced some other academics in electrical engineering to start doing research on the topic, but I don’t think anything meaningful was published until 2004.

By the way, I wasn’t alone in those early days. Some of the real stars were the visionaries like Paul Dorey (BP), Evan Hand (Kraft), and Eric Cosman (Dow), who secured the funding for my sort of bleeding-edge research into OT security. But Aris Espejo at Syncrude was definitely the first to see the problem and get management to fund this research topic. He reached out to me less than 6 weeks after the 9/11 attacks and we had significant funding by late 2001. No one else was that quick to see the problem, and without Aris, BCIT’s program probably would never have gotten started so quickly. As an industry, we owe Aris a lot.

My other advice is to build a diverse set of skills: industrial cybersecurity and supply chain security professionals need an incredibly broad range of capabilities, including technical knowledge of security technologies and OT technologies, as well as soft skills such as communication, problem-solving, critical thinking, and leadership. We try to enable that at aDolus: for example, even the most junior intern is encouraged to give Lunch & Learn presentations to the entire team so they can develop their communications skills.

IC: What is the current state/health of the industry, and what do you see as the key issues that the industrial cybersecurity and supply chain security sectors face? What does the future hold?

EB: The industry is in a difficult phase right now. Thanks to Stuxnet, Colonial Pipeline, and similar OT incidents, control systems have caught the world’s attention. Before Stuxnet, nobody outside the automation industry knew what a PLC or an RTU was. In the last decade, everybody, including the bad guys, has become aware that (1) OT is incredibly vulnerable and (2) attacking it is extremely lucrative.

As a result, we now see numerous foreign agencies and criminal groups building the capability to launch Advanced Persistent Threats (APT) that are specifically designed to target industrial control systems and other critical infrastructure. These attacks are often carried out by well-funded nation-state actors and are designed to steal sensitive information or disrupt critical infrastructures. As for criminal gangs, we see ransomware attacks becoming more common in the industrial sector because they are relatively easy and profitable. As we saw in the Colonial Pipeline incident, these types of attacks can cause significant disruptions to operations and can be very costly.

At the same time, we’ve integrated IT, OT, and the cloud in ways we never expected 20 years ago. I look back to when I started in OT security: the vision was for a single gateway to be the only link between the OT and IT systems. In most large operations today, there are hundreds of external connections, many of them to the cloud. We have connectivity between the cloud, the IT systems, and the plant floor that we never planned for, which gives the bad guys new opportunities.

Finally, we have the software supply chain attacks. This type of attack targets weak links in the supply chain and can be used to gain access to industrial control systems and other critical infrastructure. It offers the attackers an incredibly good Return on Investment (RoI) on their evil efforts: it is far easier to attack an OT software supplier or take advantage of vulnerabilities in a common open-source component than to go after the intended industrial victim directly.

How does it work? Hack into one OT supplier’s development system or software download site and replace the good software with malware. Then allow the supplier to distribute the bad software to its customers for you. Now you have a multiplier effect where one poorly secured supplier gives you access to multiple victims. In the case of the SolarWinds attack, the attackers successfully penetrated the defenses of one software company and used that to gain access to 18,000 critical government agencies and industrial operations.

IC: As a leader in international standards development, what do you think will encourage accelerated adoption of these frameworks and how can we make sure they will work for us?

EB: Two things motivate organizations to adopt cybersecurity frameworks and standards: (a) demands from their customers and (b) getting hacked. I really hope the companies and people purchasing products use their checkbooks to demand that the industry follows best practices and standards quickly. Otherwise, the bad guys will drive adoption and that is never good.

IC: What scares/excites you the most about what’s next in the industrial cybersecurity sector? What do you see as the biggest threats and challenges, and how prepared is the industry as a whole to deal with the present threat landscape?

EB: The software supply chain challenge is the biggest threat to critical infrastructure today. That is why I switched my focus from designing OT firewalls to automating SBOM creation and analysis for OT. Until we get that under control, the well-financed APT groups will have the upper hand.

In terms of preparedness, the automation industry as a whole is taking steps to address all the threats, including the supply chain threat, but there is still a significant gap in terms of the level of protection provided to many industrial control systems. Many organizations in the industrial sector are still in the process of implementing basic cybersecurity measures and are not fully prepared to deal with the current threat landscape. This is partly due to a lack of understanding of the specific risks and vulnerabilities associated with industrial control systems, along with a lack of resources and expertise to devote to OT cybersecurity. I believe this will improve significantly over the next decade, but the industry is likely to have a few serious security events in the near future.

IC: Could you list three things that still keep you interested in industrial cybersecurity and SBOMs?

EB:  

1. Rapidly improving supply chain visibility: The Software Bill of Materials (SBOM) initiatives of the last few years are finally providing organizations with urgently needed visibility into their software supply chain and the components used in their industrial control systems. I’m getting to see how this increased visibility can truly help organizations identify and address vulnerabilities and better reduce the risk and potential impact of a cyber-attack.

2. The use of AI in cybersecurity: We’ve all seen how ChatGPT can answer complicated questions in a few seconds — it is amazing. I’m experiencing the same amazement when my data science team shows me how they use AI in the generation and management of SBOMs and the correlation of vulnerabilities to those SBOMs. For example, consider looking for a vulnerability associated with a GE-Fanuc PLC; you have to be an old industry hand like me to remember to search vulnerability databases for “Emerson” rather than “GE Fanuc” because GE changed the name of the division to “GE Intelligent Platforms” in 2009 and then Emerson bought that division in 2019. AI can do that task for you for millions of products in seconds. It is solving tedious but important tasks that security professionals currently do, allowing those professionals to focus on more important things.

3. The evolving security regulations: Industrial cybersecurity and SBOM standards are rapidly maturing and becoming a requirement for compliance with regulations such as NIST SP 800-161, NIST SP 800-160, and IEC 62443. It is not often you can see new OT initiatives and standards evolve this fast — usually they take decades — but with software supply chain security, real progress is occurring weekly.

IC: What do you do when you are not working in the industrial cybersecurity space? What helps you unwind?

EB: Almost any outdoor sport excites me, and in my younger years, it was skiing and sailing that caught my attention. Today my road and gravel bikes are where I spend most of my downtime. To me, cycling is the perfect sport for a technically-obsessed engineer who loves working in a team. The modern high-performance bike is a control system on two wheels: speed sensors, cadence sensors, power meters, electronic shifters, and even radars are all integrated to help you be fast and safe.

But the best part is the focus and the teamwork of cycling. If you can ride in tight formation with a group of friends, then aerodynamic efficiency (aka drafting) lets you go faster and further with less effort. But this comes at a cost: if you are traveling at +30 km/hr less than 6 inches from the wheel in front of you, you better be paying attention 100% of the time. There is no time for me to worry about a problem at work. So I get to truly leave my work behind when I jump on my bike — it is what keeps me fresh every day.