Needs and Challenges Concerning Cyber-risk Assessment in the

Cyber-physical Smart Grid

Gencer Erdogan

1 a

, Inger Anne Tøndel

2 b

, Shukun Tokas

1 c

, Michele Garau

Cyber-risk assessment methods are used by energy companies to manage security risks in smart grids. How-

ever, current standards, methods and tools do not adequately provide the support needed in practice and the

industry is struggling to adopt and carry out cyber-risk assessments. The contribution of this paper is twofold.

First, we interview six companies from the energy sector to better understand their needs and challenges.

Based on the interviews, we identify seven success criteria cyber-risk assessment methods for the energy sec-

tor need to fulfill to provide adequate support. Second, we present the methods CORAS, VAF, TM-STRIDE,

and DA-SAN and evaluate the extent to which they fulfill the identified success criteria. Based on the evalu-

ation, we provide lessons learned in terms of gaps that need to be addressed in general to improve cyber-risk

assessment in the context of smart grids. Our results indicate the need for the following improvements: 1) ease

of use and comprehensible methods, 2) support to determine whether a method is a good match for a given

tems. In such systems, it is no longer possible to sepa-

rate the digital part of the system from the more tradi-

tional power part of the system, as these technologies

are becoming deeply integrated. From a cybersecurity

perspective, this tight integration exposes the power

grid to many cyber-risks introduced by digitalization,

e.g., via IoT systems that are increasingly used in the

context of smart power grids (Tøndel et al., 2018).

Cyber-risk assessment is the de facto approach

used by large organizations to manage cybersecurity

c

https://orcid.org/0000-0001-9893-6613

d

https://orcid.org/0000-0002-9803-9944

e

https://orcid.org/0000-0001-7127-6694

hand we have widely used cyber-risk assessment ap-

proaches such as ISO 27005 (ISO/IEC 27005:2018,

2018) and NIST 800-39 (NIST 800-39:2011, 2011),

while on the other hand we have risk assessment ap-

edge on how to best use existing approaches to carry

out a holistic cyber-risk assessment considering the

merged cyber-physical aspect of the future power grid

systems. Moreover, there is a lack of knowledge for

combining an assessment of specific types of threats

(e.g., cyber) with a more overarching assessment to

obtain a more concrete picture of the overall risk.

This paper explores the industry’s challenges and

needs for carrying out cyber-risk assessment in com-

plex and integrated cyber-physical systems, with fo-

for moving towards more integrated risk assessment

that includes both cybersecurity and power system

threats, as well as ICT dependability issues. Thus,

the contribution of this paper is twofold. First, we

carry out interviews with representatives from the in-

dustry to better understand the current and envisioned

For each of these methods, we provide a description

and evaluate the extent to which they meet the success

criteria identified from the interviews. Based on the

evaluation, we map the four methods to a qualitative

scale representing the level of fulfillment of criteria.

We also provide lessons learned in terms of identified

gaps that need to be addressed to improve cyber-risk

assessment in the context of smart grids.

The rest of the paper is organized as follows. Sec-

tion 2 describes the background and related work.

nally, Section 7 concludes the paper and summarizes

lessons learned in terms of identified gaps.

2 BACKGROUND AND RELATED

WORK

According to ISO 27005, "a risk is a combina-

tion of the consequences that would follow from

the occurrence of an unwanted event and the like-

lihood of the occurrence of the event" (ISO/IEC

27005:2018, 2018). As indicated in Section 1, there

are many standards and specialized approaches for

1999) are examples of tree-based approaches and

provide support for reasoning about the sources

and consequences of unwanted incidents, as well

as their likelihoods. Cause-consequence analysis

(CCA) (Nielsen, 1971) and Bayesian networks (Ben-

Gal, 2008) are examples of graph-based notations.

Cause-consequence analysis employs diagrams that

combine the features of both fault trees and event

trees, whereas the latter two serves as mathemati-

cal models for probabilistic and statistical calcula-

tions, respectively. Moreover, whereas alternative ap-

proaches such as CRAMM (Barber and Davey, 1992)

and OCTAVE (Alberts et al., 2003) rely on text and

mentioned steps are part of the standard risk assess-

ment processes (ISO/IEC 27005:2018, 2018; NIST

800-30, 2012). The technological trends underlying

the smart grid suggests a broad spectrum of the ICT

being deployed for more effective grid operations.

This integrated digital-power grid shift also brings

growing attack risks to the smart grid. The energy in-

dustry faces significant challenges in managing such

risks. When analyzing risks in today’s power sys-

tems, traditional risk assessment methods should be

dependencies, in order to highlight potential vulnera-

nerability assessment can be seen as a method that

aims at identifying hidden vulnerabilities in infras-

tructure systems that can bring to disruptive events,

such as blackouts, economic or social turmoil, etc.

rules of interaction, system of systems, broad spec-

trum of hazard and threats (Kröger et al., 2011). A

framework for studying vulnerabilities and risk in the

electricity supply, based on the bow-tie model, is pro-

posed in (Hofmann et al., 2012; Kjølle et al., 2012;

Hofmann et al., 2015). Among the vulnerabilities,

a specific case is represented by cyber-security risks,

which can be defined as the potential of loss as a re-

sult of cyberattack resulting from the operations of an

information system.

assessment methods traditionally adopted in power

systems. A multi-layer modelling approach is sug-

gested, where the power layer, communication and

coupling layer and decision layer interact in order to

enable the power system operation. Each of these

layers are characterized by vulnerabilities that should

sessment possible failure states in the ICT infrastruc-

tures, novel approaches have been introduced, which

adopt complex network theory (Zhu et al., 2018),

cyber-physical interface matrix (Lei et al., 2015), co-

simulation (Garau et al., 2015), and traditional event

trees (Liu et al., 2019) and reliability block diagrams

(Ding et al., 2018). These works adopt approaches

that are strongly related with the concept of probabil-

ity of failure occurrence, therefore they find a difficult

application to scenarios where the threat is deliberate

stochastic Petri net models (Ten et al., 2008), Markov

processes (Zhang et al., 2016) and Bayesian attack

graphs (Zhang et al., 2015).

that work well based on the interviewee’s experi-

ence.

also acts as a basis for lessons learned, summarized

in Section 7.

4 IDENTIFIED SUCCESS

CRITERIA

This section describes the success criteria identified

based on the interviews, as explained in Section 3. In

total, we identified seven success criteria (SC) for risk

assessment approaches, addressing needs and chal-

lenges in the industry pointed out by the interviewees.

In the following, we present each success criterion

that quantitative methods are not currently an option

for them, and that there is a need to start with very

easy methods. One interviewee even states that it is

more important that a method is easy to use than the

quality of the results of the analysis, because if the

method is too complex and requires too much effort

it will meet resistance and the risk assessment will

probably not be carried out. Currently, many of the

companies seem to opt for using the same methods for

cyber risk as for other risk. In the companies, there is

Note also that we talked with relatively large compa-

nies within this sector. However, one interviewee ex-

have dedicated in-house cybersecurity experts. If the

risk analyst does not have the necessary competence,

support, or training, interviewees explain that one risk

is that the analyst just ticks that a risk assessment has

been performed without the risks being properly as-

sessed.

SC2 Provide support to determine whether the

method is a good match for a given context. There

is a large variety in current practice and current abil-

ity to perform cybersecurity risk assessments among

lyzing cyber risk and what questions to consider in

the assessment. Further, there are different types of

risk assessments that are performed in the companies,

ranging from yearly risk assessments to smaller as-

sessments as part of procurement or changes. There

is a clearly stated need to start with simple assessment

approaches, but at the same time the complexity of the

target of analysis may point to a need to move towards

more complex assessment approaches in some cases,

including when companies have become more mature

2 15.10.2021 1 hour

3 19.10.2021 1 hour

4 04.11.2021 1 hour

5 05.11.2021 1 hour

cluding establishing a common understanding of

concepts and build necessary knowledge of partic-

ipants from IT and OT. When cybersecurity is con-

sidered in the more traditional risk assessments, it is

experienced as being abstract. Interviewees tell of

experiences where cybersecurity is represented with

only one scenario in combination with other types of

threats, e.g., technical failures, extreme weather con-

ditions. In many of the companies there seem to be

a division between IT and OT, though some explain

risk assessments without any prior understanding of

cyber risk and the risk assessment process, thus lead-

ing to misunderstandings and challenges during as-

sessment. IT and OT people may, e.g., disagree on

the interpretation of key concepts such as likelihood

and consequence and have a different understanding

of criticality. In the sector, there is some support ma-

terial available from sectorial organizations. How-

ever, there is a need for more support – concrete ex-

amples and lists of scenarios are highlighted in the

ple from OT are experts in their domain they might

not have the knowledge needed to evaluate cyber risk,

e.g., know the architecture of the OT systems.

SC4 Manage complexity in the risk assessment,

considering the target of analysis. The analysis

target is complex, and the complexity is increasing,

which makes it difficult to do good risk assessments.

There are several reasons for these challenges. There

and their use, and some of these changes happen grad-

ually. Often, manual systems are seen as backups, but

eventually the organization looses experience in us-

ing these manual backup systems, and thus they lose

much of their value. This gradual change can be dif-

ficult to capture in risk assessments. For example, if

an assessment uses a previous analysis as a starting

point, it is easy to become influenced of the previous

conclusions and not see what has changed and the as-

sumptions that may no longer be valid. Furthermore,

such as being able to bill customers. However, these

other systems may not get enough attention. It is

challenging to understand how one risk affects other

risks. Assessments are often done for single systems

or for single types of incidents, but it is challenging to

understand any relations between these and combine

analysis results to get a more holistic view of the risk.

SC5 Support risk estimation, e.g., likelihood and

consequence estimation, as well as ranking of as-

sets considered in the risk assessment. There is a

vant when assessing cyber risk. Moreover, intervie-

wees tell that disagreements between different pro-

fessions often happen related to likelihood and con-

sequence estimation. When it comes to consequence,

the main challenges are related to estimation of in-

direct consequences (e.g., reputation). One intervie-

wee points to security economy as important mov-

ing forward, to make the economic costs of secu-

rity incidents clearer to the decision makers. When

it comes to likelihood estimation, this is considered

particularly difficult as one is dealing with malicious

threats. Several interviewees consider replacing like-

lihood estimates with evaluations of threat actors and

their capacity and intention, and the vulnerabilities

explain that there is not enough data to use for esti-

mating likelihood, and point to the risk of underes-

timating the likelihood for things that have not yet

happened. One interviewee explains that support for

reuse of likelihood estimates would be highly useful.

Support for reuse would reduce the need to involve

key experts in every analysis. Many of the assess-

ments are of objects that have similar characteristics.

Moreover, many aspects about the threats are similar

for other companies of the same type.

related to risk estimation (SC5), a few interviewees

point to the need to consider uncertainty in the risk es-

timates. Trustworthiness in risk estimation is impor-

tant, to be confident in what to report to management,

risk assessments with pentesting and monitoring, in

ways that increase trustworthiness in assessment and

effectiveness in testing and monitoring. Some inter-

viewees envision a future with more real time risk as-

sessments, and wish for more tool support that can

help them in the risk assessments and that are able

to bring in data as support, e.g., to identify relevant

threat scenarios.

SC7 Facilitate risk management through docu-

mentation, maintenance of assessments, and ex-

within this sector, that everything should be secure,

may not work moving forward, and that there will

be a need to build resilience into the system so that

they can tolerate some cyber-incidents taking place.

Regarding documentation, one interviewee explained

about a lack of culture for documenting risk analy-

if information security experts need to be involved or

even responsible for such assessments. Another chal-

lenge is communicating the results of the risk assess-

ment in a way that is comprehensible to management

and that puts the cyber-risk topic on their agenda. On

the positive side, one interviewee tells about regular

reporting of cyber risk to the board, and another tells

about using threat modeling in the management group

at a high level, to discuss why attacks are possible and

what can be done. On the other hand, one interviewee

(TM-STRIDE), and Dependability analysis with

Stochastic Activity Network (SAN). Because of space

restrictions, we provide a brief description of each ap-

proach and refer to other sources for further details.

CORAS is a method for conducting security risk as-

sessment (Lund et al., 2011). In the CORAS method,

a security risk assessment is conducted in eight steps:

1) preparations for the analysis, 2) customer presenta-

tion of the target, 3) refining the target description us-

threat and risk modelling, and comes with detailed

guidelines explaining how the language should be

used to capture and model relevant information dur-

ing the various steps of security risk assessment. The

CORAS method provides a web-based tool (CORAS

Tool, 2022) designed to support documenting, main-

taining and reporting assessment results through risk

modelling.

CORAS is a general approach to cybersecurity

risk assessment and has been applied to a large va-

The Vulnerability Analysis Framework (VAF)

(Gjerde et al., 2011; Hofmann et al., 2015) is an

analysis approach aimed at identifying and analyzing

extraordinary events that can impact power system

of a "model". Incidentally, threat modelling is part of

what McGraw refers to as Architectural Risk Analy-

sis (McGraw, 2006).

The STRIDE mnemonic (Spoofing, Tampering,

Repudiation, Information disclosure, Denial of ser-

vice, Elevation of privilege) was introduced by Mi-

crosoft, and gained prominence through Swidersky &

ing tool (Bygdås et al., 2021), which conforms to the

methodology presented by Swidersky & Snyder.

The first step in this threat modeling approach is

to draw a data flow diagram (DeMarco, 1979) which

helps to understand the system’s attack surface by

providing an overview of entities, processes and data

stores, identifying trust boundaries and sketching how

data flows in the system. The resulting threat model

is thus a visual representation of four main elements:

the assets within a system, the system’s attack surface,

formances of a state estimation calculation is quanti-

tatively analysed. Among internal vulnerabilities, ra-

dio link failures, server failures, measuring devices,

etc. have been considered. Among external vulnera-

bilities, the impact of signal fading due to rain precip-

itation has been inspected.

sidering cyber risks. The approach DA-SAN needs

specialized expertise and may not be easy to use un-

less one has the specific competence and skills. Al-

though VAF, CORAS, and TM-STRIDE may be eas-

ier to comprehend and use compared to DA-SAN,

none of the methods fully meet the SC1 criterion.

Considering the criterion SC1, and the fact that all

the identified criteria described in Section 4 points out

the need for some kind of support to more easily carry

out risk assessment, indicates that comprehensibility

and ease of use is the most important success crite-

rion. One way of addressing this challenge would be

to make the existing approaches more light-weight,

but this would come at the cost of expressiveness and

the methods’ ability to handle complexity. Thus, to

successfully achieve criterion SC1, it is necessary to

develop risk assessment methods that are easy for the

energy sector to use, as well as providing guidelines to

select from a variety of approaches that balances be-

the methods do provide general guidelines for the an-

alyst to understand the context in which the method

may be applied. However, these general guidelines

are meant for security experts and are not an ade-

quate support for non security experts in the energy

sector as they are struggling to answer questions like:

"how can I carry out a simple high-level risk assess-

ment even if I don’t have cyber-risk expertise?", "what

questions should I consider when assessing risks?",

tor needs guidelines to select appropriate risk assess-

the most critical unwanted incidents. These incidents

may be used as input to the CORAS method, which

may help identify the chain of events that may cause

the unwanted incidents, including exploited vulnera-

and unwanted incidents on a power-grid system using

simulation techniques.

Regarding SC3, among the methods we have con-

sidered, CORAS and TM-STRIDE have thorough

steps to prepare a risk assessment in terms of es-

tablishing the context and making sure that all in-

volved stakeholders have a common understanding of

the context, concepts, and objectives of the risk as-

sessment. The VAF method also has the necessary

steps to prepare an assessment, but is slightly easier

Figure 2: Comparison of methods with respect to fulfillment of the success criteria described in Section 4.

ple during the preparation of an assessment partici-

pants can be introduced to risk assessment with exam-

ple scenarios specific to the energy sector. However,

a proper educational support would be to train the rel-

evant people using facilities such as cyber ranges that

are capable of simulating cyber-attacks on energy in-

frastructure. Our previous work shows that cyber-risk

training using cyber ranges are effective for a variety

of domains such as energy distribution, railroad trans-

port, and education (university) (Erdogan et al., 2020;

while the methods DA-SAN and TM-STRIDE lack

the capability to express risks the target of analysis is

exposed to as part of the target models, the methods

VAF and CORAS lack the expressiveness to repre-

sent the target of analysis as part of the risk models.

Each aforementioned method have of course been de-

veloped for their specific purpose, but it is reasonable

to argue that a method capable of capturing both the

talization of the energy systems, changes (both from

a cyber perspective and from a physical perspective)

may happen frequently. Whenever an update is intro-

duced in the energy systems, then it is important to

consider this change in the risk assessment as well.

The CORAS method has explicit support to consider

a risk picture before a change is introduced, and after

a change is introduced in the target system.

The success criterion SC5 points out the need for

support for risk estimation and ranking of assets. The

estimation using the CORAS calculus. Ideally, ac-

cording to the interviews, a risk assessment method

should provide guidelines for both likelihood and

consequence estimation. One possible approach is to

combine different methods to fully achieve SC5. For

example, DA-SAN can support CORAS with con-

sequence estimates, while CORAS can support DA-

SAN with likelihood estimates. Another option is to