Fighter Cockpit Adaption to Online Situation Awareness

Measurement

Simon Schwerd and Axel Schulte

Institute of Flight Systems, Universität der Bundeswehr München, 85577 Neubiberg, Germany

Keywords: Adaptive System, Eye-Tracking, Task Analysis, Situation Awareness.

Abstract: Effective attention management is crucial to fighter pilot performance, but attention-related problems can lead

to a loss of situation awareness, which can contribute to performance decrements. To address this issue,

researchers have proposed the development of user-aware systems that can adapt to a pilot's cognitive state.

This contribution briefly describes an approach of on an eye-tracking-based online estimation of the pilot’s

situation awareness, which is used to trigger alerts to guide the pilot’s attention to relevant information. To

transfer this approach to a military domain, we conducted a goal-directed task analysis (GDTA) with eight

fighter pilots. From the resulting task structure, relevant use-cases were identified and implemented for a

fighter cockpit simulation and evaluated with three pilots. In these trials, the approach was rated positively

for realistic task settings but failed to provide useful assistance when pilots wanted to focus on a single task.

1 INTRODUCTION

Fighter pilots face a broad task spectrum in the

cockpit. Their tasks are considerably more

challenging than those of civil pilots due to additional

mission tasks, which include control of sensors,

weapons, and other systems, under consideration of

constraints and uncertainties. In addition, the pilots

coordinate with manned and, as envisioned in fighter

development programs (e.g., FCAS), unmanned

platforms. This adds coordination with manned-

unmanned teams to the pilots’ responsibilities

(Lindner et al., 2022). In this multitasking

environment, efficient management of attention is

crucial to pilot performance (Olivier Lefrancois et al.,

2016). Attention-related problems in human-

automation interaction are well known and often

described as a possible cause of aviation accidents

(Jones & Endsley, 1996; Kelly & Efthymiou, 2019;

Shorrock, 2007). There are different problems

described in literature, such as attentional tunneling

(Wickens, 2005), inattentional blindness or deafness

(Dehais et al., 2019), vigilance performance

decrement (Thomson et al., 2015) or complacency

(Parasuraman & Manzey, 2010). The consequence of

all these phenomena is a loss of situation awareness

(SA) in specific situational aspects when the pilot

fails to attend relevant information in the cockpit

(Jones & Endsley, 1996). Effective training in

monitoring and automation operation is crucial to

improve pilot attention management, but

nevertheless, the interaction between pilots and

cockpit remains ‘hierarchical’ in the sense, that the

cockpit cannot be aware of the pilot’s errors. In this

context, the idea of developing user-aware systems

has been proposed to enable error-reducing adaption

of a workplace (Brand & Schulte, 2018; Fortmann &

Mengeringhausen, 2014; Peysakhovich et al., 2018).

In this contribution, we focus on the adaption of a

fighter cockpit to an online estimation of the pilot’s

cognitive state, more specifically their attention

allocation and situation awareness. Thus, our design

goal is a system that reduces the number of situations

where a pilot misses critical information while trying

to avoid nuisance notifications (Schwerd & Schulte,

2021b). To achieve automatic online estimation of

SA, we developed an eye-tracking analysis approach

where, for every fixation on the cockpit display, an

application provides the attended object and a

parametric description of their relevant content (e.g.,

not only “altimeter”, but also the altitude; not only

‘hostile aircraft’, but also its heading, speed, and

distance). This information is used to populate nodes

in a dynamic semantic network that represents the

relevant situational features and relationship between

information, e.g., when the pilot is aware of the

position of two objects, he can also infer a distance

Schwerd, S. and Schulte, A.

Fighter Cockpit Adaption to Online Situation Awareness Measurement.

DOI: 10.5220/0011950700003622

In Proceedings of the 1st International Conference on Cognitive Aircraft Systems (ICCAS 2022), pages 30-33

ISBN: 978-989-758-657-6

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

between them. Each network node contains a measure

of deviation between the assumed pilot’s awareness

and the actual state of this situational feature.

Therefore, the system can estimate the SA in specific

situational features. In prior studies, we validated this

approach in cockpit simulator studies and showed

correlations of our SA measure with performance and

subjective SA ratings (Schwerd & Schulte, 2020,

2021a). We used this SA model in a subsequent

experiment to trigger cockpit adaptions and alerts to

guide the pilot’s attention towards critical system

information when the deviation in relevant situational

features grew above a certain threshold. With that, we

could improve pilot performance in those situations

where a change of task-relevant system state could

not have been predicted by the participants (Schwerd

& Schulte, 2021c).

While our approach worked reasonably well in

our laboratory trials, transfer to real-world military

application is not trivial. Apart from challenges like

eye-tracking measurement in real cockpits, the

adaptions must be useful in the task context and

should provide benefits to the pilot. Thus, our central

question is: which cockpit information is relevant in

which task situation?

2 METHOD

To answer this question, we conducted a goal-

directed task analysis (GDTA) with eight fight pilots.

Based on this task analysis, we identified use-cases,

which were implemented in our cockpit simulator.

Then, we evaluated these use-cases with three fighter

pilots of the German Air Force.

2.1 Task Analysis to Identify Use-Cases

The GDTA was proposed by (Endsley & Jones, 2012)

and can be used to structure a task environment by its

goals, decisions to meet these goals, and information

requirements to make these decisions. It is especially

suitable for our research question because the task

analysis associates information with its context. For

the interview, we prepared different mission scenario

briefings (e.g., Air Interdiction Mission) to structure

the discussion. On basis of these scenarios, we went

through different mission phases to identify relevant

operational decisions. When we identified a decision,

we asked about all relevant information that is

associated with this information.

2.1.1 Procedure & Participants

We interviewed eight fighter pilots from the German

Air force (all male, mean age 36.2y). Only one pilot

was interviewed per session which lasted about two

hours for each pilot. In our setting, pilots could only

talk about non-classified information. After the

interviews, we organized goals, decisions, and

information into a tree-like structure.

2.1.2 Results

The resulting GDTA is structured by five main goals,

which are displayed in Figure 1. For example, the first

goal is to operate the aircraft under consideration of

the mission plan. Every main goal consists of several

subgoals, also illustrated for goal 1.0 in Figure 1.

Decisions are associated in the lower levels of the

structure and always associated with a subgoal.

Figure 2 shows three examples for relevant decisions,

that must be frequently done in the cockpit. For

example, the subgoal ‘assess planned flight altitude’

is associated with the decision if a change to the UAV

Figure 1: Top-level extract from analysis.

Fighter Cockpit Adaption to Online Situation Awareness Measurement

Figure 2: Example decisions from different goals.

altitude is necessary because there is a more secure

route available. To make this decision, the pilot must

collect information in the cockpit about the current

position and types of threats, the current UAV

altitude, and its planned UAV altitude.

Given the many decisions and information

requirements from the analysis, we selected suitable

use-cases suitable to evaluate them in the cockpit

simulator.

2.2 Prototype Evaluation

The second step of this study was the implementation

and evaluation of a cockpit adaption in a

representative task setting. For this, we implemented

12 different adaption use cases in our cockpit

simulator (see Figure 3). Cockpit adaptions were

either a computer-generated text message or an

indication on the tactical map.

Figure 3: Cockpit simulator with integrated eye-tracking.

2.2.1 Procedure & Participants

We invited three fighter pilots to evaluate our system.

Mean age was 36y with mean flight hours on a fighter

jet of 600h. After a training of 2 hours, we fully

explained the basic principle of the cockpit adaption

and introduced all use cases. Then, we evaluated the

implemented assistance system in three scenarios.

Each scenario emphasized different mission types

and pilot tasks (Reconnaissance, UAV Control and

MUM-T Air Interdiction). After each scenario, the

participants were asked to fill out a subjective

usability rating about every specific assistance use

case they encountered. After all scenarios, we

replayed a recording of each trial and interviewed the

pilots about specific situations to gain further insight

into their rating and possible improvements.

2.2.2 Results

The subjective rating, debriefings and preliminary

analysis of the logging data showed the following:

• The SA-based adaption was accepted well by

the pilots and subjective rating was positive. The

use-cases were considered to be useful in a real-

world task setting. Especially indications in the

tactical map were evaluated as very helpful. In

addition, pilots asked for phase-of-flight

specific use cases (e.g., take-off, landing).

• In a few situations, pilots did not monitor a

certain cockpit display because they relied on the

auditive text messages telling them relevant

information. In other cases, pilots criticized when

text messages contained a lot of information.

• Pilots ignored system indications as soon as the

workload was high. Because of this, they often

could not recall an encountered use-case in

some trials.

• Pilots disliked to system indications that merely

told them information they should know from a

perspective of SA. In their opinion, the SA-

based indication should only appear when there

is action required.

3 DISCUSSION

Our experiment showed that we successfully

implemented use-cases, that are useful in a real-world

fighter jet cockpit. However, one flaw of our

approach is, that in situations where the pilot focuses

ICCAS 2022 - International Conference on Cognitive Aircraft Systems

on a single demanding task while ignoring other

information, our system tends to trigger more

indications since SA in task-irrelevant information

suffers from the attentional focus on a single task. But

these indications are often ignored due to high

workload or might even add more workload. This

problem could be solved by two approaches: Either

delaying indications until the pilot is finished with his

task depending on a measurement of activity

(Honecker & Schulte, 2017) or, in critical cases,

interrupt the pilot with more drastic cockpit adaptions

such as cognitive counter measures (Saint-Lot et al.,

2020). We are planning to evaluate these approaches

in our future studies.

REFERENCES

Brand, Y., & Schulte, A. (2018). Design and Evaluation of a

Workload-Adaptive Associate System for Cockpit

Crews. In D. Harris (Ed.), Engineering Psychology and

Cognitive Ergonomics (pp. 3–18). Springer International

Publishing.

Dehais, F., Roy, R. N., & Scannella, S. (2019). Inattentional

deafness to auditory alarms: Inter-individual differences,

electrophysiological signature and single trial

classification. Behavioural Brain Research, 360, 51–59.

https://doi.org/10.1016/j.bbr.2018.11.045

Endsley, M. R., & Jones, D. G. (2012). Designing for

situation awareness: An approach to user-centered

design / Mica R. Endsley and Debra G. Jones (2nd ed.).

CRC Press.

Fortmann, F., & Mengeringhausen, T. (2014). Development

and Evaluation of an Assistant System to Aid Monitoring

Behavior during Multi-UAV Supervisory Control. In C.

Stary (Ed.), Proceedings of the 2014 European

Conference on Cognitive Ergonomics - ECCE '14 (pp. 1–

8). ACM Press. https://doi.org/10.1145/2637248.263

7257

Honecker, F., & Schulte, A. (2017). Automated Online

Determination of Pilot Activity Under Uncertainty by

Using Evidential Reasoning. In D. Harris (Ed.), Lecture

Notes in Computer Science. Engineering Psychology and

Cognitive Ergonomics: Cognition and Design (Vol.

10276, pp. 231–250). Springer International Publishing.

https://doi.org/10.1007/978-3-319-58475-1_18

Jones, D. G., & Endsley, M. R. (1996). Sources of situation

awareness errors in aviation. Aviation, Space, and

Environmental Medicine, 67(6), 507–512.

Kelly, D., & Efthymiou, M. (2019). An analysis of human

factors in fifty controlled flight into terrain aviation

accidents from 2007 to 2017. Journal of Safety Research,

69, 155–165. https://doi.org/10.1016/ j.jsr.2019.03.009

Lindner, S., Mund, D., & Schulte, A. (2022). How Human-

Autonomy Teams change the Role of future Fighter

Pilots: An Experimental Assessment. In AIAA SCITECH

Forum. https://doi.org/10.2514/6.2022-2551

Olivier Lefrancois, Nadine Matton, Yves Gourinat, Vsevolod

Peysakhovich, & Mickaël Causse (2016). The role of

Pilots’ monitoring strategies in flight performance. In

European Association for Aviation Psychology

Conference EAAP32. https://oatao.univ-toulouse.fr/16

173/

Parasuraman, R., & Manzey, D. H. (2010). Complacency and

bias in human use of automation: An attentional

integration. Human Factors, 52(3), 381–410.

https://doi.org/10.1177/0018720810376055

Peysakhovich, V., Lefrançois, O., Dehais, F., & Causse, M.

(2018). The Neuroergonomics of Aircraft Cockpits: The

Four Stages of Eye-Tracking Integration to Enhance

Flight Safety. Safety, 4(1), 8. https://doi.org/10.3390/

safety4010008

Saint-Lot, J., Imbert, J.‑P., & Dehais, F. (2020). Red Alert: A

Cognitive Countermeasure to Mitigate Attentional

Tunneling. In R. Bernhaupt, F. '. Mueller, D. Verweij, J.

Andres, J. McGrenere, A. Cockburn, I. Avellino, A.

Goguey, P. Bjørn, S. Zhao, B. P. Samson, & R. Kocielnik

(Eds.), Proceedings of the 2020 CHI Conference on

Human Factors in Computing Systems (pp. 1–6). ACM.

https://doi.org/10.1145/3313831.33 76709

Schwerd, S., & Schulte, A. (2020). Experimental Validation

of an Eye-Tracking-Based Computational Method for

Continuous Situation Awareness Assessment in an

Aircraft Cockpit. In D. Harris & W.-C. Li (Eds.), Lecture

Notes in Computer Science. Engineering Psychology and

Cognitive Ergonomics. Cognition and Design (Vol.

12187, pp. 412–425). Springer International Publishing.

https://doi.org/10.10 07/978-3-030-49183-3_32

Schwerd, S., & Schulte, A. (2021a). Measuring the Deviation

between Ground Truth and Operator Awareness in a

UAV Management Scenario: An Eye-Tracking

Approach. In AIAA Scitech 2021 Forum. American

Institute of Aeronautics and Astronautics. https://doi.org/

10.2514/6.2021-1579

Schwerd, S., & Schulte, A. (2021b). Operator State

Estimation to Enable Adaptive Assistance in Manned-

Unmanned-Teaming. Cognitive Systems Research, 67,

73–83. https://doi.org/10.1016/j.cogsys.2021.01.002

Schwerd, S., & Schulte, A. (2021c). Situation Awareness

Adaptive Alerting in an Aircraft Cockpit: A Simulator

Study. Preprint. Advance online publication.

https://doi.org/10.31224/osf.io/9gmnv

Shorrock, S. T. (2007). Errors of perception in air traffic

control. Safety Science, 45(8), 890–904.

https://doi.org/10.1016/j.ssci.2006.08.018

Thomson, D. R., Besner, D., & Smilek, D. (2015). A

resource-control account of sustained attention: Evidence

from mind-wandering and vigilance paradigms.

Perspectives on Psychological Science: A Journal of the

Association for Psychological Science, 10(1), 82–96.

https://doi.org/10.1177/174569161455 6681

Wickens, C. D. (2005). Attentional Tunneling and Task

Management. International Symposium on Aviation

Psychology.

Fighter Cockpit Adaption to Online Situation Awareness Measurement