Can We Use EOG to Identify When Attention Switches Away from

the Outside World to Focus on Our Mental Thoughts?

Anaïs Servais

1,2 a

, Raphaël Poveda

, Hanna Gerony

, Emmanuel J. Barbeau

and Christophe Hurter

Centre de Recherche Cerveau et Cognition (CerCo), CHU Purpan, Toulouse, France

École Nationale d’Aviation Civile (ENAC), Toulouse, France

Keywords: Attentional Switch, Mind-Wandering, Eye Movements, Electro-Oculography, Gaze Aversion.

Abstract: Aviation incidents resulting from attentional failures continue to occur. Attention is a limited resource and

perceptual decoupling occurs when the attention switches away from the outside world to focus on the inner

mental world. This phenomenon dramatically decreases visual perception, but is common and frequent since

human beings spend nearly half of their time immersed in their thoughts, implying numerous switches

between the outside and the inner world each day. We believe that detecting these attentional switches in air

traffic controllers could improve safety. We suggest gaze aversion as a potential behavioural objective marker

and therefore aim to find a method to measure gaze aversion in the lab. Our preliminary study tested EOG

and provided encouraging results since movements of gaze aversion differed significantly from visual

saccades in terms of amplitude and velocity, two characteristics measurable with the EOG signal. Gaze

aversions are faster and related to wider movements. This opens up great perspectives in aviation since the

EOG is a non-invasive method.

1 INTRODUCTION

Observe what’s going on in your mind while

answering the following question: "How did you

celebrate your last birthday?". Answering this

question required the creation of a temporary internal

mental space where your attention was focused

(Tulving, 2002). Thus, there was an attentional

switch from the external world to the internal world

— a change in mental state commonly known as

mind-wandering— a general term popularly used to

designate daydreaming and zoning-out (Smallwood

& Schooler, 2006). Indeed, since attention is a limited

resource, when it is focused on the internal world, the

external world vanishes (Fernandes & Moscovitch,

2000), a phenomenon known as perceptual

decoupling. Though useful for managing attentional

resources, this decoupling can have dramatic

consequences—impairing performance and safety in

operational tasks (such as driving a car or piloting a

plane) where attention must be directed to the

https://orcid.org/0000-0002-0032-2953

https://orcid.org/0000-0003-0836-3538

https://orcid.org/0000-0003-4318-6717

external world (Gouraud et al., 2018; Smallwood et

al., 2011). Reports show that aviation accidents

related to attentional lapses occur (NTSB, 2014), but

the origin of these lapses remains unknown. Given

perceptual decoupling, mind-wandering could be a

potential cause but objective, behavioural, markers to

monitor attention remain elusive.

Here, we propose eye movements as one such

potential candidate. Mind-wandering is associated

with oculomotor features reducing visual processing:

more and longer blinks, fewer and longer fixations,

longer saccades durations, and so on (Benedek et al.,

2017). However, these eye behaviours cannot be used

as markers to detect mind-wandering in real-time

because they require posteriori averages. Instead,

here we suggest investigating another eye behaviour

– gaze aversion — a very common behaviour while

answering memory questions. People routinely look

away as if they were searching for the answer on the

ceiling or “in the sky” (Doherty-Sneddon & Phelps,

2005; Glenberg et al., 1998).

Servais, A., Poveda, R., Gerony, H., Barbeau, E. and Hurter, C.

Can We Use EOG to Identify When Attention Switches Away from the Outside World to Focus on Our Mental Thoughts?.

DOI: 10.5220/0011950500003622

In Proceedings of the 1st International Conference on Cognitive Aircraft Systems (ICCAS 2022), pages 24-29

ISBN: 978-989-758-657-6

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

So far, mind-wandering has seldomly been

studied in aeronautics, perhaps because it is a

spontaneous mental state and therefore difficult to

assess. Current methods are based on self-reporting,

which has several biases, including social desirability

for pilots (Casner & Schooler, 2014). To overcome

this problem, since people tend to retrieve events that

occurred in their past during 60% of their zoning out

time (Mildner & Tamir, 2019), we propose to trigger

attentional switches in the lab using questions like the

one you answered, i.e., autobiographical memory

questions (Conway, 2001) while recording eye

movements with electro-oculography (EOG). EOG

records the electrical potential between electrodes

placed on the muscles around the eyes and allows to

quantify vertical and horizontal eye movements. In

airplane cockpits, the EOG may provide a less

invasive alternative to infrared cameras, which

generally obstruct part of the visual field. Recent

studies have also shown that EOG can be recorded

around the ears (Favre-Félix et al., 2017) – opening

perspectives for integration of the electrodes in the

pilot's helmet. In this preliminary work, we aim to

estimate the relevance of EOG to study gaze aversion

during internal attention.

2 MATERIALS AND METHODS

2.1 Participants

Data were collected on 4 participants (2 females, aged

between 23 and 28, 1 left-handed) without

oculomotor, visual, neurological, or psychiatric

disorders.

2.2 Experimental Task

We designed a task during which participants

answered autobiographical memory questions. Each

trial started with a fixation cross (5 sec) followed by

an oral question delivered through loudspeakers (5

sec). The questions were generated using

astread.com. After the question, the participant

completed a short visual task to keep attention

towards the external world: for 8 to 12 letters, the

participant pressed a key if the letter contained a

curve line and another if it did not. After the

presentation of the letters, the participant started to

search in autobiographical memory. This “reflection

phase” was divided into an earlier “access” phase

(max 12 sec) and a later “elaboration” phase (5 sec)

during which the participant respectively selected and

explored a personal memory corresponding to a

unique

and short event (< 24 hours), with a defined

Figure 1: This figure illustrates the method used to collect the data. On the left, is the position of the electrodes. On the right,

is the description of the cognitive task.

Can We Use EOG to Identify When Attention Switches Away from the Outside World to Focus on Our Mental Thoughts?

spatiotemporal context. An audio “debriefing”

indicated the start of the verbalization. Stimuli

presentation was controlled using OpenSesame 3.2.8

(Mathôt et al., 2012). Then, the patient pressed a key

to start the next trial (see Figure 1 for details).

To isolate the impact of autobiographical

memories, we included two control conditions: a

semantic memory task (also involving internal

attention) concerning famous events or people, and a

visual task (external attention) that required finding

the digit “5” hidden in a visual scene. The whole

experiment was divided into 5 blocks of 15 questions

(5/condition) presented in random order. Each

participant completed 25 trials per cognitive

condition. Explanations about the task were first

provided and then participants had 2 practice trials

per condition (not further analysed).

2.3 EOG Recording

A BioSemi Active Two amplifier was used to record

the EOG with 5 electrodes: a reference on the

forehead, one electrode at each external canthus for

horizontal movement, one electrode above the right

eyebrow, and one electrode on the top of the right

cheek for vertical movements. The frequency rate of

acquisition was 2048 Hz. To guarantee the quality of

the EOG data throughout the session, a calibration

was performed at the beginning of each block. In

order to have a ground truth, we recorded a video of

the participant's face using a SONY FDR-AX33

Handycam, with 1920x1080px resolution, positioned

above the screen.

2.4 Data Analyses

The signal was preprocessed—resampled to 256Hz,

and filtered with a low-pass at 40Hz. All the saccades

were labelled manually by two students. While

labelling the saccades, they were blind concerning the

part of the signal they were processing. Only the

saccades labelled by both judges were kept for

analysis.

The periods of interest were the visual task and

the 2 memory conditions. The visual trials allowed

us to get a sample of natural visual saccades, of the

type usually done while exploring the environment.

During memory retrieval, we focus on the trials

where gaze aversion occurred. With the video as

ground truth, the saccades initiating gaze aversion

were noted as aversion saccades. The

synchronization between the EOG and the video was

performed using BrainStorm (https://neuroimage.

usc.edu/brainstorm/).

Event related potentials (ERPs) were calculated

on epochs aligned on the aversion time (settled to

zero), a baseline of - 300ms before (because 300ms is

the min. time between the start of the memory

retrieval and a gaze aversion), and duration of 600ms

(the min. duration of a gaze aversion). ERPs allow

observing if a similar EOG pattern is repeated

between aversions.

The sensitivity of Eogert (Toivanen, 2015), a

probabilistic online method to detect EOG saccades,

was compared for visual and aversion saccades.

Compared to the saccades labelled manually, false

negatives and true positives were counted:

sensitivity=TP/(TP+FN). For visual saccades, the

sensitivity was 36% against only 9% for the aversions

saccades. The algorithm is more accurate on visual

saccades than aversion saccades, pleading for

different EOG features between these types of eye

movements (see Figure 2 for an illustration of the

signal during a memory trial with gaze aversion and

a visual trial with visual saccades).

Figure 2: Illustration of the EOG signal during A) a memory

trial; B) a visual trial for the same subject.

To investigate the differences between the EOG

features of visual compared to aversion saccades, we

conducted exploratory analyses comparing the gaze

angle and velocity of the two kinds of saccades with

Mann-Whitney U-tests (the size effect is the rank-

biserial correlation (r),

α was corrected with

Bonferroni to .016). For each gaze direction, a

constant of reference was calculated based on the

calibration (in

µV/degree). Gaze angle was calculated

following a linear relation.

Gaze angle =

EOG amplitude (µV)

Constant (µV/degree)

(1)

Velocity =

Gaze angle (degrees)

Duration of the saccade (sec)

(2)

ICCAS 2022 - International Conference on Cognitive Aircraft Systems

3 RESULTS

Out of 100, gaze aversion was observed in 35 trials

for autobiographical memory and 26 trials for

semantic memory questions. There was inter-

individual variability in the occurrence of gaze

aversion since the proportion of questions where gaze

aversion was observed varied between participants

from 4% to 68% for autobiographical questions and

from 0% to 56% for semantic questions. A total of

1062 visual saccades were counted during the visual

trials.

The visualization of the ERPs was possible only for 3

subjects because the fourth one did not do enough

gaze aversions. It revealed individual similarity of the

vertical EOG pattern (Figure 3).

The aversion saccades showed different gaze

angle and velocity than visual saccades (Figure 4).

For the gaze angle, Mann-Whitney U-tests showed

significant differences between visual and aversion

saccades. Gaze angles were much larger for

aversions, both in autobiographical (EOGh: U= 7429,

p<.001, r=.48; EOGv: U=3669, p<.001, r=.74) and

semantic memory trials (EOGh: U=6961, p=.007,

r=.33; EOGv: U = 713, p<.001, r=.93). The angle did

not differ significantly between the aversions from

the two memory conditions memory (EOGh: U = 263,

p=.22, r=.20; EOGv: U =424, p=.08, r=-.28). The

velocity of the saccades was also significantly

different between visual saccades and aversion

saccades, which appeared either in autobiographical

memory (EOGh: U= 10957, p=.03, r=.23; EOGv:

U=5901, p<.001, r=.59) or in semantic memory trials

Figure 3: ERPs per participant. Each colour represents the epoch of a trial. Aversions from autobiographical and semantic

trials have been pooled together. Vertical red lines indicate the start of the aversion separating the baseline from the gaze

aversion period. EOGh: EOG signal for horizontal movements; EOGv: EOG signal for vertical movements.

Figure 4: Gaze angle and velocity for visual and aversion saccades. One colour stands for all the saccades of a single

participant. ** p-value < .001.

Can We Use EOG to Identify When Attention Switches Away from the Outside World to Focus on Our Mental Thoughts?

(EOGh: U = 11754, p=.32, r=-.12; EOGv: U= 2486,

p=<.001, r=.76). Vertical, but not horizontal,

saccades were much faster during gaze aversion than

visual saccades. The velocity did not differ

significantly between the two memory conditions

(EOGh: U = 218, p=.04, r=.34; EOGv: U =405,

p=.17, r=-.23).

4 DISCUSSION

From these preliminary results, it seems that the EOG

signal associated with eye saccades initiating a gaze

aversion during attentional switch differs from the

EOG signal associated with visual eye saccades that

people do when they explore their environment.

Saccades initiating gaze aversion were faster and had

larger amplitude than visual saccades. As expected,

aversions seemed to be more distinguishable from the

visual saccades on the vertical rather than on the

horizontal EOG.

However, the present study is far from real-time

detection of attentional switches. We are facing many

limits. First, we show that even on a small sample of

participants, there is huge individual variability in the

gaze aversion behaviour. The ERP analysis shows

that the pattern of the aversion differs between

participants. To be applicable, an individual personal

calibration of the system would be necessary. Here,

we used a small sample size because we aimed to

determine the potential interest of the method, but a

validation of the method would require a higher

sample. Second, to refine the results, future studies

should include recordings of head movements and a

better calibration to infer gaze angle more accurately.

Here, we show that gaze aversions induce a large gaze

angle. However, the linear relation with EOG is not

true for gaze at high eccentricities (e.g., Hládek et al.,

2018). Interestingly however, the results for gaze

aversion during autobiographical memory do not

differ significantly from the one during semantic

memory. It seems therefore linked to internal

attention in general, and maybe not specific to

autobiographical memory.

Despite these limits, we want to emphasize the

importance of studying such behaviour in

aeronautics. Although here we have focused on gaze

aversion occurring during memory retrieval for the

sake of developing an experimental paradigm, in our

view such behaviour is similar to what occurs during

mind-wandering and in fact any behaviour requiring

access to internal thoughts including when one is

speaking to an interlocutor. Monotonous tasks

generate higher rates of mind-wandering, which is a

problem given the increasing automation in the

cockpits (Gouraud et al., 2018). Therefore, we

urgently need an objective marker allowing real-time

detection of internal thoughts switching to monitor

attention in critical situations. In this context, our

work aims to open discussions and perspectives.

5 CONCLUSION

Although our results are preliminary, they are

encouraging. First, we propose a protocol to trigger

attentional switches in the lab. Second, we show that

these switches are associated with gaze aversions.

Third, since we observe that gaze aversion have

different EOG features compared to visual saccades,

we think that EOG could be a potential method to

study and detect attentional switch. In an early

laboratory phase, EOG could be coupled with

augmented reality helmets to characterize gaze

aversion better before reaching a reliable detection

using EOG only.

REFERENCES

Benedek, M., Stoiser, R., Walcher, S., & Körner, C. (2017).

Eye behavior associated with internally versus

externally directed cognition. Frontiers in Psychology,

8, 1092. https://doi.org/10.3389/fpsyg.2017.01092

Casner, S. M., & Schooler, J. W. (2014). Thoughts in flight:

Automation use and pilots

’task-related and task-

unrelated thought. Human Factors, 56(3), 433–442.

https://doi.org/10.1177/0018720813501550

Conway, M. A. (2001). Sensory-perceptual episodic

memory and its context: Autobiographical memory.

Philosophical Transactions of the Royal Society of

London. Series B, Biological Sciences, 356(1413),

1375–1384. https://doi.org/10.1098/rstb.2001.0940

Doherty-Sneddon, G., & Phelps, F. G. (2005). Gaze

aversion: A response to cognitive or social difficulty?

Memory and Cognition, 33(4), 727–733.

https://doi.org/10.3758/bf03195338

Dorronsoro, G., Álvarez, J. R. & Soto, J. M. (2019).

"Nonlinear analysis of electrooculogram signals using

higher-order statistics." IEEE Transactions on

Biomedical Engineering, 47, 5, 674-680, 2000.

https://doi.org/10.1109/RBME.2019.2951328

Favre-Félix, A., Graversen, C., Dau, T., & Lunner, T.

(2017). Real-time estimation of eye gaze by in-ear

electrodes. 39th Annual International Conference of the

IEEE Engineering in Medicine and Biology Society

(EMBC), 2017, 4086–4089. https://doi.org/10.1109/

EMBC.2017.8037754

Fernandes, M. A., & Moscovitch, M. (2000). Divided

attention and memory: Evidence of substantial

ICCAS 2022 - International Conference on Cognitive Aircraft Systems

interference effects at retrieval and encoding. Journal

of Experimental Psychology: General, 129(2), 155–

176. https://doi.org/10.1037//0096-3445.129.2.155

Glenberg, A. M., Schroeder, J. L., & Robertson, D. a.

(1998). Averting the gaze disengages the environment

and facilitates remembering. Memory & Cognition,

26(4), 651–658. https://doi.org/10.3758/bf03211385

Gouraud, J., Delorme, A., & Berberian, B. (2018). Out of

the loop, in your bubble: Mind wandering is

independent from automation reliability, but influences

task engagement. Frontiers in Human Neuroscience,

12, 1–13. https://doi.org/10.3389/fnhum.2018.00383

Hládek, Ľ., Porr, B., & Owen Brimijoin, W. (2018). Real-

time estimation of horizontal gaze angle by saccade

integration using in-ear electrooculography. PLoS

ONE, 13(1), 1–24. https://doi.org/10.1371/journal.

pone.0190420

Mathôt, S., Schreij, D., & Theeuwes, J. (2012).

OpenSesame: An open-source, graphical experiment

builder for the social sciences. Behavior Research

Methods, 44(2), 314-324. https://doi.org/10.3758/

s13428-011-0168-7

Mildner, J. N., & Tamir, D. I. (2019). Spontaneous thought

as an unconstrained memory process. Trends in

Neurosciences, 42(11), 763–777. https://doi.org/10.10

16/j.tins.2019.09.001

National Transportation Safety Board. (2014). Aircraft

Accident Report (NTSB/AAR-14/01 PB2014-105984).

https://www.ntsb.gov/investigations/accidentreports/re

ports/aar1401.pdf

Smallwood, J., Brown, K. S., Tipper, C., Giesbrecht, B.,

Franklin, M. S., Mrazek, M. D., Carlson, J. M., &

Schooler, J. W. (2011). Pupillometric evidence for the

decoupling of attention from perceptual input during

offline thought. PLoS ONE, 6(3), e18298.

https://doi.org/10.1371/journal.pone.0018298

Smallwood, J., & Schooler, J. W. (2006). The restless mind.

Psychological Bulletin, 132(6), 946–958.

https://doi.org/10.1037/0033-2909.132.6.946

Toivanen, M., Pettersson, K., & Lukander, K. (2015). A

probabilistic real-time algorithm for detecting blinks,

saccades, and fixations from EOG data. Journal of Eye

Movement Research, 8(2), 1–14. https://doi.org/

10.16910/jemr.8.2.1

Tulving, E. (2002). Episodic memory: From mind to brain.

Annual Reviews, 53, 1–25. https://doi.org/10.1146/

annurev.psych.53.100901.135114

Yanko, M. R., & Spalek, T. M. (2014). Driving with the

wandering mind: The effect that mind-wandering has

on driving performance. Human Factors, 56(2), 260–

269. https://doi.org/10.1177/0018720813495280

Can We Use EOG to Identify When Attention Switches Away from the Outside World to Focus on Our Mental Thoughts?