Study of the User Behaviour Caused by Automatic Recommendation

Systems Call to Action

Georgy Kopanitsa

and Sergey Kovalchuk

2,1 b

ITMO University, Birzhevaya 4, Saint-Petersburg, Russia

Almazov National Medical Research Centre, Akkuratova Street, Saint-Petersburg, Russia

Keywords: Symptom Checker, User Acceptance, Follow-up Rates.

Abstract: Diagnostics accuracy and usability of symptom checkers have been researched in several studies. Their ability

to set a correct diagnosis especially in the urgent cases is questionable. There is one aspect of symptom

checkers that has not been deeply studied yet. It is their ability to motivate patients to follow up after receiving

a direct recommendation and to decrease a load on the health care professionals. The goal of this research is

to study how patients behave after receiving a recommendation from a symptom checker and motivation of

this behaviour. We studied how patients react on the symptom checker recommendations and the motivation

behind this behaviour. In total we invited 3615 patients to have a symptom checker screening; 2374 of them

agreed to run a symptom checker screening; 867 of them agreed to participate in the study. The proportion of

the patients who agreed to have a symptom checker screening. So, we can clearly see that symptom checker

screening doesn’t result in a significant decrease of the load on healthcare professionals. This is supported by

the quantitative study results. The patients emphasized the ease of use of the tool and clearness of the

recommendations it gives. However, they perceived it as rather a second opinion tool or a tool that helps to

prepare to the doctor’s visit.

1 INTRODUCTION

The World Health Organisation (WHO) stated in

2013 that the world was short of 7.2 million

healthcare professionals of all disciplines. This

number will likely increase to 12.9 million by 2035

(Aluttis, Bishaw, & Frank, 2014). Primary care is

particularly (Karimi‐Shahanjarini et al., 2019)

affected, with severe lack of health care professionals

in most of the countries (Poppe et al., 2014; Wu,

Zhao, & Ye, 2016; Zihindula, John, Gumede, &

Gavin, 2019).

In large countries with remote area there is

another problem to transport patients to the points of

care.

This has several consequences for the healthcare

systems. One of the main and most problematic

issues are a reduced access to health care, growing

inequity, long waiting times (Griffith, Evans, & Bor,

2017), increased load on the emergency services

(Djahel, Smith, Wang, & Murphy, 2015; Stewart,

https://orcid.org/0000-0002-6231-8036

https://orcid.org/0000-0001-8828-4615

Lang, Wang, & Innes, 2019). This all lead to the late

diagnostics and not optimal treatment. Traditional

doctors oriented clinical decision support systems

could reduce a workload on the doctors. This was

not enough to solve all the listed problems. Machine

learning (ML) and Artificial Intelligence (AI)

methods in data-driven decision support systems

helped to reduce healthcare professionals workload,

allowed more efficient management of clinical

resources and help patients access medical services

(Battineni, Sagaro, Chinatalapudi, & Amenta, 2020;

Fraser, Coiera, & Wong, 2018).

Online and point of care symptom checkers are

becoming a tool that is supposed to be patient

oriented and solve the problem of initial screening

and help people in remote areas to have a first

medical contact (Baker et al., 2020).

Diagnostics accuracy and usability of symptom

checkers have been researched in several studies.

Their ability to set a correct diagnosis especially in

the urgent cases is questionable (Ceney et al., 2021).

Kopanitsa, G. and Kovalchuk, S.

Study of the User Behaviour Caused by Automatic Recommendation Systems Call to Action.

DOI: 10.5220/0010840000003123

In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2022) - Volume 5: HEALTHINF, pages 499-503

ISBN: 978-989-758-552-4; ISSN: 2184-4305

499

There is one aspect of symptom checkers that has

not been deeply studied yet. It is their ability to

motivate patients to follow up after receiving a

direct recommendation and to decrease a load on the

health care professionals.

The goal of this research is to study how

patients behave after receiving a recommendation

from a symptom checker and motivation of this

behavior.

2 METHODS

2.1 Symptom Checker

A symptom checker that is being studied is a general-

purpose diagnostic symptom checker that is based on

the Bayesian classifier. It askes multiple choice

questions (figure 1), suggests 3 most probable

diagnosis after a conversation (figure 2), and offers a

call to action (figure 3) to follow up with a doctor or

not. The average number of questions in one patient

interaction was 11. There were 3 types of call for

action messages: You need to see a doctor urgently

(red), it would be good to see a doctor soon if you feel

worse (yellow), there is no need to see a doctor now

(green).

Figure 1: Question example. Translation from Russian: Do

you have enlarged lymph nodes? Yes, No, Not Sure;

Continue.

Figure 2: Diagnosis suggestions.

Translation from Russian:

Acute respiratory disease

A group of infectious diseases that mainly affects the

upper respiratory tract and bronchi.

Nine out of 10 people with the same symptoms have

been diagnosed with this disease.

Diagnosis and treatment are carried out by a general

practitioner;

Warning! Respiratory system symptoms may be

characteristic of COVID-19. If you suspect a

coronavirus infection, call the hotline at 8-800-2000-

112. To learn about noncontact testing for COVID-

19, visit сдайнакоронавирус.рф

Figure 3: Call to action. Translation from Russian: Result

of symptom screening.

It would be good to see your doctor soon.

Your symptoms may need your doctor's attention,

schedule a consultation in the next few days. If you

feel significantly worse than usual and your

symptoms persist or get worse, see your doctor today.

2.2 Recommendations Follow Up

The first part of the study was dedicated to calculation

of a follow-up rate.

The patients were offered to work with a symptom

checker before a doctor’s visit. Facilitators, who were

not healthcare professionals supported patients with

the symptom checker operation. The patients were

motivated by a gift voucher that they could use for the

medical services in the clinic.

A proportion of patients who accepted the offer

was calculated by the research team.

HEALTHINF 2022 - 15th International Conference on Health Informatics

500

After the patients have completed a symptom checker

session, we invited them to participate in the study.

Inclusion criteria were:

 Age > 18

 Purpose of visit: primary visit

 Completed a symptom checker session

Exclusion criteria were:

 Age < 18

 Secondary visit scheduled by a doctor

Those who had agreed were offered to sign a consent

for that contained the purpose and the details of the

study. They were also asked to fill in a short

questionnaire that contained basic demographic

information: age, gender, education and contact

details.

After the consent form and a questionnaire were

submitted to the research team, the patients were

asked to provide the results of the symptom checker

session.

The patients were asked if they would like to have a

follow up visit with a doctor or not.

We calculated the proportion of patients who wanted

to have a follow up and those how didn’t for the

following groups of the patients:

 Green call to action (No need to see a doctor)

 Yellow call to action (Self-observation and

may be see a doctor later)

 Red call to action (Urgently see a doctor)

The study was performed in a private outpatient clinic

in March-April 2021.

The study got the approval by the local ethics

committee.

2.3 Motivation Study

The second part of the study was dedicated to

understanding the motivation to follow up or not on

the symptom checker recommendations. We invited

all the patients to take part in the interview. All the

patients invited to interview were notified of the

objectives of the study and of the purpose of the

interviews in written.

The study was designed as a series of semi-structured

interviews based on a topic guide, which was

developed by the study team and reviewed by the

local ethics committee.

Interviews has the following agenda:

1. Was the symptom checker user friendly and easy

to use?

1.1. Did you understand the recommendations?

1.2. Did you understand the diagnosis that the

symptom checker provided?

2. Did the symptom checker influence your decision

to follow up with a doctor?

3. Why did you decide to follow up or not after a

symptom checker session?

The guide included open questions to provide good

understandings of the patients’ opinion, without

influencing from an interviewee. We applied a

probing approach (Eeg-Olofsson, Johansson, Linder,

& Leksell, 2020) during interviews to encourage

patients to express openly.

Demographic characteristics of the study population

are presented in the table 1.

Table 1: Demographic details of the study population.

Gender Aver

age

Age

> 60

Education

Higher Secondary Below

secondary

346

Males

42.6 106 98 238 10

521

Females

43.1 134 125 384 12

Total

867

42.9 240 223 622 22

The interviews were performed by two members of

the research team.

Each interviewee was in written assured of anonymity

and confidentiality of the study participation. They

were informed of their right to revoke personal data

from the study at any time.

The interview duration was 10 to 20 minutes and all

of them provided sufficient information for the study.

A combination of thematic analysis and a grounded

theory approach was used to analyze the

interviewees’ responses (Marcellus, 2005; Skeat &

Perry, 2008).

3 RESULTS

3.1 Recommendations Follow Up

Results

In total we invited 3615 patients to have a symptom

checker screening; 2374 of them agreed to run a

symptom checker screening; 867 of them agreed to

participate in the study. The proportion of the patients

who agreed to have a symptom checker screening.

Study of the User Behaviour Caused by Automatic Recommendation Systems Call to Action

501

Table 2 contains the details of the patients’

acceptance rates.

Table 2: Study population.

Patients

approached

Agreed to the

symptom checker

screening

Agreed to

participate in

the study

3615 2374 (65,67%) 867 (23,98%)

Table 3 provides a classification of the study

participants into groups.

Table 3: Distribution of the patients according to the call to

action.

Gender Green Yellow Red

Male

(346)

(18,79%)

213

(61,6 %)

(19,65%)

Female

(521)

101

(19,38%)

341

(65,45%)

(15,16%)

Total

(867)

166

(19,15%)

554

(63,9%)

147

(16,96%)

Table 4 presents follow up rates for each group of

patients.

Table 4: Follow-up rates.

Gender Green Yellow Red

Male

(70,8%)

204

(95,8 %)

68 (100%)

Female 74

(73,3%)

329

(96,5%)

79 (100%)

Total 120

(72,3%)

545

(96,2%)

147

(100%)

3.2 Behaviour Motivation

We have interviewed 48 patients, who agreed on this

part of the study and signed a consent form.

The thematic analysis identified 4 core topics

related to the patients’ responses, namely: (1)

usability, (2) intention to use, (3) trust, and (4) data

protection.

The first core topic highlighted the usability and

clearness of the questions and recommendations. The

second core topic demonstrated that the patients have

strong wish to use the symptom-checker for

screening. The core topic number 3 identified the

issue of trust. The topic 4 identified personal data

security and protection awareness. Table 5 present

selected responses to support the obtained results.

Table 5: Selected answers.

Core topic and

Code

Selected responses

Usability Time It doesn’t take much time to do a

screening [A1]

Usability clearness The questions were clear to me, I could

answe

them easily [A2]

Usability Time This tool can save my time [A3]

Intention to use

Second opinion

I think this tool can be best used as a

second opinion [A4]

Intention to use

Second opinion

I can use it to better prepare to the

doctor’s questions [A5]

Intention to Use

Second opinion

The portal allows to quickly update It

can help me to be better informed on

my health conditions [A6]

Intention to Use

Second opinion

I still want to talk to a real doctor [A7]

Trust Explanation The system explains the results clearly,

so I can make an informed decision

[A8]

Trust Explanation I can see the logic behind the decision,

so I can base my decision on what the

system recommends [A9]

Trust User

enthusiasm

The recommendations are clear, so I

can rely on them when deciding to

follow up [A10]

Data protection

Privacy

It’s good that no personal data is

collected [A11]

Data protection

Privacy

The session is anonymous, so I can trust

the system [A12]

4 DISCUSSION

As we can see from the follow up rate results, the

urgent recommendations cause a 100% follow up

rate. A red message always resulted in the doctor’s

visit. Yellow messages with a recommendation to

visit a doctor soon if the health conditions do not

improve also resulted in almost a 100% follow up rate

with actual number of 96,2%. The only

recommendation that filtered some of the patients and

prevented some of the not required visits was a green

message with 72,3% follow up rate. So, we can

clearly see that symptom checker screening doesn’t

result in a significant decrease of the load on

healthcare professionals. This is supported by the

quantitative study results. The patients emphasized

the ease of use of the tool and clearness of the

recommendations it gives. However, they perceived

it as rather a second opinion tool or a tool that helps

to prepare to the doctor’s visit.

Our results add a discussion point to the

efficiency of the symptom checker in terms of being

able to decrease a load on the healthcare professionals

or even replace them in some situations.

We didn’t observe any behavioral difference

between males and females.

HEALTHINF 2022 - 15th International Conference on Health Informatics

502

4.1 Limitations of the Study

In this study we did not assess the accuracy and other

efficiency characteristics of the symptom checker.

This will become a subject of the further study. The

second limitation of the study is that we didn’t

research how the screenings affected the length of the

actual doctor’s visit, which will also become an

objective of the further studies.

5 CONCLUSIONS

Our findings suggest that automatic symptom

checkers don’t significantly decrease a load on

healthcare professionals as they are mostly perceived

as a second opinion tool.

ACKNOWLEDGEMENTS

This work was supported financially by the Ministry

of Science and Higher Education of the Russian

Federation, Agreement No. 075-15-2021-1013

(08.10.2021) (Internal project number

13.2251.21.0067).

REFERENCES

Aluttis, C., Bishaw, T., & Frank, M. W. (2014). The

workforce for health in a globalized context – global

shortages and international migration. Global Health

Action, 7(SUPP.1). Retrieved October 6, 2021 from

https://doi.org/10.3402/GHA.V7.23611

Baker, A., Perov, Y., Middleton, K., Baxter, J., Mullarkey,

D., Sangar, D., Johri, S. (2020). A Comparison of

Artificial Intelligence and Human Doctors for the

Purpose of Triage and Diagnosis. Frontiers in Artificial

Intelligence, 3. Retrieved October 6, 2021 from

https://doi.org/10.3389/FRAI.2020.543405

Battineni, G., Sagaro, G. G., Chinatalapudi, N., & Amenta,

F. (2020). Applications of Machine Learning Predictive

Models in the Chronic Disease Diagnosis. Journal of

Personalized Medicine 2020, Vol. 10, Page 21, 10(2),

21. Retrieved October 6, 2021 from https://doi.org/

10.3390/JPM10020021

Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S.,

& Palser, T. (2021). Accuracy of online symptom

checkers and the potential impact on service utilisation.

PLOS ONE, 16(7), e0254088. Retrieved October 6,

2021 from https://doi.org/10.1371/JOURNAL.PONE.0

254088

Djahel, S., Smith, N., Wang, S., & Murphy, J. (2015).

Reducing emergency services response time in smart

cities: An advanced adaptive and fuzzy approach. 2015

IEEE 1st International Smart Cities Conference, ISC2

2015. Retrieved October 6, 2021 from https://doi.org/

10.1109/ISC2.2015.7366151

Eeg-Olofsson, K., Johansson, U.-B., Linder, E., & Leksell,

J. (2020). Patients’ and Health Care Professionals’

Perceptions of the Potential of Using the Digital

Diabetes Questionnaire to Prepare for Diabetes Care

Meetings: Qualitative Focus Group Interview Study.

Journal of Medical Internet Research, 22(8). Retrieved

October 6, 2021 from https://doi.org/10.2196/17504

Fraser, H., Coiera, E., & Wong, D. (2018). Safety of

patient-facing digital symptom checkers. The Lancet,

392(10161), 2263–2264. Retrieved October 6, 2021

from https://doi.org/10.1016/S0140-6736(18)32819-8

Griffith, K., Evans, L., & Bor, J. (2017). The affordable care

act reduced socioeconomic disparities in health care

access. Health Affairs, 36(8), 1503–1510. Retrieved

October 6, 2021 from https://doi.org/10.1377/

HLTHAFF.2017.0083/ASSET/IMAGES/MEDIUM/2

017.0083FIGEX1.EPS.GZ.GIF

Karimi-Shahanjarini, A., Shakibazadeh, E., Rashidian, A.,

Hajimiri, K., Glenton, C., Noyes, J., Colvin, C. J.

(2019). Barriers and facilitators to the implementation

of doctor-nurse substitution strategies in primary care:

a qualitative evidence synthesis. The Cochrane

Database of Systematic Reviews, 2019(4). Retrieved

October 6, 2021 from https://doi.org/10.1002/

14651858.CD010412.PUB2

Marcellus, L. (2005). The Grounded Theory Method and

Maternal-Infant Research and Practice. Journal of

Obstetric, Gynecologic & Neonatal Nursing, 34(3),

349–357. Retrieved October 6, 2021 from

https://doi.org/10.1177/0884217505276053

Poppe, A., Jirovsky, E., Blacklock, C., Laxmikanth, P.,

Moosa, S., Maeseneer, J. de, Peersman, W. (2014).

Why sub-Saharan African health workers migrate to

European countries that do not actively recruit: a

qualitative study post-migration. Global Health Action,

7(1), 24071. Retrieved October 6, 2021 from

https://doi.org/10.3402/GHA.V7.24071.

Skeat, J., & Perry, A. (2008). Grounded theory as a method

for research in speech and language therapy.

International Journal of Language and Communication

Disorders, 43(2), 95–109. Retrieved October 6, 2021

from https://doi.org/10.1080/13682820701437245

Stewart, D., Lang, E., Wang, D., & Innes, G. (2019). Are

emergency medical services offload delay patients at

increased risk of adverse outcomes? Canadian Journal

of Emergency Medicine, 21(4), 505–512. Retrieved

October 6, 2021 from https://doi.org/10.1017/

CEM.2018.478

Wu, Q., Zhao, L., & Ye, X.-C. (2016). Shortage of

healthcare professionals in China. BMJ, 354, i4860.

Retrieved October 6, 2021 from https://doi.org/

10.1136/BMJ.I4860

Zihindula, G., John, R. A., Gumede, D. M., & Gavin, M. R.

(2019). A review on the contributions of NGOs in

addressing the shortage of healthcare professionals in

rural South Africa. http://www.editorialmanager.com/

cogentsocsci, 5(1). Retrieved October 6, 2021 from

https://doi.org/10.1080/23311886.2019.1674100

Study of the User Behaviour Caused by Automatic Recommendation Systems Call to Action

503