Ensuring Socio-technical Interoperability in Digital Health

Innovation Processes: An Evaluation Approach

Tim Scheplitz

Research Group Digital Health, Technische Universität Dresden, Dresden, Germany

Keywords: Digital Health Innovation, Evaluation, Interoperability, ReEIF.

Abstract: Integrating Digital Health Innovations (DHI) into healthcare practice remains a challenging task for

innovators. They continuously seek for actionable ways to fulfil the complex web of requirements set by the

target environment. A socio-technical understanding of interoperability offers structurization to this

complexity and becomes a key property that innovators want to ensure during the innovation process.

Nevertheless, scientific guidance remains abstract rather than applicable. This research paper builds on this

point and follows the question how innovators can evaluate their DHI process holistically and tangibly to

promote the later integration into complex healthcare systems. It therefore presents an evaluation approach

based on the Refined eHealth European Interoperability Framework (ReEIF) and results of a qualitative

content analysis. Here, detailed descriptions of the six ReEIF levels and 181 potential parameters for a self-

assessment tool have been derived from prior literature. These findings stimulate future research on

interdependencies within identified aspects of socio-technical interoperability and promote applicable tools

for digital health innovators.

1 INTRODUCTION

Successful implementation of Digital Health

Innovations (DHI) into daily healthcare remains a

challenging task for science and practice. Prior

research is facing this issue from different points of

view. It provides definitional relationships of key

concepts and types of digital health soultions (Iyawa et

al., 2016; Otto et al., 2018), consolidates valuable

insights of domain-specific diffusion barriers (Hobeck

et al., 2021; Otto and Harst, 2019), and derives success

factors to overcome hurdles (Kowatsch et al., 2019;

Otto, 2019). Also, such scientifically stated knowledge

already found its way into international political

programs, recommendations and interdisciplinary

frameworks (European Commision, 2019; WHO,

2015). Despite this knowledge gain, the capability to

foster DHI and to create intelligent and valuable digital

health systems varies immensely from country to

country (Prim et al., 2017).

Extending the knowledge base might also cause

an increase of complexity, that innovators have to be

aware of and that they have to manage. They face the

challenge to overview all crucial factors valid for

https://orcid.org/0000-0003-0070-4561

their context and to derive the right actions at the right

time. Practice-oriented research lacks thereby in

offering usefull supporting tools to ensure the later

integration into complex health systems and their

Health Information Systems (HIS) landscapes.

This paper adresses the practice-oriented focus

mentioned above. It follows the research question

how innovators can be supported in evaluating their

DHI process holistically to promote the later

integration into complex healthcare systems. It

presents an evaluation approach that will lead to an

evaluation tool for DHI practioners in further work.

This approach seeks to assess the integration

capability of DHI in the modern healthcare practice.

It uses a socio-technical understanding of Digital

Health (DH) interoperability, basically defined as the

ability of two or more (health information) systems to

effectively and efficiently perform tasks together

(HIMSS, 2020; HL7 International, 2021). Thus,

interoperability is required on different technical and

non-technical levels when a DHI has to be integrated

into healthcare practice. The presented approach

bases on a European consented framework of socio-

technical interoperability, the Refined eHealth

264

Scheplitz, T.

Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach.

DOI: 10.5220/0011009800003123

In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2022) - Volume 5: HEALTHINF, pages 264-275

ISBN: 978-989-758-552-4; ISSN: 2184-4305

European Interoperability Framework (ReEIF), to

structure the complex requirement environment

(eHealth Network, 2015). Furthermore, this paper

presents results of a qualitative content analysis of

existing, domain-specific evaluation approaches to

operationalize the chosen key property. For all six

interoperability levels defined in the ReEIF,

comprehensive descriptions as well as a set of

potential evaluation parameters were formulated.

This paper is structured as follows: Within the

next section, foundations of this work are given by

presenting its practice-oriented motivation, the

conceptual evaluation approach as well as a socio-

technical interpretation of interoperability in the DH

domain. After presenting methodical details in

section 3, findings of the analysis are presented for

each ReEIF level as enriched descriptions. Further

thoughts on ensuring interoperability from an

innovator’s perspective are discussed before

limitations and a conclusion of this work are given.

2 FOUNDATIONS

2.1 Use Case of DHI Practice

This work is intended to stimulate both research and

practice, but focusses primarily the latter perspective

of DH innovators. In this paper, “DH innovators”

describe one or more professionals who are

responsible for the management of a DHI process

starting from defining an initial idea and ending

(hopefully) in integration of a new DH artifact into

healthcare practice. The DHI process itself might

differ due to the artifact’s specificity, intended usage

scenario and organizational circumstances but

somehow pass typical stages of idea creation,

conceptualization, requirements analysis,

development and prototyping as well as a final

integration into existing HIS landscapes. Whether a

DHI process is managed by using agile process

models like SCRUM, traditional sequential

development models (e.g., Waterfall- or V-model) or

hybrid models, evaluating the current progress with

intended objectives is always essential. Also,

evaluation is broadly used from a quality

management perspective within Plan-Do-Check-Act

cycles as stated, for instance, in DIN EN ISO 9000,

ISO/IEC 20000 or ISO/IEC 27001.

In all mentioned stages of a DHI process,

innovators are confronted with the domain’s

complexity. Various interdisciplinary requirements

have to be managed to reach the inherent goal of a

successful integration into healthcare practice. This

task becomes even more crucial as it is mandatory for

further scaling objectives. Thus, innovators shall be

supported in continuous or repeating evaluation

activities to assess how the ongoing DHI process

ensure the integration capability of their DHI artifact

in a pilot environment or, later on, in healthcare

practice.

2.2 Evaluation Approach

Based on the formulated support scenario for DH

innovators, a contextual concept of an evaluation tool

was created (see Figure 1). For completeness, this

concept is presented here in simplified form. Starting

by the target group, Innovators aim to develop one or

more DHI and integrate the new artifact(s)

successfully into healthcare practice. They are

thereby confronted with the challenge of managing

technical and non-technical requirements set by the

target environment. Here, an evaluation tool (working

title "Interoptimeter") intends to support innovators

assessing the integration capability of their

innovation (evaluation object). The “Interoptimeter”

provides self-assessment questionnaires with selected

items from different interoperability perspectives and

presents innovators a structured report. This report

includes information about how different

interoperability perspectives are already addressed

and what topics DH innovators should pay more

intention to in further work.

As motivated above, socio-technical

interoperability is used as a key property to ensure

this integration task (evaluation top criterion). It

describes the ability of two or more systems - in this

case the DHI as an artifact) and the target

environment of the digital healthcare practice - to

harmonize with each other and to perform common

tasks effectively and efficiently. For this purpose,

“socio-technical interoperability” is generally

systematized via the ReEIF that defines six

interoperability levels for the DH domain (evaluation

sub-criteria). Details of this conceptualization are

presented in the following sections.

Typical for frameworks, the ReEIF systemizes

interoperability but does not provide their tangible

operationalization. Here, the presented research

wants to contribute on this need. It strives for

actionable activities, tasks, or duties that can be

reviewed within a self-assessment by innovators. The

conducted qualitative content analysis enhanced

existing descriptions and derived potential evaluation

parameters. In Figure 1 the parts of the overall

evaluation approach that are served by the results

presented here are highlighted in dark gray.

Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach

265

Figure 1: Paper's focus (dark gray) within concept of the “Interoptimeter” evaluation tool.

The evaluation approach also takes differences of

DHI types into account. For this purpose, established

taxonomies are to be used for systematic description

of a DHI. Such characterizations enable adaptive

evaluation activities as evaluation parameters can be

sorted or filtered by relevance. However, the design

of this functionality is the focus of future research and

is only mentioned here as a supplement.

2.3 Interoperability as Key Property

Interoperability is basically defined as the ability of

two or more application or information systems to

effectively and efficiently perform tasks together

(Gibbons et al., 2007; HIMSS, 2020; HL7

International, 2021; Zeinali et al., 2016). Following

the socio-technical understanding of HIS research,

interoperability is understood as a construct of

technical and non-technical dimensions (da Silva

Serapião Leal et al., 2019; Kuziemsky and Weber-

Jahnke, 2009). Within this paper, the attribute “socio-

technical“ is provocatively chosen to highlight the

societal dimensions besides technical interpretations.

Socio-technical interoperability is seen as a key

property for a DHI’s successful integration as this

general construct comprises the ability of the DHI and

the status quo environment to commonly perform on

four general perspectives. (Figure 2).

These perspectives might be simplified as

follows: Interoperability of a DHI and its target

environment requires a symbiosis according to:

• Existing Technical Systems that collaborate

directly or indirectly with a DHI

• People who use a DHI or who are affected by its

usage (professionals and patients)

• Organizations that manage a DHI’s operation

• Regulations that define duties and limits

Figure 2: Construct of socio-technical interoperability.

2.4 ReEIF Model

In 2015, the European Commission's Working Group

“eHealth Network” published the Refined eHealth

European Interoperability Framework (ReEIF)

(eHealth Network, 2015; European Commission et

al., 2013). This unifying framework is primarily

intended to support activities in the context of

interoperability and standardization challenges. It

thereby provides a structuring benefit for

communication and decision-making processes

regarding DH solutions. In this sense, the ReEIF

serves as a consented language for the analysis of DH

solutions. It defines six technical and non-technical

levels of interoperability within the context of DH:

Legal & Regulatory, Policy, Care Process,

Information, Applications and IT Infrastructure.

Figure 3 presents the explanations of each of the

ReEIF's interoperability levels. The eHealth Network

Evaluation object

Digital Health Innovation

Evaluation tool

“Interoptimeter”

Evaluation top-criterion

Socio-technical

interoperability

Primary evaluators

Innovators

Complex target environment

Digital healthcare practice

Taxonomies

of Digital Health artifacts

Evaluation sub-criteria

Refined eHealth European

Interoperability Framework

has to be integrated into

develop

use

evaluates

systemizes

structured via

enable adaptive

evaluation

key property

can be

described by

Evaluation parameters

Tangible activities, tasks

and duties

provide items concretized by

Regulations Organizations

Technical

Systems

People

Socio-Technical

Interoperability

HEALTHINF 2022 - 15th International Conference on Health Informatics

266

Figure 3: Description of ReEIF Levels by eHealth Network, 2015.

formulates these in light of interoperability between

two or more organizations. Despite some vagueness

for the context of integrating DHI as artifacts into

practice, the explanations still allow for delineation of

relevant topics especially in the light of DHI for

interorganizational healthcare delivery.

From a top-down perspective, the ReEIF is

already part of international recommendations. The

WHO recommends its member states to adopt the

ReEIF within their eHealth strategies and action plans

to support all involved stakeholders on the way from

innovation to implementation (Peterson et al., 2016).

The eStandard initiative (2015-2017) also built on the

ReEIF conceptualization and provided, among other

outputs, the “Interoperability guideline for eHealth

deployment projects” as well as a “Roadmap for a

sustainable and collaborative standard development”

to promote cross-border interoperability, use and

evaluation of domain-specific standards and

beneficial eHealth systems for the European people

(eStandards, 2017a, 2017b; Schulz et al., 2019).

The research community applied the ReEIF in

selected contributions. Scientists from Greece

postulated an adopted framework for digital

transformation of the national health system that

facilitates especially citizen empowerment, health

process alignment and integration of information

technology (Katehakis and Kouroubali, 2019;

Kouroubali and Katehakis, 2019). Dutch researchers

contributed a reference architecture for primary care

that uses the ReEIF to define essential functionalities

which need to be ensured by future digital platforms

or ecosystems (d’Hollosy et al., 2018). Overall, there

are single contributions from science and leading

international institutions that push the adoption of the

ReEIF but, to the author’s knowledge, its transfer to

guidance for DHI practice remains uncomplete.

3 METHODS

The presented evaluation concept requires a suitable

operationalization of socio-technical interoperability

regarding the development of an innovative DH

artifact and its integration into modern healthcare

practice. A structuring, deductive Qualitative Content

Analysis (QCA) according to Mayring was conducted

to identify actionable evaluation items for all ReEIF

levels (Mayring, 2014). Details of the research

activities are listed below.

3.1 Literature Selection

Criteria-based evaluations are often underpinned by

literature-guided definitions of the evaluation criteria

and corresponding parameters (Alalwany, 2010).

Socio-technical interoperability was chosen as the

evaluation top criterion and further structured into six

sub-criteria covered by the six levels of the ReEIF. In

order to derive adequate parameters from existing

evaluation concepts, an explorative literature review

was opened. The PubMed, ScienceDirect,

EBSCOHost, and SpringerLink databases were

searched for articles between 2009 and 2019 that

combined "digital health" or related terms [“eHealth”,

Refined eHealth European Interoperability Framework (ReEIF)

Legal &

Regulatory

On this level, compatible legislation and regulatory guidelines define the boundaries for

interoperability across borders, but also within a country or region.

Policy On this level, contracts and agreements between organisations have to be made. The

purpose and value of the collaboration must be set. Trust and responsibilities between the

organisations are formalised on the Policy level. In governance documents the governance

of collaboration is anchored.

Care Process After the organisations have agreed to work together, specific care processes are analysed

and aligned, resulting in integrated care pathways and shared workflows. This level

handles the tracking and management of the workflow processes. The shared workflow

prescribes which information is needed in order to deliver the integrated care.

Information This level represents the functional description of the data model, the data elements

(concepts and possible values) and the linking of these data elements to terminologies that

define the interoperability of the data elements.

Applications On this level, agreements are made about the way import and export of medical

information are handled by the healthcare information systems. The technical specification

of how information is transported is at this level (communication standards). The

information systems must be able to export and import using these communication

standards. Another aspect in this level is the integration and processing of exchanged

information in user-friendly applications.

IT Infrastructure The generic communication and network protocols and standards, the storage, backup, and

the database engines are on this level. It contains all the “generic” interoperability

standards and protocols.

Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach

267

“mhealth”, “telemedicine”, “telehealth”] (Otto et al.,

2020) with "evaluation" or "assessment" in the title or

abstract. Only articles that discuss evaluation in the

context of integration efforts of DHI into practice

were included. Therefore, articles have to address

DHI diffusion, adoption, implementation or

integration as their contribution objective.

Additionally, only articles that discuss DHI from a

generic point of view have been included. Thus,

articles that focus on single DHI or on DHI for a

specific medical context were excluded from

analysis. 34 contributions were finally selected (see

Appendix 1). These contributions include concepts,

methods, frameworks as well as initiatives or

programs for DHI evaluation.

3.2 Qualitative Content Analysis

The 34 relevant sources were selected as the analysis

set for a deductive, qualitative content analysis

(Mayring, 2014). A structuring approach was

implemented to detect concretizing text passages for

each ReEIF level. In this regard, the definitions listed

in Figure 3 were set as detection criteria. Passages

were assigned to a ReEIF level if they concretize that

definition for the scenario of DHI integration to a

tangible item of action or consideration.

The free online tool QCAmap was used to

perform the coding collaboratively with four research

assistants. After about 10% of the material run, a

check of the coding rules took place. At this point, we

identified the issue that a consistent degree of

abstraction is difficult to apply during coding. We

decided to continue for the moment as we could not

define a suitable rule as well as anchor examples and

added a second analysis iteration afterwards. Here,

we decided to whether a marking was suitable to

provide an actionable parameter for our evaluation

approach (low degree of abstraction) or enriched a

more detailed description of a ReEIF level (medium

degree of abstraction). Both objectives of the

conducted QCA are illustrated in Figure 4.

After completion of the material run,

approximately 4500 markings were set. The coding

results are provided as a raw data set in Appendix 2

to ensure the traceability of detected findings. As

mentioned, the markings showed differences in the

degree of abstraction but could be subsumed into 122

descriptive aspects and 181 potential parameters.

The aspects for the detailed description of the

ReEIF levels are distributed as follows: Legal &

Regulatory 8; Policy 21; Care Process 49;

Information 20; Application 7; IT-Infrastructure 17.

The detected potential evaluation items are

distributed as follows: Legal & Regulatory 13; Policy

66; Care Process 39; Information 11; Application 32;

IT-Infrastructure 20.

Figure 4: Objectives of QCA.

4 FINDINGS

The extent of findings allows only a condensed

presentation of the analysis results at this place. For

each ReEIF level, detailed descriptions are given

below that are taken from those detected text passages

that represent concretizations but no actionable

parameters for evaluation. These detailed

descriptions enrich the existing explanations of

ReEIF given in Figure 3 and adopt them for the

context of integration activities of DHI into

healthcare practice. A complete list of these

descriptive aspects can be found in Appendix 3.

Within Appendix 4, all potential evaluation

parameters are formulated as self-assessment

questions and are offered specially to practice.

As mentioned above, Appendix 2 provides the

raw data set of literature markings which are source

of the following findings. In addition, Table 1

highlights those sources that particularly shaped the

findings for each ReEIF level. Due to their extent, the

reports of PAHO and WHO as well as the study by

Dattakumar et al. caused a majority of the markings

across all ReEIF levels (approximately 60%) and are

therefore listed separately. However, this observation

is purely quantitative in nature and is relativized by

the adjusting subsumption of the second analysis

iteration. The qualitative influence of articles of

smaller length is particularly highlighted for the

contributions of Lau & Price 2017 and Greenhalgh et

al. 2017, as they also push a holistic, socio-technical

approach to increase the adaptation of DHI into

healthcare practice.

ReEIF Levels

Detailed

Description of

ReEIF Levels

Parameters

of ReEIF Levels

Degree of

Abstraction

Coded Text

Degree of Applicability for Evaluation Approach

HEALTHINF 2022 - 15th International Conference on Health Informatics

268

Table 1: Most influential sources of each ReEIF level.

ReEIF Level Ke

Resource

Overall

Dattakumar et al., 2013; PAHO, 2016;

WHO, 2016; WHO, 2015

Legal &

ulator

Kowatsch et al., 2019; Momentum, 2012

Policy

Chang, 2015; Khoja et al., 2013;

Kowatsch et al., 2019; Lau and Price,

2017; Scirocco, 2016

Care Process

Chang, 2015; Lau et al., 2017; Lau and

Price, 2017; National Qualit

Forum, 2017

Information HIMSS Analytics, 2017; Lau et al., 2017

Application

Chang, 2015; Kowatsch et al., 2019;

Tamburis et al., 2012

IT-

Infrastructure

Chang, 2015; Khoja et al., 2013; Van Dyk

et al., 2012

4.1 Legal & Regulatory

This level describes fundamental as well as domain-

specific public regulations and laws at regional,

national or international level with regard to certain

rights and values, esp. equity, equality, justice,

security, liability, privacy, confidentiality and ethics.

They regulate and ensure, among other things,

personal rights, medical procedural rules, public

structures of healthcare delivery, and data processing

conditions. To ensure interoperability in this

dimension, DH innovators can take action by

identifying relevant regulations and guidelines,

ensuring compliance to them, and/or requesting of

specific consulting services and advisory.

Ensuring interoperability on the “Legal &

Regulatory” level is mostly understood from the

innovator’s perspective of a specific DHI as being

aware of and comply with the current and/or future

legal circumstances. Unlinked from a specific DHI

project, there might be also opportunities for

innovators to participate in design or reformation

processes on a legal level depending on an

innovator’s influence, position and possibilities to

invest the required amount of time.

4.2 Policy

This interoperability level includes basic as well as

specific policies or guidelines between organizations,

esp. between the organizational background of

innovators and contractually involved parties

(Clinicians, IT businesses, funding agencies and

individuals (e.g., Patient) and their compliance

(governance). Also, intraorganizational policies (for

instance of single hospitals) are of interest for

innovators, as they eventually ensure or inhibit the

adoption of a DHI within clinical practice. Those

policies need to be clarified, negotiated, documented,

communicated and fulfilled by innovators and

appropriate parties regarding aspects like:

organizational compatibility; liability; sustainability;

safety, security and privacy; competencies; quality

assurance & management; value propositions;

business principles; technical support; operations and

maintenance; working collaborations and

cooperation; education and training as well as

licenses and accreditations.

Ensuring interoperability on the “Policy” level

requires, trivially speaking, the negotiation and

confirmation of agreements between all involved

stakeholders of a DHI. Depending on the specific

DHI, its usage context and its innovational degree,

this task becomes more or less complex. As policy

activities refer to a broad variety of aspects that are

also part of other interoperability levels and due to the

unknown balance between compatibility to existing

policies and the need for new agreements, the

required efforts shall be rather over- than

underestimated.

4.3 Care Process

This interoperability level addresses the alignment or

reorganization of: workflows of care delivery;

business and administrative processes; healthcare

models and programs; care plans and pathways as

well as personal interaction and communication. This

includes (re)definitions and statements about:

cooperation; coordination; competences and

responsibilities; liability of practice as well as error

prevention and risk descriptions. Among others, such

alignments or reorganizations aim to ensure: quality

of care; accuracy and disease specificity; continuity;

validity; safety and security; usability, user-

friendliness, acceptance and satisfaction of patients

and professionals as well as customizability and

individualization. Innovators may support these

efforts by confirmation of effects like: clinical

effectiveness and outcomes; patient-related

outcomes; efficiency and/or quality benefits; process

measures; treatment or medication adherence. They

therefore should consider or provide: comprehensive

description of DHI functionality; guidelines and

standards of health practice; deviation in regular

practice; patient engagement, user empowerment or

education initiatives.

Summing up, the demand for interoperability at

the care process level entails a large number of

aspects that innovators should address. Based on the

Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach

269

concrete needs from care practice, the core process

for which a DHI offers a solution must be analyzed

intensively. Of particular interest are the questions:

How do apply which users the DHI and which people

are directly affected by it and how? How does the

DHI change the existing core process? Furthermore,

dependencies or the influence on accompanying care,

administration or business processes must be taken

into account. Innovators need to balance whether a

DHI should be designed to be compatible with

established processes or the design of a DHI and its

value proposition requires changes of the status quo.

4.4 Information

The “Information” level comprises aspects of

semantic and syntactic interoperability, esp. data

types, formats and structures; data flows; and the use

of terminologies and standards. Considered data and

information sets typically consist of general health

information, clinical data, information about

decisions, system-generated data as well as

timestamps or log files. Innovators may generally

align with existing standards or participate in

standardization initiatives to ensure: accuracy;

comparability; completeness; comprehensiveness;

consistency; relevance and value; confidentiality;

reliability as well as integrity.

Ensuring interoperability on the “Information”

level requires on the innovator’s site the balancing

task of: identification and alignment with data

models, structures and formats that are determined by

the target environment of a DHI; harmonization of

those (eventually heterogenous) compatibility

requirements with own development; identification

and re-use of consented interoperability standards

provided by relevant institutions, e.g., HL7 or IHE;

and the promotion of standard adoption or initiating

standardization processes of new specifications.

Reflecting these subtasks, interoperability on

“information” level seems to be primarily ensured by

compatibility activities especially towards existing

semantic and syntactic standards. Nevertheless, a

DHI that offers a new solution for an existing problem

will probably hit a spot where the state of practice

does not offer a health information standard. Here,

innovators are able to fill this gap with self-defined

specifications and might contribute their

achievements to the synergetic community.

4.5 Application

The “Application” level comprises agreements and

their realization according to interconnectivity of

distinguished (information and) application systems,

esp. in terms of: interconnection services and data

exchange; use of communication standards and

unified terminologies to ensure robustness of

technical interfaces, sustainability as well as usability

of technical interfaces. This generally technical

dominated interoperability level does also include

human-centered aspects like end-user satisfaction and

user acceptance but with a focus on the

interconnection of a DHI with other application

systems. While the “Care Process” level addresses

usability of a DHI itself – simplified as its use without

involvement of any other technical system – this level

considers usability aspects of DHI within an

interconnected, synergetic HIS landscape. For

instance, a professional documentation tool for a

specific indication can be autonomously usable,

intuitive and, thus, valuable but if data exchange with

central Electronic Health Record systems is not

ensured then double documentation might occur and

will decrease user acceptance.

Increasing interoperability on the “Application”

level requires knowledge about (potentially)

mandatory communication scenarios of a DHI with

existing or future application systems. Definition and

prioritization of these scenarios are key tasks for

innovators before technical interface solutions can be

derived and realized. Thereby, innovators are not

exclusively responsible on the required realization

efforts as changes of the target environment could

also foster interoperability, e.g., by supporting

communication standards like HL7 FHIR.

4.6 IT-Infrastructure

Interoperability on “IT-Infrastructure” level includes

considerations of specific properties, e.g.,

availability, performance, capacity, scalability,

reliability, stability as well as safety and security of

infrastructural components, like basic infrastructure

of electricity, physical and mobile communication

networks, required hardware, distributed server

architectures and physical databases as well as

storage units. Activities that may be considered to

fulfill interoperability on this level are, among others,

the use of technical infrastructure standards and

protocols, the establishment of infrastructural data

protection measures and validation mechanisms as

well as maintenance and failure prevention activities.

Depending on the specific characteristics of a DHI,

innovators need to consider infrastructural aspects on

international, national, regional or local level. As

infrastructures do not change rapidly, innovators shall

search for a DHI design that is compatible with existing

HEALTHINF 2022 - 15th International Conference on Health Informatics

270

infrastructures. Thus, specifying the access to required

server structures or networks, clarifying how

continuity of operations can be ensured and

implementing mechanisms to prevent or handle

potential failure as well as IT attacks are main tasks.

5 DISCUSSION

5.1 Relevance

The extent of aspects for the given descriptions of the

ReEIF levels as well as for potential parameters for

the presented evaluation approach motivates the

relevance question for each item. At the highest level

of abstraction (ReEIF levels), no differentiation of

relevance can be stated in general as neglecting each

level makes the failure of a DHI integration likely.

Although the detailed descriptions are formulated

generically, the characteristics of a specific DHI, its

usage context, and the DHI project’s organizational

circumstances may assign a single aspect more or less

relevant. These three influencing factors require an

individual assessment of relevance at the level of the

formulated parameters, which cannot be provided in

a blanket manner within this paper. In order to take

this sensitivity into account, the presented approach

comprises a selection of relevant parameters for a

concrete evaluation instance based on a previous DHI

characterization (Figure 1).

5.2 Critique on ReEIF

Generally, the ReEIF suited the task of systemizing

interoperability from a socio-technical HIS point of

view. Nevertheless, from the author’s perspective,

two themes could be assigned within the ReEIF but

do not match a level’s intention perfectly and, thus,

should be highlighted more explicitly.

As the ReEIF is originally focused on

interoperability between organization, the usage of a

DHI or the user itself is not prominently represented.

Especially findings regarding usability have to be

assigned to the “Care Process” level, as a DHI

generally intends to support healthcare activities, or

to the “Application” level, as data exchange within

interconnected HIS components might be crucial for

usability to ensure continuity of information flow.

Considering the extent of the “Care Process” level

presented here, it might be valuable to distinguish

user-centered topics (“Use of DHI”) from process-

centered topics. Other authors promote a similar

separation of a DHI’s usage without any

communication scenario to other technical systems

from the alignment and continuity of process

landscapes in a target environment of connected HIS

(van Mens et al., 2020).

Another vagueness occurs while placing aspects

about required data for a DHI’s functionality into the

right ReEIF levels. Especially in the light of data-

centered DHI and the progress of AI application in

healthcare, valid access to required data sources

becomes a central topic for innovators. Thereby,

“required data” rather combines all three technical

ReEIF levels than perfectly fit into a single one. Even

though the interplay of syntax and semantics

(Information), technical system interfaces and

communication standards (Application) as well as

appropriate connection to networks, server

architectures and databases is implicated, it shall be

highlighted for future data-centered DHI.

5.3 Dominance in Interoperability

Interoperability, in its technical and non-technical

manner, is a property that targets two or more systems

as a unit, not as single parts. Ensuring interoperability

in the context of this paper depends therefore on the

constitution of both the DHI (as an artifact) and the

target environment. Reflecting the findings against

this background, it might be valuable for innovators

to differentiate the way of how they should act to

ensure interoperability: by alignment and providing

compatibility or by declaration of requirements on the

target environment’s site. Simplified, when a DHI

takes its place within an existing target environment,

three principles of dominance in interoperability

might occur (Figure 5): I. Dominance of DHI – the

DHI stimulates changes in the target environment

which ensure interoperability; II. Dominance of target

environment – the target environment declares

mandatory requirements that have to be aligned with;

and III. Interdependent adjustments – interoperability

ensured by coordinated changes on both sites.

Figure 5: Principles of Dominance in Interoperability.

Target

Environment

Digital Health

Innovation

I. DHI stimulates changes in target environment

II. Target environment declares mandatory requirements

III. Interdependent adjustments

Continuum

II.

III.

Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach

271

The findings as well as the dependency on a

specific DHI indicate that these principles should not

be seen as absolute categories. Rather, it shall be

understood as a continuum within interoperability

efforts can be assessed from an innovator’s

perspective. Regarding the findings, an innovators

opportunity to ensure interoperability on “Legal &

Regulatory” aspects as well as in existing “IT-

infrastructures” tend to compatibility activities to the

status quo (II). On the other hand, DHI that provide

new solutions for healthcare practice and new

beneficial value propositions will influence the way

how healthcare is delivered and how “Care

Processes” are conducted (I). “Policy”, “Information”

and “Application” level require a balance of activities

striving to changes within established structures and

the alignment with mandatory conditions (III).

5.4 Limitations

The scope and quality of the presented results have to

be assessed under consideration of some limitations.

In particular, inherent constraints on objectivity due

to the qualitative, interpretive research approach

result in three aspects that are stated here and

motivate future research.

Degree of Abstraction. The chosen differentiation of

two types of analytical findings (detailed descriptive

aspects vs. potential evaluation parameters) as well as

the high rate of subsumption (4500 initial markings to

300 finally used descriptive aspects and evaluation

items) point to one issue: the definition and

application of a common degree of abstraction as a

coding guideline. Despite the fact that the coding

rules were adjusted for comprehension within the

material run, a relatively high variance had to be

handled during the interpretation cycles. The decision

whether an "actionable" evaluation item was detected

could not be made for all markings without doubt. In

front of this circumstance, the two types of findings

were defined. The created detailed descriptions of the

ReEIF levels could be used as a starting point for

argumentative-deductive derivation of further

actionable evaluation items to improve completeness

and fit for different DHI types.

Suitability. One of the guiding motivations of this

work is making knowledge about DHI integration

into healthcare practice accessible and actionable for

innovators. For this purpose, no restrictions were

made with regard to DHI types, neither in the design

of the evaluation approach nor during the

parameterization of socio-technical interoperability.

No artifact classes were explicitly excluded or

prioritized. The scope of detected aspects achieved in

this way was purchased with an initial lack of general

fit of the individual item. For example, the question

about confirmation of positive effects on patient self-

management (CP-13) is irrelevant for DHI without

patient involvement. Other items, such as the question

about mechanisms to prevent system overload (ITI-

02), may have universal relevance. Against this

background, the detected items are to be assessed in

terms of fit for different DHI types along established

taxonomies in order to correspond to the adaptive

character of the underlying evaluation approach

(Figure 1).

Fuzziness. The method-related limitation of

objectivity as well as the interrelation of detected

aspects causes a certain fuzziness between separately

listed evaluation items or gives the impression of

redundancy in certain cases. For example, the items

CP-10 ("Is continuity of care ensured?") and CP-34

("Are seamless transitions between tasks of care

ensured?") differ only slightly in their different

perspectives (patient-centered vs. professional-

centered) on continuous, trouble-free care processes.

Despite this limitation, the results presented benefit

from the diversity of perspectives gained as well as

from the breadth of detected aspects. Further

investigations could contribute to an improved

distinction of the evaluation items, for example, by

using a matrix structure.

Additionally, the limited topicality of this work

has also to be named. This analysis started in 2020

and included only articles published until 2019. Due

to the Covid-19-Pandemie and other circumstances,

conduction, documentation and publishing of this

work were delayed. Therefore, chosen literature data

bases have been checked for additional resources, but

the extent of articles matching the inclusion criteria is

scarce. Nevertheless, three articles are mentioned

here for completeness that generally confirm

motivation and presented findings (Bashi et al., 2020;

Guo et al., 2020; Villumsen et al., 2020). Bashi et al.

reviewed science articles about the development of

DH frameworks for chronic healthcare scenarios and

recommend the re-use of frameworks for evidence-

based DHI processes including evaluation activities.

Guo et al. see the need of more pragmatic DHI and

evaluation approaches to face the “no evidence, no

implementation – no implementation, no evidence”

paradox in DH. They highlight the awareness of

socio-technical requirements faced by different

stakeholders and call for new approaches to facilitate

responsible growth of the DH domain. Villumsen et

HEALTHINF 2022 - 15th International Conference on Health Informatics

272

al. provide “an overview of the predominant

approaches and methodological recommendations to

national and regional monitoring and evaluation of

eHealth”. Even though their main perspective

addresses policy makers and appropriate initiatives,

they recommend continuous, transparent monitoring

and evaluation to facilitate learnings and

implementation progress.

5.6 Further Research

The given results are currently being accompanied by

an ongoing expert study. In 1-to-1 interview sessions,

the experiences of experts from various professions

(science, medicine, management and IT) are being

collected in order to investigate the following

questions, among other: How should differences in

ReEIF levels in terms of relevance and criticality be

assessed for definable DHI types? How shall

interdependencies between ReEIF levels as well as

between items taken into account? How can

evaluation parameters be linked to action items and

their termination within typical innovation phases?

6 CONCLUSION

This research paper addresses the challenge of

innovators to fulfill the complex, interdisciplinary

web of requirements for a successful integration of a

DHI into modern healthcare practice. It presents an

evaluation approach based on the key property of

interoperability in a socio-technical manner. Along

six interoperability levels defined by the ReEIF, this

paper explores potential evaluation parameters for a

self-assessment tool and provide detailed descriptions

of ReEIF levels. While the organizational intended

ReEIF generally suits the scenario of integrating a

DHI into healthcare practice, the framework could

benefit from little adjustments by a sound

distinguishment of usability facets and the

consideration of dominance in interoperability. The

findings enrich both further research and practice to

support innovators handling the complexity of

domain specific target environments and, thus, to

increase successful integration rates of future DHI.

ACKNOWLEDGEMENTS

This research is part of the EFRE-funded research

project “Häusliche Gesundheitsstation”. I thank all

colleagues, partners and supporters who made this

research possible.

REFERENCES

Alalwany, H., 2010. Cross disciplinary evaluation

framework for e-health services (PhD Thesis). Brunel

University Brunel Business School PhD Theses.

Bashi, N., Fatehi, F., Mosadeghi-Nik, M., Askari, M.S.,

Karunanithi, M., 2020. Digital health interventions for

chronic diseases: a scoping review of evaluation

frameworks. BMJ Health Care Inform. 27, e100066.

https://doi.org/10.1136/bmjhci-2019-100066

Chang, H., 2015. Evaluation Framework for Telemedicine

Using the Logical Framework Approach and a

Fishbone Diagram. Healthc. Inform. Res. 21, 230.

https://doi.org/10.4258/hir.2015.21.4.230

d’Hollosy, W.O.N.–, van Velsen, L., Henket, A., Hermens,

H., 2018. An Interoperable eHealth Reference

Architecture for Primary Care, in: 2018 IEEE

Symposium on Computers and Communications

(ISCC). Presented at the 2018 IEEE Symposium on

Computers and Communications (ISCC), pp. 01090–

01095. https://doi.org/10.1109/ISCC.2018.8538576

da Silva Serapião Leal, G., Guédria, W., Panetto, H., 2019.

Interoperability assessment: A systematic literature

review. Comput. Ind. 106, 111–132. https://doi.org/

10.1016/j.compind.2019.01.002

Dattakumar, A., Gray, K., Jury, S., Biggs, B.-A., Maeder,

A., Noble, D., Borda, A., Schulz, T., Gasko, H., 2013.

A unified approach for the evaluation of telehealth

implementations in Australia.

eHealth Network, 2015. Refined eHealth European

Interoperability Framework. European Commission,

Brussels.

eStandards, 2017a. Interoperability guideline for eHealth

deployment projects.

eStandards, 2017b. Roadmap for a sustainable and

collaborative standard development: recommendations

for a globally competitive Europe.

European Commision, 2019. MAFEIP Monitoring and

Assessment Framework for the European Innovation

Partnership on Active and Healthy Ageing [WWW

Document]. URL https://www.mafeip.eu/

European Commission, Directorate-General for the

Information Society and Media, Deloitte & Touche,

2013. eHealth European Interoperability Framework

study report. Publications Office, Luxembourg.

Gibbons, P., Phd, N.A., Flewelling, T., Jepsen, T., Md,

D.K., Larson, J., Ritter, J., Md, M.R., Selover, S.,

Stanford, J., 2007. Coming to Terms: Scoping

Interoperability for Health Care Health Level Seven

EHR Interoperability Work Group.

Guo, C., Ashrafian, H., Ghafur, S., Fontana, G., Gardner,

C., Prime, M., 2020. Challenges for the evaluation of

digital health solutions—A call for innovative evidence

generation approaches. NPJ Digit. Med. 3, 110.

https://doi.org/10.1038/s41746-020-00314-2

Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach

273

HIMSS, 2020. What is Interoperability in Healthcare?

[WWW Document]. URL https://www.himss.org/

what-interoperability

HIMSS Analytics, 2017. Electronic Medical Record

Adoption Model: EMRAM [WWW Document]. URL

https://www.himssanalytics.org/europe/electronic-

medical-record-adoption-model

HL7 International, 2021. General Information about HL7

[WWW Document]. URL http://www.hl7.org/about/

FAQs/index.cfm?ref=nav

Hobeck, R., Schlieter, H., Scheplitz, T., 2021. Overcoming

Diffusion Barriers of Digital Health Innovations

Conception of an Assessment Method, in: Proceedings

of the 54th Hawaii International Conference on System

Sciences. p. 3654.

Iyawa, G.E., Herselman, M., Botha, A., 2016. Digital

Health Innovation Ecosystems: From Systematic

Literature Review to Conceptual Framework. Procedia

Comput. Sci. 100, 244–252. https://doi.org/10.1016/

j.procs.2016.09.149

Katehakis, D.G., Kouroubali, A., 2019. A Framework for

eHealth Interoperability Management. J. Strateg. Innov.

Sustain. 14, 51–61.

Khoja, S., Durrani, H., Scott, R.E., Sajwani, A., Piryani, U.,

2013. Conceptual Framework for Development of

Comprehensive e-Health Evaluation Tool. Telemed. E-

Health 19, 48–53. https://doi.org/10.1089/tmj.20

12.0073

Kouroubali, A., Katehakis, D.G., 2019. The new European

interoperability framework as a facilitator of digital

transformation for citizen empowerment. J. Biomed.

Inform. 94, 103166. https://doi.org/10.1016/j.jbi.20

19.103166

Kowatsch, T., Otto, L., Harperink, S., Cotti, A., Schlieter,

H., 2019. A design and evaluation framework for digital

health interventions. It - Inf. Technol. 61, 253–263.

https://doi.org/10.1515/itit-2019-0019

Kuziemsky, C.E., Weber-Jahnke, J.H., 2009. An

eBusiness-based Framework for eHealth

Interoperability. J. Emerg. Technol. Web Intell. 1, 129–

136. https://doi.org/10.4304/jetwi.1.2.129-136

Lau, F., Hagens, Simon, Zelmer, Jennifer, 2017. Benefits

Evaluation Framework, in: Handbook of EHealth

Evaluation: An Evidence-Based Approach [Internet].

University of Victoria.

Lau, F., Price, M., 2017. Clinical adoption framework, in:

Handbook of EHealth Evaluation: An Evidence-Based

Approach [Internet]. University of Victoria.

Mayring, P., 2014. Qualitative content analysis: theoretical

foundation, basic procedures and software solution.

Klagenfurt.

Momentum, 2012. Momentum European Momentum for

Mainstreaming Telemedicine Deployment in Daily

Practice [WWW Document]. URL http://www.teleme

dicine-momentum.eu/

National Quality Forum, 2017. Creating a Framework to

Support Measure Development for Telehealth 1–53.

https://doi.org/10.1007/s00442-013-2847-9

Otto, L., 2019. Implementing and Scaling up Telemedicine

Initiatives: Beyond User-Centeredness, in: 2019

IEEE/ACS 16th International Conference on Computer

Systems and Applications (AICCSA). Presented at the

2019 IEEE/ACS 16th International Conference on

Computer Systems and Applications (AICCSA), IEEE,

Abu Dhabi, United Arab Emirates, pp. 1–6.

https://doi.org/10.1109/AICCSA47632.2019.9035232

Otto, L., Harst, L., 2019. Investigating Barriers for the

Implementation of Telemedicine Initiatives: A

Systematic Review of Reviews, in: AMCIS 2019

Proceedings. Presented at the AMCIS 2019, Cancun.

Otto, L., Harst, L., Schlieter, H., Wollschlaeger, B., Richter,

P., Timpel, P., 2018. Towards a Unified Understanding

of eHealth and Related Terms - Proposal of a

Consolidated Terminological Basis. Proc. 11th Int. Jt.

Conf. Biomed. Eng. Syst. Technol. 2, 533–539.

https://doi.org/10.5220/0006651005330539

Otto, L., Harst, L., Timpel, P., Wollschlaeger, B., Richter,

P., Schlieter, H., 2020. Defining and Delimitating

Telemedicine and Related Terms-An Ontology-Based

Classification. Stud. Health Technol. Inform. 268, 113–

122.

PAHO, 2016. Framework for the Implementation of a

Telemedicine Service. Pan American Health

Organization, Washington DC.

Peterson, C.B., Hamilton, C., Hasvold, P., 2016. From

innovation to implementation: eHealth in the WHO

European region. WHO Regional Office for Europe,

Copenhagen, Denmark.

Prim, A.L., Filho, L.S., Zamur, G.A.C., Di Serio, L.C.,

2017. The relationship between national culture

dimensions and degree of innovation. Int. J. Innov.

Manag. 21, 1730001. https://doi.org/10.1142/S13639

1961730001X

Schulz, S., Stegwee, R., Chronaki, C., 2019. Standards in

healthcare data. Fundam. Clin. Data Sci. 19–36.

Scirocco, 2016. Scirocco Maturity Model for Integrated

Care [WWW Document]. URL https://www.scirocco-

project.eu/maturitymodel/

Tamburis, O., Mangia, M., Contenti, M., Mercurio, G.,

Rossi Mori, A., 2012. The LITIS conceptual

framework: measuring eHealth readiness and adoption

dynamics across the Healthcare Organizations. Health

Technol. 2, 97–112. https://doi.org/10.1007/s12553-

012-0024-5

Van Dyk, L., Schutte, C.S., Fortuin, J., 2012. Development

of a Maturity Model for Telemedicine. South Afr. J.

Ind. Eng. 23 (2), 61–72.

van Mens, H.J.T., Duijm, R.D., Nienhuis, R., de Keizer,

N.F., Cornet, R., 2020. Towards an Adoption

Framework for Patient Access to Electronic Health

Records: Systematic Literature Mapping Study. JMIR

Med. Inform. 8, e15150. https://doi.org/10.2196/15150

Villumsen, S., Adler-Milstein, J., Nøhr, C., 2020. National

monitoring and evaluation of eHealth: a scoping

review. JAMIA Open 3, 132–140. https://doi.org/

10.1093/jamiaopen/ooz071

WHO, 2016. Monitoring and evaluating digital health

interventions: A practical guide to conducting research

and assessment. World Health Organization.

HEALTHINF 2022 - 15th International Conference on Health Informatics

274

WHO, 2015. The MAPS Toolkit: mHealth Assessment and

Planning for Scale 106.

Zeinali, N., Asosheh, A., Setareh, S., 2016. The conceptual

model to solve the problem of interoperability in health

information systems, in: 2016 8th International

Symposium on Telecommunications (IST). Presented

at the 2016 8th International Symposium on

Telecommunications (IST), IEEE, Tehran, Iran, pp.

684–689. https://doi.org/10.1109/ISTEL.2016.78819

APPENDIX

All appendices are available via the following link:

https://cloudstore.zih.tu-dresden.de/index.php/s/kSsi

ZwNXdd24XKF

Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach

275