Usability Evaluation of a Community-led Innovation Mobile App

André Castello Branco

, Eveline Sacramento

, Eliza Oliveira

, Oksana Tymoshchuk

Maria Antunes

, Margarida Almeida

, Luís Pedro

, Fernando Ramos

and Daniel Carvalho

Department of Communication and Arts, University of Aveiro, DigiMedia, Aveiro, Portugal

Department of Communication and Arts, University of Aveiro, INET-md, Institute of Ethnomusicology, Aveiro, Portugal

Keywords: Mobile Application, Usability, Heuristic Evaluation, Community-led Innovation.

Abstract: Digital media can facilitate collaborative processes among local agents, value endogenous resources, and

promote assets associated with territory. This article presents the results of a study concerning the

development and validation of a mobile app for promoting the relationship among agents of the Portuguese

Centro region’s communities/entities. This paper focuses on the results of a heuristic evaluation of the mobile

app carried out with two groups of experts in Digital Technologies, Tourism, Health, and Well-Being, besides

providing an overview of the mobile app that was developed and a theoretical background regarding

community-led innovation, usability, and heuristics. For the CeNTER app prototype evaluation itself, the use

of Nielsen's heuristics, a MATCH-MED scale, together with a Think-Aloud Protocol allowed us to improve

its usability. This article contributes to a reflection about the evaluation of mobile apps in the scope of

territorial-based innovation initiatives, engaging its stakeholders in the process.

1 INTRODUCTION

Community-led initiatives have great potential in the

development of cultural, tourism, and environmental

projects, enabling valuing endogenous resources and

promoting innovation in the territory. Digital

technologies can be particularly useful in community-

led initiatives, allowing to recreate a "virtual

proximity" between the different actors involved in

the territory's development process (Martínez-Rolán

et al., 2019). They can offer several advantages to

community-led initiatives, as, for example,

facilitating innovation, because of access to

information and experiences; sharing practices,

leading to knowledge co-creation and the emergence

https://orcid.org/0000-0002-6493-6938

https://orcid.org/0000-0003-0839-4537

https://orcid.org/0000-0002-3518-3447

https://orcid.org/0000-0001-8054-8014

https://orcid.org/0000-0002-7819-4103

https://orcid.org/0000-0002-7349-457X

https://orcid.org/0000-0003-1763-8433

https://orcid.org/0000-0003-3405-6953

https://orcid.org/0000-0003-0108-8887

of new ideas; and enabling their members to improve

their practices through a continuous engagement in a

meaningful participatory environment (Saint-Onge &

Wallace, 2012; Snyder & Wenger, 2010).

However, several studies indicate that

community-led initiatives face several challenges,

especially in rural areas (Brown & Nylander, 2009;

Marré & Weber, 2010). These challenges are related

to the fact that local participants, namely elderly

people, have difficulties accessing the Internet,

present limited digital skills, and lack access to

technological equipment. To overcome these

challenges, new political measures are required to

empower local communities to the social, cultural,

and economic valorisation of territories.

Branco, A., Sacramento, E., Oliveira, E., Tymoshchuk, O., Antunes, M., Almeida, M., Pedro, L., Ramos, F. and Carvalho, D.

Usability Evaluation of a Community-led Innovation Mobile App.

DOI: 10.5220/0011263000003323

In Proceedings of the 6th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2022), pages 81-88

ISBN: 978-989-758-609-5; ISSN: 2184-3244

This study was developed under the CeNTER

Program, which has as one of its objectives to

understand the active role those digital technologies

can play in the process of territorial innovation (Silva

et al., 2021). In this context, a digital platform (mobile

app) is being designed, whose primary focus is to

promote collaboration between the various agents

involved in territorial-based innovation processes.

The CeNTER mobile app intends to be a tool that

encourages internal and external interactions and

facilitates communication and collaboration

processes, to strengthen existing mediation strategies

and stimulate the creation of new ones, as well as new

ideas/activities (Tymoshchuk et al., 2019). However,

the design of a mobile app for community mediation

is still a great challenge due to the variety of

initiatives and the diversity of stakeholders (public

institutions, companies, and communities) involved

in territorial-based innovation processes.

This paper presents the results of a heuristic

evaluation of the CeNTER app prototype, which will

provide the development cycle with crucial usability

data, required to converge to a final solution that will

offer a coherent and effective experience about the

distinguishing features of this app to its users. The

study is supported by a User-Centered Design

approach, which focuses on satisfying the user desires

and needs (Hartson & Pyla, 2012). Also, it is framed

by the User Experience (UX) theoretical basis,

guiding the prototype evaluation in the CeNTER

Program scope, which will crucial the development

of the CeNTER mobile app.

2 BACKGROUND

2.1 Territorial Innovation

Innovation can be translated as the systematization of

knowledge shared between actors in the spaces in

new products and services. Systematized knowledge,

when interacting with the social, historical, and

cultural constructs for which they are intended,

change the behaviour of social groups in a systemic

way, enabling the development by promoting

territorial innovation (Ferreira, 2020). In

geographically close groups, these historical-socio-

cultural constructs are perceived in a systemic way,

which characterizes the territories (Keating, 2020).

Territorial innovation is a complex process that

results from an intentional collective action promoted

by a set of actors and organizations (companies,

universities, local government bodies, communities)

that interact to develop innovation (Morgan, 1997).

So, the initiatives led by groups coordinated from the

bottom up respond to the development needs of

territories, as they address ecological, social,

economic, and political problems of global resonance

(Seyfang & Smith, 2007, p. 585).

Furthermore, the sharing of information to solve

local problems has become more effective as a result

of the integration provided by the dissemination of

information and telecommunication technologies,

especially mobile apps (Diniz et al., 2019), further

enhancing the importance of mobile apps in the

territorial-based innovation domain. These digital

solutions must meet the community's needs to foster

the local economy. To ensure that a product meets

users' needs, such mobile digital solutions must be

developed under the User Experience (UX)

theoretical basis, which is crucial to gather relevant

information regarding the interaction with the

product.

2.2 User Experience

An experience is a complex event, which is created in

the mind of the user and is influenced by many

factors, being a completely personal issue (Knight,

2019). Therefore, User experience is subjective and is

related to how the user feels regarding a created

product. Further, the current concept of UX is strictly

related to emotional results that emerged from the

user when interacting with a product, such as pride,

joy, and fun (Bernhaupt & Pirker, 2013). Therefore,

creating an experience is not just about how the

product is designed, which structures were

implemented or whether state-of-the-art technology

is used (Knight, 2019). It is also about how the

specific product can help the user to accomplish their

tasks successfully and how the user feels when

getting involved with the product.

UX is composed of three factors: usability,

usefulness, and emotional impact, and generates a

memory related to the product interaction. Emotional

impact is the affective component of the user

experience, focusing on the system as a means that

affects the user's feelings. Usefulness focuses on the

use of a system to achieve goals and accomplish

specific tasks. Usability is constituted by

effectiveness, efficiency, ease-of-use, learnability,

and the user satisfaction, being the practical

component of user experience (Hartson & Pyla,

2012).

The usability factor contributes to the elaboration

of a product that is easy to use and is pivotal in

creating digital products since it provides products

with a low level of difficulty, reducing the platform's

CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications

disuse (Partridge, 2017). Since ensuring high

usability for a mobile app is of great importance, as a

predictor of its acceptance and of its success, is

crucial in all phases of app testing (Muchagata &

Ferreira, 2019), including early testing with experts.

2.3 Usability

Usability is an essential concept in developing digital

interfaces, focusing on users and contexts of use, and

ensuring they can achieve their goals with efficiency,

effectiveness, and satisfaction. The usability concept

was defined by Nielsen (2012) as an attribute with

qualitative nature that assesses the ease of use of user

interfaces. Usability directly relates to methods for

enhancing this attribute during the design process

phase that includes the following five main

components: (1) the ability to learn - the ease with

which users complete basic tasks during the primary

interaction with the interface; (2) efficiency - the

speed with which users reach their goals after they

have learned to interact with the platform; (3)

memory - the easiness with which users interact with

the interface after a period without using it; (4) errors

- the number of errors that are made by the user and

the ease with which it is possible to correct them; (5)

satisfaction - a component related to the desire and

pleasure expressed by the user during and after the

interaction with a specific product (Nielsen, 2012).

Due to several specificities related to the small

screen, storage capacity, and energy consumption of

mobile phones, it is essential that these apps meet a

set of basic requirements, such as: being easy to use,

being flexible, having a simple and intuitive interface,

allowing the user to easily adapt to different contexts

of usage, etc. (Feijó et al., 2013; Kumar & Mohite,

2018).

In this sense, usability evaluation of mobile apps

is a mandatory process to ensure that such apps are

practical, effective, and easy to use (Kumar &

Mohite, 2018). Usability evaluation is the generic

designation for a set of methods aiming at interface

inspection. The final objective is to identify usability

issues, through the indication of the severities’ level

of each issue (Nielsen, 1994). These methods include

heuristic evaluation, cognitive pathway, consistency,

pattern inspection, among others. One of the

commonly available and employed methods for

assessing and improving interfaces is heuristic

evaluation (Nielsen, 1994).

2.4 Heuristic Evaluation

Nielsen's heuristics (1994) is a systematic evaluation

method allowing the identification of problems in

user interface design that involves interface analysis.

As the author states, heuristic evaluation requires

fewer resources than other methods to detect usability

issues. As Nielsen (1994) states, “basically, a set of

evaluators inspects the interface with respect to a

small set of fairly broad principles, which are referred

to as the ‘heuristics” (p. 152).

This specific method is based on a 10 “heuristic”

items checklist, which can be used in interface

specifications, prototypes, or complete systems.

Nielsen's 10 heuristics include the following

guidelines (Nielsen, 1994): (1) Visibility of the

system; (2) Match between system and real-world; (3)

User control and freedom; (4) Consistency and

standards; (5) Error prevention; (6) Minimise the

user’s memory load; (7) Efficiency of use and

performance; (8) Aesthetic and minimalist design; (9)

Help and documentation; (10) Help users identify,

analyse, and improve from errors.

According to Nielsen and Landauer (1993),

between three to five evaluators are recommended to

perform a heuristic evaluation, which can identify

around 75% to 95% of the problems, considering that

an individual evaluator usually finds around 35% of

them.

2.4.1 MATCH-MED Scale

Testing the usability of mobile apps can be done using

several sets of checklists and protocols available to

carry out heuristic evaluations. However, the vast

majority of these scales are geared towards generic

systems, demanding adaptations to fulfil specific

requirements of mobile apps (Hashim & Isse, 2019;

Inostroza et al., 2016). Considering that the CeNTER

application is aimed at Tourism, Health, and Well-

being clusters, a set of MATCH-MED usability

heuristics was used.

The MATCH-MED scale aims to evaluate the

usability of mHealth systems on smartphones and was

developed based on Nielsen's generic heuristics, with

the addition of three heuristics specifical for mobile

devices (Lacerda et al., 2015; Salazar et al., 2012),

which consist of: (1) Minimization of human-

computer interaction: Considering that typing on

mobile touch screen keyboards is more error-prone

than on conventional keyboards, it is essential to

minimise user interaction with the mobile app; (2)

Physical interaction and ergonomics: Given the

limited screen size of a mobile device, the action

Usability Evaluation of a Community-led Innovation Mobile App

controls must present suitable sizes and minimal

distance from each other, ensuring the user does not

press a button by mistake; (3) Readability and Quick

View: Considering that mobile apps are generally

used in dynamic contexts, it is essential to ensure

quick access to system information by the user (e.g.,

at a glance).

According to Zhang and Adipat (2005), a

comprehensive usability study for a mobile

application should assess a variety of issues such as

interface design, ease of use, and perceived attitude

by the user and measures related to the application's

performance.

3 THE CENTER PROTOTYPE

The team researchers developed a CeNTER mobile

application prototype based on an in-depth study

under this project. That research included a

systematic review of the literature, a benchmarking of

the communities' digital platforms, a series of

interviews with local stakeholders, and two focus

groups with representatives of community-led

initiatives (Oliveira et al., 2021). The results of these

methodological procedures made it possible to

identify the main difficulties found on community-led

initiatives and led to the proposed digital solution that

aimed to fill these gaps.

The researchers chose a card-based user interface

design that is both simple and innovative in

appearance. This type of interface design tries to

provide information in a readable format, easy to

navigate, allowing an overview of the application's

content and quick access to all categories (Seifi,

2015). Considering the requirements and

functionalities defined as important in the proposed

digital solution, the medium-fidelity prototype was

developed with the support of “Principle” software,

an interactive user interface, and animation design

software.

The prototype’s content evaluated by experts was

organised according to the categories defined by the

team as being essential in the mobile application:

Initiatives; Events; Entities; Volunteers; Resources

and Highlights. These categories were defined during

the Focus Group sessions with the local initiatives,

through a User-Centered Design approach. In the

CeNTER app context, these categories are called

"Tabs" and represent the main relevant elements in

the community dynamics. Such “Tabs” are organised

in horizontal lines on the screen and have secondary

content that is modified according to the preferences

defined by the user in the app settings (Branco et al.,

2021).

By touching each one of the Tabs, a vertical

opening occurs on the screen, where it is possible to

view suggestions (of local Initiatives, for example)

through cards displayed on the screen. Each card

presents a single content and appears ordered in a

carousel layout.

Figure 1: Home screen of the version of the medium-

fidelity prototype “CeNTER” evaluated by experts.

4 METHODOLOGY

The evaluation process for the CeNTER mobile

application prototype is outlined according to an

iterative design methodology (Lorenz et.al., 2010). It

provides for the initial testing of app mockups with

team members, evaluating medium-fidelity

prototypes by experts and laboratory tests with end-

users. This article presents the second stage of

evaluation of the prototype, which consisted of

heuristic evaluation by experts to detect and correct

issues related to the prototype's usability.

Through dissemination seminars, organized by

the CeNTER team, it was possible to recruit local

stakeholders to actively participate in the project as

experts. To accomplish the assessment, two panels of

experts were organized according to the evaluators’

expertise. In this sense, the first group consisted of

five experts in the field of digital technologies, having

deep knowledge and experience in developing

interfaces. The second group consisted of five experts

in the fields of Tourism, Health, and Well-being, who

have knowledge of their specific domain and

involvement in local community projects.

To determine the severity of usability issues for

this study, a heuristic checklist was developed to

evaluate the application prototype, based mainly on

Nielsen's 10 heuristics (1994). Besides that, three

heuristics of MATCH-MED scale (Salgado et al.,

CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications

2019) were added to allow the evaluation of mobile

apps in health and well-being areas. It is important to

note that the researchers chose to add these items on

the MATCH-MED scale since Nielson's heuristics do

not consider specific characteristics and limitations of

mobile devices (Hashim & Isse, 2019), as it was

pointed out before.

A total of 44 items were identified in this

checklist, which was employed to evaluate the

prototype usability. The heuristic evaluation process

took place in three stages, according to the proposal

of Nielsen (1994): (i) preparation phase, in which the

prototype screens for evaluation and the list of

heuristics were defined and organised; (ii) evaluation

phase, which consisted of collecting data from each

expert, individually. The evaluators tested the

prototype by identifying the guidelines that were

violated and the degree of severity of the problem.

In addition, the Think-Aloud Protocol (Jaspers,

2009) was also employed to obtain immediate

feedback from experts about their experience of

interacting with the prototype. The application of this

method allowed the qualitative evaluation of the

prototype based on the experts' verbal comments.

The tests took place at the facilities of the

University of Aveiro, with the first group on October

28-31, 2019, and the second group on November 11-

29, 2019. In total, 78 screens of the developed

application prototype were evaluated. This evaluation

consisted of two phases: in the first phase, the experts

freely explored the prototype and commented on their

doubts about the CeNTER program, and in the

second, the experts filled a table of design-oriented

heuristics for mobile phone interfaces and apps.

User-Centered Design approach

was used as an

interactive design process to get users feedbacks

and their needs in each phase of design evaluation

process.

5 RESULTS AND DISCUSSION

This section presents the results obtained from the

usability test, through a heuristic evaluation, of the

CeNTER application prototype with the two groups

of experts (five users in each group). To consolidate

the heuristic validation results performed by both

groups, a grid was created for each group to gather all

of the obtained results.

A total of 174 usability problems were identified.

After eliminating the duplicate issues, 155 unique

usability problems remained. Table 1 shows the

number of problems identified by each group of

evaluators, the severity of such problems and their

average severity.

Regarding the severity of problems, it is important

to refer that Group 1 (G1) identified 46 (42,2%)

cosmetic problems, 39 (35,8%) small usability

problems, 21 (19,3%) main usability problems and 2

(1,8%) usability “catastrophes”. The group of digital

technologies experts identified two problems with

score level four. These problems referred to the

second heuristic: “Correspondence between the

system and the real world” and were related to the

following items: “The proposed interactions in the

application are similar to real actions”; and

“Information appears in a logical and natural order”.

Those are relevant feedback because the prototype is

applied to promoting community-led initiatives and

those participants may present different levels of

digital literacies. In this sense, a simple and intuitive

interface, easily adaptable to different contexts of

usage (Feijó et al., 2013; Kumar & Mohite, 2018) is

important to make sure individuals use the application

properly, which in this case is represented by different

territorial agents (communities, entities, networks

and citizens).

Table 1: Number of Problems and Average Severities

identified by each group of evaluators.

Group

Experts

Total

problems

Average

severity

Digital Technologies

Experts

109 1,78

Tourism, Health and

Well-being Experts

46 1,53

Regarding the severity of the found problems, it is

important to note that Group 2 identified 14 (30,4%)

cosmetic problems, 26 (56,5%) small usability

problems, and 6 (13%) main usability problems. It

should be noted that no problems with a score of level

4 (usability catastrophes) were reported by the second

group.

Tables 2 and 3 show the type of usability

problems and average severities identified by each

group of evaluators. With the highest Average

Severity, Group 1 identified the heuristics: Help and

Documentation (2.75), Recognition rather than

Reminder (2.5), and Interaction between person and

application (2.33). Group 2 reported the following

Heuristic Severities Averages: Flexibility and

Efficiency (2.25), Help and Documentation (2.2)

Recognition rather than Reminder (2), and Interaction

between person and application (2). Therefore, both

Usability Evaluation of a Community-led Innovation Mobile App

groups consistently assessed the severity of the

heuristic violation. The Mean Average Severity in

these two groups was low, with a score close to 1.78

(G1) and 1.53 (G2).

The difference in results among both groups also

occurred in the type of the identified problems. The

problems identified by Group 2 were more related to

the suitability of the prototype for each application

domain, as well as the efficiency and type of

functionality available to users (Table 3). For

example, experts of Group 2 did not observe

inconsistencies rectified after the first round of tests,

such as different icons representing the same

function, or the absence of the return icon on some

screens.

Table 2: Type of usability problems and average severities

identified by Group 1 of evaluators.

Heuristic

G 1 - Digital

Technologies Experts

Frequency

N=109

Average of

Severities

Visibility of system status 11 2.18

Match between system and the

real world

11 2.27

User control and freedom 13 1.54

Consistency and standards 7 1.57

Error prevention 5 1.6

Recognition rather than recall 4 2.5

Flexibility and efficiency of use 7 1.85

Aesthetic and minimalist design 12 1.66

Help users recognize, diagnose,

and recover from errors

10 1.1

Help and documentation 4 2.75

Interaction between person and

application

3 2.33

Physical interaction and

ergonomics

8 1.75

Readability and layout 14 1.5

As it is possible to see in Table 2, the maximum

number of usability problems, identified by Group 1,

included: Readability and layout (14, mean severity

1.5), followed by User controls and free will (13,

mean severity 1, 54) and Aesthetics and minimalist

design (12, medium severity 1.66). In Table 3, Group

2 identified the following number of heuristic

violations: User controls and exercises free will (11,

mean severity 1.45), Avoid errors (7, mean severity

1.86), and Matching the system to the real world (6,

mean severity 1.5). Identifying these heuristic

violations will make it easier to identify and prioritise

issues that need urgent attention before the final

deployment of the application.

The application of the three heuristics proposed

by the MATCH-MED scale allowed the identification

of 28 usability problems related to the different

specificities of mobile devices, namely "Readability

and layout" (16) and "Physical interaction and

ergonomics" (8).

Table 3: Type of usability problems and average severities

identified by Group 2 of evaluators.

Heuristic

G 2 - Tourism, Health,

and Well-being Experts

Frequency

N=46

Average of

Severities

Visibility of system status 1 1

Match between system and the

real world

6 1.5

User control and freedom 11 1.45

Consistency and standards 3 1.66

Error prevention 7 1.86

Recognition rather than recall 1 2

Flexibility and efficiency of use 4 2.25

Aesthetic and minimalist design 1 1

Help users recognize, diagnose,

and recover from errors

4 1.9

Help and documentation 5 2.2

Interaction between person and

application

1 2

Physical interaction and

ergonomics

0 0

Readability and layout 2 1

The Think-Aloud Protocol application allowed

the collection of 124 suggestions for improving the

prototype under evaluation, which will be considered

in the following stages of the mobile application

development. The first group of experts made 83

suggestions for improvement, which were very

focused on technical aspects and improving

interactions. For example, “standardise the ‘save’

icons on the right side of the ideas screen with the rest

of the application” (G1E1) and “Drag and drop visual

feedback is required” (G1E2), respectively, for the

evaluators E1 and E2 of Group 1.

In contrast, the second group of experts mainly

commented on the concept of the CeNTER

application, functionalities, and effectiveness and did

not mention technical aspects. This group made

several suggestions regarding the lexical inaccuracies

CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications

used in the application. It is important to note that

both experts' groups highlighted the "Resources" and

"Volunteer" components as being the most innovative

features of this application: "Resources and

volunteers are the differentiating elements of the

platform" (G2E5). Overall, the prototype was

described as having a simple interface, with an

appropriate layout, easy to use and not requiring

much effort from the user.

The study's results demonstrate that usability

testing is an effective way to significantly improve

the interface of a future mobile app, favouring the

user experience, as mentioned by other authors

(Lacerda et al., 2016; Salazar et al., 2012). The

methods and techniques of application production

were supported by methodologies oriented to the

user’s requirements and were anchored in a spiral of

evaluation of the prototype, resulting in improved

versions of the pilot app, attending to users'

statements, and developing an application with a

more attractive look & feel.

6 CONCLUSIONS

As the development of the CeNTER prototype

requires an iterative design process, performing early

tests is a crucial part of the system design to detect

usability problems and make essential improvements.

Through the techniques used for the presented

evaluation, it was possible to achieve value data, as

the experts are from different areas of knowledge.

Therefore, according to the current study findings,

the identification of usability problems facilitated the

identification and prioritisation of problems that need

urgent attention before the final implementation of

the application. In this case, the heuristic "control and

freedom" deserves special attention, since it received

the major severity score by both groups. Furthermore,

a decrease in the heuristic values between the two

groups was verified, which leads to the conclusion

that the changes made after the first round of tests

helped for the improvement of the prototype.

This study also showed the importance of

developing heuristic usability evaluation scales. As

already mentioned, the researchers applied three

items proposed by the MATCH-MED scale, which

allowed us to identify usability problems related to

the specificities of mobile devices. The evaluation of

the prototype, not only by professionals in the field of

digital technologies but also by professionals in areas

related to the application, such as tourism, health, and

well-being is also important. It is essential to mention

that the professionals in the areas of tourism, health,

and well-being, who participated in this study, have a

strong connection with different community

initiatives and deeply know the local associative

dynamics, which was very advantageous for

evaluating the prototype.

Future work includes performing laboratory tests

with end-users, which are of most importance in this

project, requiring accommodation of the experts’

suggestions for improvements. Some limitations of

this work include the fact that Principle software does

not allow to test some features and interactions of the

prototype, such as writing on the interface and

changing the letters’ size for accessibility. Besides,

the Principle software only works with iPhone, which

limits the evaluation using Android devices.

Specifically, it is hoped that this app can make a

difference in everyday life, promoting community-

led initiatives, towards adding value to local

resources and, therefore, the increment of the region’s

social, cultural, and economic levels.

ACKNOWLEDGEMENTS

This work is financially supported by national funds

through FCT – Foundation for Science and

Technology, I.P., under the project UIDB/05460/

2020

REFERENCES

Bernhaupt, M. & Pirker, R. (2013). Evaluating User

Experience for Interactive Television: Towards the

Development of a Domain-Specific User Experience

Questionnaire, Human-computer Interact. - Lect. Notes

Comput. Sci. 8118, 642–659.

Brown, R. & Nylander, A. (2009). Community Leadership

Structure: Differences Between Rural Community

Leaders' and Residents' Informational Networks.

Journal of the Community Development Society, 29, p.

71-89. https://doi.org/10.1080/15575339809489774

Branco, A.C., Carvalho, D., Sacramento, E., Tymoshchuk,

O., Oliveira, E., Antunes, M.J., Pedro, L., Almeida, M.,

Ramos, F. (2021). Prototyping and evaluating a mobile

app to promote territorial innovation. In Proceedings of

CeNTER Conference, (pp. 73-81).

Diniz, E., Cernev, A., Bastos, T. P., Pinheiro, A., La

Rovere, R., Querette, E., Morel, L., & Arroio, A.

(2019). Inovação em Serviços na Economia do

Compartilhamento. Publisher: Saraiva.

Feijó, V. C., Gonçalves, B. S. & Gomez, L. S. (2013).

Heurística para avaliação de usabilidade em interfaces

de aplicativos smartphones: Utilidade, produtividade e

imersão. Design e Tecnologia, 3(06), 33-42.

Usability Evaluation of a Community-led Innovation Mobile App

Ferreira, L.M. (2020). Sustainable generation of innovative

enterprises in territories: a proposal of an ecosystem

management model. In Proceedings of CeNTER, 55-62.

Hartson, R. & Pyla, P. S. (2012). The UX Book: Process

and guidelines for ensuring a quality user experience.

Elsevier.

Hashim, N. & Isse, A. (2019). Usability Evaluation Metrics

of Tourism Mobile Ap-plications. Journal of Software

Engineering and Applications, 12, 267-277.

https://doi.org/10.4236/jsea.2019.127016

Inostroza, R., Rusu, C., Roncagliolo, S., Rusu, V., &

Collazos, C. A. (2016). Developing SMASH: A set of

SMArtphone's uSability Heuristics. Computer

Standards & Interfaces, 43, 40-52.

Jaspers, M. (2009). A comparison of usability methods for

testing interactive health technologies: methodological

aspects and empirical evidence. International journal of

medical informatics, 78(5), 340-353.

Keating, M. (2020) Beyond the nation-state: territory,

solidarity, and welfare in a multiscalar Europe.

Territory, Politics, Governance, 9, 2021 - Issue 3:

Inequalities, territorial politics, nationalism.

Knight, W., 2019. UX for Developers. Northampton.

APRESS.

Kumar, B. A. & Mohite, P. (2018). Usability of mobile

learning applications: a systematic literature review.

Journal of Computers in Education, 5(1), 1-17.

Lacerda, T. C., Nunes, J. V. & von Wangenheim, C. G.

(2015). Usability heuristics for mobile phone

applications: a literature review. In Emerging

Perspectives on the De-sign, Use, and Evaluation of

Mobile and Handheld Devices (pp. 143-157). IGI

Global.

Lorenz, T., Richter, A., Köster, F. & Rataj, J. (2010, April).

Iterative design process for the development and testing

of cooperative applications. In Proceedings of the 2010

Spring Simulation Multiconference, (pp. 1-6).

Marré, A. & Weber, B. (2010). Assessing community

capacity and social capital in rural America: lessons

from two rural observatories. Community Development,

41(1), p. 92-107, https://doi.org/10.1080/15575331

003661099

Martínez-Rolán, X., Tymoshchuk, O., Piñero-Otero, T. &

Renó, D. (2019). Instagram como red de promoción e

hipermediación del turismo rural: el caso de Aldeias

Históricas. Revista Latina de Comunicacion Social, 74,

1610 - 1632

Morgan, K. (1997). The learning region: institutions,

innovation and regional renewal. Regional Studies, 31,

491-503.

Muchagata, J. & Ferreira, A. (2019). Visual Schedule: A

Mobile Application for Autistic Children-Preliminary

Study. In Proceedings of ICEIS 2, (pp. 452-459).

Nielsen, J. (1993). Usability Engineering. Morgan

Kaufmann Publishers Inc. San Francisco, CA, USA.

Nielsen, J. (1994). Enhancing the explanatory power of

usability heuristics. In Conference on Human Factors

in Computing Systems - Proceedings, 152–158.

https://doi.org/10.1145/259963.260333

Nielsen, J. (2012). Usability 101: Introduction to usability

(2012). Retrieved from URL: http://www. nngroup.

com/articles/usability-101-introduction-to-usability/.

Nielsen, J. & Landauer, T. K. (1993). A mathematical

model of the finding of us-ability problems. In

Proceedings of the INTERACT'93 and CHI'93

conference on Human factors in computing systems

(pp. 206-213). ACM.

Nielsen, J. (15 de Novembro de 2020). 10 Usability

Heuristics for User Interface Design. Retrieved from

https://www.nngroup.com/articles/ten-usability-heuris

tics/

Partridge, D. (2017). Launch Your Dream: A 30-day Plan

for Turning Your Passion In-to Your Profession. Harper

Collins Leadership.

Oliveira, E., Branco, A. C., Carvalho, D., Sacramento, E.

R., Tymoshchuk, O., Pedro, L., & Ramos, F. (2021).

End-user Evaluation of a Mobile Application Prototype

for Territorial Innovation. In ICEIS (2), (pp. 495-504).

Saint-Onge, H., & Wallace, D. (2012). Leveraging

communities of practice for strategic advantage.

Routledge.

Salazar, L. H., Lacerda, T., von Wangenheim, C. G. &

Barbalho, R. A. (2012). Customizando heurísticas de

usabilidade para celulares. In Companion Proceedings

of the 11th BSHFCS, (pp. 37-38).

Salgado, A. D., Fortes, R. P. M., Hung, P. C. & Moreira, D.

D. A. (2019). A Method for Classifying Usability

Findings to Enhance Validation of New Heuristics.

Revista de Sistemas e Computação-RSC, 9(1), 96-118.

Seifi, S. (2015). Developing an Engaging Local Community

Application as a Tool for Promoting Social Integration.

(Doctoral dissertation, Malmö University). Retrieved

from http://ls00012.mah.se/handle/2043/19614

Seyfang, G. & Smith, A. (2007). Grassroots innovations for

sustainable development: Towards a new research and

policy agenda. Environmental politics, 16(4), 584-603.

Silva, A., Antunes, M. J., Tymoshchuk, O., Pedro, L.,

Almeida, M., & Ramos, F. (2021). Barriers and

incentives to territory-based innovation processes: from

technology to interaction among actors. In Handbook of

Research on Cultural Heritage and Its Impact on

Territory Innovation and Development, (pp. 44-61). IGI

global.

Snyder, W. M. & Wenger, E. (2010). Our world as a

learning system: A communities-of-practice approach.

Social learning systems and communities of practice

(pp. 107-124). Springer, London.

Tymoshchuk, O., Almeida, A.M., Ramos, F., Pedro, L.,

Silva, P., & Renó, D. (2019). Mediação digital para a

inovação territorial: um estudo de caso múltiplo em

redes sociais digitais. Observatorio (OBS*), 13(4), 70-

88.

Zhang, D. & Adipat, B. (2005). Challenges, methodologies,

and issues in the usability testing of mobile

applications. International journal of human-computer

interaction, 18(3), 293-308. https://doi.org/10.1207/

s15327590ijhc1803_3

CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications