Implementation of Usability Testing in Measuring the Effectiveness

and Efficiency of Mobile Application

Ni Kadek Dessy Hariyanti

1 a

, IGP Fajar Pranadi Sudhana

, I Gusti Ngurah Sanjaya

and Ketut Vini Elfarosa

Department of Business Administration, Politeknik Negeri Bali, Jalan Kampus Bukit Jimbaran, Badung Bali, Indonesia

Department of Tourism, Politeknik Negeri Bali, Jalan Kampus Bukit Jimbaran, Badung Bali, Indonesia

Keywords: Usability Evaluation, Usability Testing, Effectiveness, Efficiency, Mobile Application.

Abstract: The KPN PNB has developed a mobile-based application for its members who handle transactions in

cooperatives. The mobile application released since 2021 is targeting 520 members. The KPN PNB Mobile

application is currently in the acceptance test stage so it is necessary to evaluate usability to ensure the

application has been able to meet user needs. Usability evaluation will be carried out using the usability testing

method to find out how effective and efficient the application is. Usability testing was carried out by 10

respondents by completing 4 tasks. Usability testing on the aspect of effectiveness, obtained a success rate of

83.75 percent with a very effective level of achievement category. Usability testing user on the efficiency

aspect is shown by the average Time-based efficiency of 0.0181 goals per second. When compared with the

TBE expert's time of 0.0466 goals per second, the respondent takes 0.38 times longer to complete the task

compared to the expert. The results of this usability test indicate that the application already has a good level

of usability effectiveness in its use, and respondents are able to complete tasks efficiently in using the

application.

1 INTRODUCTION

Indonesian Digital Report 2022 (hootsuite We are

Social, 2022) states that the use of mobile phones

increased by 3.6 percent during the period January-

December 2021. The use of mobile phones to support

various activities is increasingly popular in all circles.

By using a mobile phone everyone can communicate

with anyone and anywhere very quickly. They can

complete homework, school, to the office using only

one device. Even now mobile phone not only serves

as a communication tool but also used as a medium to

develop a business. The data shows 94.1 percent of

the total internet users in Indonesia. access the

internet via a mobile phone. Various needs are met

from internet access via mobile phones including

accessing information, using social media,

conducting online learning, to running business

activities. The increase in the use of mobile phones

has triggered many emerging mobile-based

applications that are designed to meet the needs of its

users. There is an increase of 16 percent in

https://orcid.org/0000-0002-3116-2366

downloading mobile-based applications throughout

2021. This shows that mobile applications are getting

special attention and experiencing rapid growth.

Business people are forced to keep up with current

mobile-based technology developments, as

evidenced by the many emerging mobile applications

that provide solutions for various needs. In its

implementation, mobile applications must be

designed attractively so that they will be effective,

efficient, and satisfying when used by users (Hashim,

Abubakar, & Hussain, 2019). However, there are still

many mobile applications on the market that are

difficult to use and learn due to low usability (Ali,

Ouda, & Capretz, 2020).

Technological developments require all parties to

digitally transform their business processes

(Hariyanti, Sanjaya, , Sutawinaya, & Sudhana, 2021).

The Bali State Polytechnic Civil Servant Cooperative

organization (KPN PNB) whose members consist of

employees in the PNB agency environment, have

utilized information technology to support their

business processes. The Bali State Polytechnic Civil

836

Hariyanti, N., Sudhana, I., Sanjaya, I. and Elfarosa, K.

Implementation of Usability Testing in Measuring the Effectiveness and Efﬁciency of Mobile Application.

DOI: 10.5220/0011892900003575

In Proceedings of the 5th International Conference on Applied Science and Technology on Engineering Science (iCAST-ES 2022), pages 836-842

ISBN: 978-989-758-619-4; ISSN: 2975-8246

Servant Cooperative has developed the Mobile

Application with the aim of responding to various

digital transformation challenges that are currently

developing. The Mobile application is an application

developed by the Bali State Polytechnic KPN in

collaboration with Bank BPD Bali, to make it easier

for members to get various services from KPN PNB.

With this application, members can make transfers

between members, transfer to and from various other

national banks, apply for loans, find out the history of

savings and loans, open deposits, pay various bills,

top up e-money and view member voucher balances.

For security, this application has been equipped with

biometric technology with fingerprint login. The

Mobile application, which was launched in August

2021, can not only be used by 520 PNB KPN

members, but also targets all students of Bali State

Polytechnic. Through this application, the entire PNB

family is expected to be able to use the cooperative's

services to get a variety of very up-to-date features,

and make it easier for them to make various payment

transactions on campus. The Mobile application is

relatively new to use in the campus environment.

Figure 1: V Model Software development life cycle.

Seeing the lack of use of the mobile application,

because KPN members do not understand the use of

the application, the level of trust in the reliability of

the application, and there are still some errors that

occur so it is necessary to carry out an evaluation.

Based on Figure 1 (Kharchenko, Kovalenko, Siora, &

Andra, 2018) where the final stage of development is

to carry out acceptance testing to find out whether the

application meets the needs of its users. The Mobile

application is at the acceptance testing stage to find

out whether the application is able to meet user needs

and can be accepted for use in supporting transaction

activities in cooperatives. In an effort to increase

application use and future development, measuring

the usability aspect of mobile applications is

important and essential in order to capture the

efficiency, effectiveness and accuracy of the interface

(Hashim & Adamu, 2017). Likewise, with the Mobile

application, because this is related to providing trust

and satisfaction to cooperative members with the

service facilities provided. Usability evaluation on

mobile applications, especially in applications related

to financial transactions, needs to be carried out with

the appropriate criteria dimensions, so that it is known

whether the application can be used effectively,

efficiently and fulfils the satisfaction and trust of its

users (Liao & Ho , 2021). In addition, the lack of

usability can be a major barrier in the process of

adopting mobile application technology (Islam,

Islam, & Mazumder, 2019). In this study, usability

evaluation of the mobile application will be carried

out using the usability testing method to determine

how effective and efficient the use of the Mobile

application is.

2 LITERATURE REVIEW

2.1 Usability Evaluation

Evaluation is a planning process, obtaining

measurement values resulting from a series of work

that has been done in an effort to provide information

that is used for other options. the information can be

used as a guide in making better decisions. Usefulness

comes from the word usable which means good use.

A good application can minimize failure and can

provide satisfaction and benefits for its users.

Usability evaluation is an evaluation conducted to

find out about the ease of using something that

describes how effectively users interact with a

product.

According to (Maguire, 2011) usability is the

main key that determines the success of an

application system. Usability is also a characteristic

of product quality where the sub-characteristics

consist of the ability for appropriate recognizability,

ease of learning (learnability), ease of operation

(operability), protection against user error (user error

protection), the beauty of the interface (user interface

aesthetics), and accessibility (ISO/IEC 25010, 2011).

A system that is not well designed and not easy to use

will make it difficult for users to learn and operate.

This will result in the system being used infrequently

or incorrectly so that the costs that must be borne by

the system user organization become high. it also

threatens the reputation of the company that

developed the system (Maguire, 2011) The

importance of usability shows that usability

evaluation is an important process in the development

stage to produce a good system. Inspection methods

involve usability experts to find usability problems by

following clues, heuristics, or tracing. Meanwhile,

Implementation of Usability Testing in Measuring the Effectiveness and Efﬁciency of Mobile Application

837

through the testing method, usability problems are

found by observing users when using the product or

system (Neilsen, 1993).

According to (Dumas & Redish, 1999) usability

is a measurement method for user experience when

using system products (software, websites, and

mobile phones). Usability can refer to how users

experience when using a product and their level of

satisfaction with the product. According to (Neilsen,

1993), usability is the experience of interacting with

the application so that users can use the application

quickly even though the user is still in the new user

category. The usability of the system can be identified

by the ease with which users perform their tasks.

According to (Neilsen, 1993) in measuring the

usability level, there are five main measures, namely:

▪ Learnability is defined as the speed at which

the user can use the system and the ease of use

in the performance of a function and the desire

of the user.

▪ Efficiency can be interpreted as a resource

instrument that has a function to achieve better

goals.

▪ Memorability is defined as user testing to

remember using the application for a certain

period of time.

▪ Errors and security can be defined how many

problems are caused by users, errors that

contain inconsistencies with what users expect

from the system.

▪ Satisfaction can be defined as freedom of

expression in using the application as well as

user comfort in using the application as well as

a subjective measure of the user's view of the

system.

Usability according to ISO 9241 is a measure of

the extent to which a product can be used correctly by

certain users to achieve certain goals with

effectiveness, efficiency, and satisfaction. Usability

measurement needs to consider several things,

namely as follows:

▪ Optimum, is considering the advantages and

disadvantages of the current analysis.

▪ Effective, which relates to the ease and speed

of users in completing the tasks given by the

researcher properly and correctly.

▪ Satisfaction, is freedom of expression on the

application or product that is being used.

In this study, we will conduct tests to determine

the success of the KPN PNB Mobile application for

users by focusing on the effectiveness and efficiency

of its use

2.2 Usability Testing

Usability testing is a method used to evaluate the user

experience of a product, whether on a website or

application. Usability testing is one way to find out

whether the user can easily use the application, how

efficiently and effectively an application can help the

user achieve its goals and whether the user is satisfied

with the application used. Usability testing is divided

into three methods, the first is expert-based testing,

which is a test that requires expert system and

interface developers who have been involved since

planning the application by using a structured method

to look for deficiencies that exist in the application

being tested. The second is user-based testing, which

in this test involves application users to complete the

tasks given by the testers. While the third is

Automatic testing; automated testing can be carried

out with the software. On the device used apply

guidelines to an interface and then can be taken into

consideration as a comparison.

Usability testing method is done by testing a

product directly to its users. Usability Testing aims to

find out the experience from the user's point of view

in a real way by looking directly and directly into the

field of seeing the process when the user is using the

application being tested. Usability testing is task-

oriented, meaning that the user receives a set of pre-

defined tasks to work on (A, M, & Alomar, 2017).

Usability testing will carry out a more in-depth

analysis of the application or system by testing the

application used by the user. In the process, the user

will complete a predetermined series of jobs. Then

direct observations will be made to the user by seeing

whether the user can be successful in completing the

given task. The test will observe the time it takes the

user to complete the given task. In addition, the

examiner also asked questions about the opinions of

application users.

The steps needed to carry out usability testing

include:

▪ Determination of Respondents. In data

collection, respondents are required to perform

usability testing. Determination of respondents

is done to determine which users will

participate in conducting the test. In this

research, 10 users of KPN PNB Mobile will be

selected as respondents in conducting usability

testing. The ten users consist of five new users

and five active users.

▪ Determination of Task Scenario. A task

scenario is a set of tasks that are determined by

the examiner and must be carried out by the

usability testing respondent. This task covers

iCAST-ES 2022 - International Conference on Applied Science and Technology on Engineering Science

838

the usability and superior features of the KPN

PNB Mobile application. The task scenario

consists of four tasks as shown in the table.

Table 1: Task Scenario Usability Testing.

Task scenario

Do a mutation check on the savings

Make a transfer to your virtual account

Do a loan simulation

▪ Implementation of Usability Testing. After the

respondent and task scenario are determined,

the next step is the implementation of usability

testing. Testing will be carried out one by one

to the respondents so that the examiners can

perform maximum analysis in finding usability

problems in KPN PNB Mobile. In the

implementation of usability testing, all

activities will be recorded including screen

recording on the respondent's mobile device.

▪ Usability testing data processing. When

conducting usability testing, observation and

data recording are carried out to analyse the

effectiveness and efficiency aspects. The data

recorded is the success in completing the task

and the time required to complete the task. The

data will be analysed with Calculations and

data analysis.

2.2.1 Effectiveness

The level of effectiveness will be analyzed based on

the level of user success in completing each task

given in usability testing. In this study, the calculation

of success rate is used to measure the usability testing

component. The success rate is used to determine the

user's understanding of the user interface and the

success of the user in completing operations based on

previously prepared tasks. In the calculation of the

success rate using the abbreviation S indicates the

abbreviation of success is given for each time the

respondent can complete the task. When the

respondent makes a mistake while doing the assigned

task, it will be given a PS statement or the

abbreviation for partial success.

   









(1)

Where SR = Success Rate; S = Success (Task

Successfully Done); PS = Partial Success; T =

Number of Tasks

After obtaining the results from the SR

calculation, then the results will be interpreted into

the level of effectiveness with range Very Ineffective

(less than 40 percent) to Very effective (more than 80

percent).

2.2.2 Efficiency

The efficiency level will be analysed based on the

completion time of each task in usability testing. And

here are the parameters used to analyse:

 

 



















(2)

3 IMPLEMENTATIONS

The evaluation of the Mobile application testing

implemented the usability testing method which was

carried out by 10 respondents as application users.

Respondents were 10 people consisting of 5 active

users and 5 inactive users. Determining respondents

based on transaction data made by application users,

taken the top 5 and bottom 5 from 40 data users,

sorted by the number of activities taken from the

application system database. Usability testing was

tested on 10 respondents to get user experience data

when accessing application features. The feature

being tested is the daily transaction feature that users

make on the mobile banking application. Based on the

results of scenario testing for the four tasks, an

effectiveness analysis will be carried out based on the

user's level of success in completing the given task to

obtain a success rate calculation. While the efficiency

analysis is obtained by calculating time-based

efficiency. The results of these two components are

used as an analysis to determine the usability level of

the Mobile application.

3.1 Effectiveness

The respondent data obtained, which has been

described in Table 1 is the result of all respondents

having completed the assigned tasks as many as four

tasks that must be done.

Based on the measurement results using equation

(1), the success rate is 83.75 percent with the level of

achievement being very effective. This shows that the

mobile application has been effective in terms of its

use and has a good usability effectiveness level.

When testing is carried out, users can indeed perform

the tasks that have been prepared, although there are

several obstacles, including adjustments to the

application interface, especially for users who rarely

Implementation of Usability Testing in Measuring the Effectiveness and Efﬁciency of Mobile Application

839

use it. However, the user has sufficient time to get to

know the interface and complete the given task.

Table 2: Data success rate.

Name

Task1

Task2

Task3

Task4

PM1

PM2

PM3

PM4

PM5

DM1

DM2

DM3

DM4

DM5

When tested, the user was able to perform the

specified task although there were some obstacles

which were mostly due to adjustments to the

application interface. However, the user has sufficient

time to get to know the interface and complete the

given task (Lazar, Feng, & Hochheiser, 2017) From

the Usability Survey it was found that most of the

participants found problems on tasks 3 and 4 which

were marked by partial success. Task 3 is to make a

transfer to the BPD account virtual account. In task 3,

the difficulty they faced was forgetting the virtual

account number from the BPD bank. They were

forced to fail the task because they had to look for

information from other menus. As for task 4 in the

form of doing credit simulations, they block the credit

time limit. Users do not understand that there are

KPN rules, the longest loan limit is when the

employee retires.

3.2 Efficiency

The efficiency level will be analyzed based on the

completion time of each task in usability testing. The

parameter used to analyze the efficiency is Time-

based efficiency, which is the testing phase used to

obtain how much time is used and the success of the

respondents in completing the tasks that have been

given. The success value obtained from the successful

completion of the task is worth one. For respondents

who can only complete several tasks, a success value

of 0.5 is given and for those who cannot complete the

task at all, a score of zero is given. The processing

time is obtained from scenario testing on respondents.

Initial testing will be carried out by experts who

are responsible for the creation and management of

the application. The testing time of the expert is used

as a basic benchmark for the normal time of the

respondent to do the tasks given by the author. The

time of the test results carried out by the expert is

obtained by running a timer when the expert starts

doing the task and stops the timer when the expert

finishes doing the task. The test carried out by the

expert gets a success score of one because it can

complete all the tasks that have been prepared in the

usability scenario. The next test scenario will be

tested on the ten respondents The results of the data

testing shown in Figure 2.

Figure 2: Data testing.

Based on equation (1) the results of TBE expert

are 0.0466 goals per second. TBE expert will be the

reference, where the maximum time by the user to

complete the task is (Shneiderman & Plaisant, 2017)

three times the expert time that has been obtained.

After getting the calculation of the average results of

the TBE expert, calculation also done for data

respondents. Based on the data test, then time-based

efficiency is calculated along with the average for the

respondents. The results of the time-based efficiency

calculated using equation (2) for the usability

scenario test of the expert and respondents are listed

in Figure 3.

Figure 3: Time-based Efficiency.

Based on calculation using equation (2), the time-

based efficiency values expert is task 1 of 0.0625

goals per second, task 2 of 0.0769 goals per second,

task 3 of 0.0278 goals per second, task 4 of 0.0192

goals per second. TBE users for the four tasks are

generated, namely for task 1 of 0.0101 goals per

second, for task 2 of 0.0128 goals per second, for task

3 of 0.0051 goals per second and for task 4 of 0.0046

goals per second. From the four TBE calculations,

iCAST-ES 2022 - International Conference on Applied Science and Technology on Engineering Science

840

the average user TBE is 0.0181 goals per second. And

the expert's time of 0.0467 goals per second.

When tested, the user's speed in using the

application and focusing on the features of the

existing application is still said to be normal and the

error rate made is also in the reasonable category

(Hussain, Mkpojiogu, Jamaludin, & Somia, 2017)

The results obtained are normal and the user gets

enough time to understand the location and function

of the features contained in the KPN PNB Mobile

application so that it does not take too long when

completing the given task. When compared to the

user's time, the number is 0.38 times longer than the

respondent in completing the task compared to the

expert, which is still in the reasonable category

(Hashim & Adamu, 2017).

4 CONCLUSIONS

Usability testing on the effectiveness analysis of the

success rate of using the KPN PNB Mobile

application, obtained a success rate of 83.75% with

the category of the level of achievement being very

effective, which indicates that the KPN PNB Mobile

application already has a good usability effectiveness

level in its use.

Usability testing on efficiency analysis of the

level of task completion in using the KPN PNB

Mobile application obtained time-based efficiency

for task 1 of 0.0101 goals per second, for task 2 of

0.0128 goals per second, for task 3 of 0.0051 goals

per second and for task 4 of 0.0046 goals per second.

the average TBE of users is 0.0181 goals per second.

When compared with the TBE expert time of 0.0466,

it is obtained that the number is 0.38 times longer for

the respondents in completing the task compared to

the expert. This means that respondents are quite fast

and able to complete tasks in using the KPN PNB

Mobile application.

From the research conducted, it can be concluded

that the usability level of the application is quite good.

Based on data from task scenarios, many application

users encounter difficulties when making inter-bank

transfers and credit simulations. Suggestions that can

be given are the need for additional transaction

facilities, for example in the form of a QRIS facility,

so that users can make transactions more easily. Users

also have not been able to optimize the use of mobile

applications outside of the tasks given, especially for

new users. This can be overcome by continuing to

provide socialization to users which can be done

regularly and also providing access to user manual

documents for new users. Meanwhile, the user's

speed in using the application and access to existing

application features is still normal and the error rate

made is also in the reasonable category. To expand

the use of the application, it is necessary to develop

the IOS version so that it can reach more users.

ACKNOWLEDGEMENTS

The authors would like to thank to Politeknik Negeri

Bali for the financing of this research and the

Research Centre and Community Services for the

assistance.

REFERENCES

A, A., M, A., & Alomar. (2017). A model for evaluating the

security and usability of e-banking platforms.

Computing 99(https://doi.org/10.1007/s00607-017-

0546-9), 519–535.

Ali, A., Ouda, A., & Capretz, L. F. (2020). A Conceptual

Framework for Measuring the Quality Aspects of

Mobile Learning. Bulletin of the IEEE Technical

Committee on Learning Technology, 14(4)(3), 31-49.

Dumas, J. S., & Redish, J. (1999). A Practical Guide to

Usability Testing. UK: Intellect.

Hariyanti, N., Sanjaya, , I., Sutawinaya, I., & Sudhana, I.

(2021). Conceptual Model for Adoption of Mobile

Banking Technology in Savings and Loans

Cooperatives. Advances in Social Science, Education

and Humanities Research, Atlantis Press, 544, 114-

120.

Hashim, N. L., Abubakar, H. I., & Hussain, A. (2019).

Evaluating mobile banking application: Usability

dimensions and measurements. Proceedings of the 6th

International Conference on Information Technology

and Multimedia, 6(doi:

10.1109/ICIMU.2019.7066618), 136-140.

Hashim, N., & Adamu, R. (2017). Usability Evaluation of

Mobile Banking Application Interfaces. Journal of

Engineering and Applied Sciences, 12(DOI:

10.36478/jeasci.2017.5490.5494), 5490-5494.

hootsuite We are Social. (2022). Indonesia Digital 2022.

Canada: hootsuite.

Hussain, A., Mkpojiogu, E., Jamaludin, N., & Somia, M.

(2017). A Usability Evaluation of Lazada Mobile

Application. AIP Conference Proceedings 1891,

020059 (2017), 1891, , 020059-1-020059-6.

Islam, R., Islam, R., & Mazumder, T. A. (2019). Mobile

Application and Its Global Impact. International

Journal of Engineering & Technology IJET-IJENS, 10

(6), 104-107.

Kharchenko, V., Kovalenko, A., Siora, O., & Andra, A.

(2018). V-models of safety critical system life cycle:

Classification and application. IEEE 9th International

Conference on Dependable Systems, Services and

Implementation of Usability Testing in Measuring the Effectiveness and Efﬁciency of Mobile Application

841

Technologies (DESSERT), doi:

10.1109/DESSERT.2018.8409088, 1-5.

Lazar, J., Feng, J. H., & Hochheiser, H. (2017). Research

Methods in Human-Computer. New York: Morgan

Kaufmann.

Liao, S.-H., & Ho , C.-H. (2021). Mobile Payment and

Mobile Application (App) Behavior for Online

Recommendations. Journal of Organizational and End

User Computing (JOEUC), 33(6)(DOI:

10.4018/JOEUC.20211101.oa2), 26-34.

Maguire, C. M. (2011). Guidelines on Website Design and

Colour Selection for International Acceptance .

Marcus, A., Ed.,Design, User Experience, and

Usability, Springer Berlin/Heidelberg, 162-171.

Mohan, L., Mathur, N., & Reddy, R. (2015). Mobile App

Usability Index (MAUI) for Improving Mobile

Banking Adoption. Proceedings of the 10th

International Conference on Evaluation of Novel

Approaches to Software Engineering (ENASE-

2015)(ISBN: 978-989-758-100-7), 313-320.

Neilsen, J. (1993). Usability Engineering. New York:

Morgan Kauffmann.

Shneiderman , B., & Plaisant, C. (2017). Designing the

User Interface: Strategies for Effective Human-

Computer Interaction 6th edition. University of

Maryland: Human Computer InteractionLab.

iCAST-ES 2022 - International Conference on Applied Science and Technology on Engineering Science

842