Conclusion Validity - Low Statistical Power: To try
mitigating this threat, some statistical methods were
adopted, such as the Shapiro-Wilk normality test, the
paired T-test as a hypothesis test for dependent sam-
ples, and the F-Measure for qualitative analysis of the
models produced. Reliability of Measurements: To
mitigate this threat, it was adopted objective measure-
ments that did not depend on subjective judgment (ef-
fort, measured in time spent, and F-Measure). On the
other hand, the metrics used for the qualitative evalu-
ation still served as a complementary input in the dis-
cussion of the results obtained. Experimental Envi-
ronment: In order to mitigate this possible threat, we
instructed subjects that conversations could not take
place during the entire activities execution, or leave
the environment or access electronic devices.
6 CONCLUSION
This study presented a controlled experiment evalu-
ating ERText, a proposed textual DSL for database
conceptual modeling. ERText is compared with the
brModelo, a graphical DSL well-known in Software
Engineering ER lectures.
From the analysis it is possible to highlight the fol-
lowing aspects: (i) Effort: the graphical approach to
ER modeling requires less effort to perform the eval-
uated tasks. However, we considered that this differ-
ence is small and it can be reduced with future im-
provements in the proposed DSL. (ii) Effectiveness:
the computed average difference states that there is
no differences between the approaches, i.e., one ap-
proach is not better than the other. However, we ob-
served that there is a need to carry out tests involv-
ing problems of greater complexities for better as-
sessment. (iii) Qualitative comparison between treat-
ments: We observed a certain balance between treat-
ments, but with a positive evaluation for ERtext re-
garding the “Productivity” attribute. Because it was
the first time that the subjects had contact with our
grammar, and also considering a first release of our
DSL, we conclude that ERText is on the rails for
achieving better productivity indexes.
We also collected qualitative feedback from par-
ticipants. As a result, there are some improvements
regarding the language design that need to be revised,
in particular to the cardinalities and ternary relation-
ships. From the experimental results we conclude
that there is feasibility and good perspectives for the
motivated context, i.e., as a tool for teaching entity-
relationship modeling with the differential of adopt-
ing a textual approach for conceptual database mod-
eling in classrooms instead of a graphical notation.
This conclusion is sustained by the results, which did
not obtained expressive differences with regard to the
evaluated quality attributes between the tools.
ACKNOWLEDGEMENTS
This study was partially funded by PROPESQ
through AGP, and by FAPERGS, through the ARD
project N
o
19/2551-0001268-3.
REFERENCES
Al-Dmour, A. (2010). A cognitive apprenticeship based ap-
proach to teaching relational database analysis and de-
sign. Inf. & Comp. Science, 7(12):2495–2502.
Brambilla, M., Cabot, J., and Wimmer, M. (2017). Model-
Driven Software Engineering in Practice, Second Edi-
tion. Synthesis Lectures on Software Engineering.
Morgan & Claypool, San Rafael, CA, USA.
Celikovic, M., Dimitrieski, V., Aleksic, S., Ristic, S., and
Lukovic, I. (2014). A DSL for EER data model
specification. In 23rd Int. Conf. on Information Sys-
tems Development, pages 290–297, Varaždin, Croatia.
Springer.
Cook, T. D. and Campbell, D. T. (1979). Quasi-
Experimentation: Design and Analysis Issues for
Field Settings. Houghton Mifflin, Chicago, IL, USA.
Davis, F. D. (1989). Perceived Usefulness, Perceived Ease
of Use, and User Acceptance of Information Technol-
ogy. Management Inf. Systems Quarterly, 13(3):319–
340.
Derczynski, L. (2016). Complementarity, f-score, and NLP
evaluation. In 10th Int. Conf. on Language Resources
and Evaluation, pages 261–266. ELRA.
Dimitrieski, V.,
ˇ
Celikovi
´
c, M., Aleksi
´
c, S., Risti
´
c, S.,
Alargt, A., and Lukovi
´
c, I. (2015). Concepts and eval-
uation of the extended entity-relationship approach
to database design in a multi-paradigm information
system modeling tool. Comput. Lang. Syst. Struct.,
44:299–318.
Elmasri, R. and Navathe, S. (2011). Sistemas de Banco de
Dados. Pearson Universidades, Franca, SP, Brasil.
Kelly, S. and Tolvanen, J.-P. (2008). Domain Specific Mod-
eling: Enabling Full Code Generation. John Wiley &
Sons.
Kessler, J., Tschuggnall, M., and Specht, G. (2019). Relax:
A webbased execution and learning tool for relational
algebra. In Datenbanksysteme für Business, Technolo-
gie und Web, pages 503–506. Gesellschaft für Infor-
matik.
Obeo and TypeFox (2020). Xtext/sirius - integration the
main use-cases. Technical report, Obeo and TypeFox.
Wohlin, C., Runeson, P., Hst, M., Ohlsson, M. C., Regnell,
B., and Wessln, A. (2012). Experimentation in Soft-
ware Engineering. Springer, London, England.
Empirical Evaluation of a Textual Approach to Database Design in a Modeling Tool
215