Context: Usability is a major factor in the acceptance of digital health (DH) solutions. Problem: Despite its importance, usability experts have expressed concerns about the insufficient attention given to usability evaluation in practice, indicating potential efficiency problems of common evaluation methods in the healthcare domain. Objectives: This research paper aimed to analyse industrial usability evaluation practices in digital health to identify potential threats to the efficiency of their application. Method: To this end, we conducted an online survey of 144 usability experts experienced in usability evaluations for digital health applications. The survey questions aimed to explore the prevalence of techniques applied, and the participants’ familiarity and perceptions regarding tools and techniques. Results: The prevalently applied techniques might impose efficiency problems in common scenarios in digital health. Participant recruitment is considered timeconsuming and selecting the most appropriate evaluation method for a given context is perceived difficult. The results highlight a lack of utilisation of tools automating aspects of usability evaluation. Conclusions: A more widespread adoption of tools for automating usability evaluation activities seems desirable as well as guidelines for selecting evaluation techniques in a given context. We furthermore recommend to explore AI-based solutions to address the problem of involving targeted user groups that are difficult to access for usability evaluations.

Context: Digital Health (DH) is widely considered essential for sustainable future healthcare systems. Software quality, particularly usability, is crucial for the success and adoption of most DH products. However, concerns about the effectiveness and efficiency of usability evaluation of DH products have been raised. Objective: This article aims to analyse the prevalence and application contexts of usability evaluation methods in DH and to highlight potential issues related to their effectiveness and efficiency. Method: A systematic literature review of usability evaluation studies, published by (academic) practitioners between 2016 and April 2023, was conducted. 610 primary articles were identified and analysed, utilising five major scientific databases. Results: Our findings show a preference for inquiry (85%) and testing (63%) methods, with inspection used less frequently (17%). The published studies employed methods like questionnaires (75%); notably the SUS (49%), semi-structured interviews (25%), and heuristic evaluations (73%), with percentages based on their group. Data collection mainly involved the use of participant feedback (45%), audio/video recordings (44%), and system logs (20%), with both qualitative and quantitative data analyses prevalent in studies. However, several usability characteristics such as accessibility, memorability, and operability were found to be largely overlooked, and automation tools or platforms were not widely used. Among the systems evaluated were mHealth applications (70%), telehealth platforms (36%), health information technology (HIT) solutions (29%), personalized medicine (Per. Med.) (17%), wearable devices (12%), and digital therapeutics (DTx) interventions (6%), with the participation of general users, patients, healthcare providers, and informal caregivers varying based on the health condition studied. Furthermore, insights and experiences gathered from 24 articles underscored the importance of a mixed-method approach in usability evaluations, the limitations of traditional methods, the necessity for sector-specific customisation, and the potential benefits of remote usability studies. Moreover, while eye-tracking emerged as a promising evaluation technique, careful execution and interpretation are crucial to avoid data misinterpretation. Conclusion: The study’s findings showed that employing a combination of inquiry and testing-based methods is prevalent for evaluating DH platforms. Despite an array of DH systems, method distribution remained consistent across platforms and targeted user groups. The study also underlines the importance of involving target user groups in the process. Potentially affected cognitive abilities of participants and potential user groups of interest have to be taken into account when choosing evaluation methods, and methods might therefore need to be tailored. Complementary inspection methods might be particularly useful when recruiting representative participants is difficult. Several potential paths for future research are outlined, such as exploring novel technologies like artificial intelligence, for improved automation tool support in the usability evaluation process. 

Effective usability evaluation of user interface (UI) designs is essential. Particularly in digital healthcare, frequently involving relevant user groups in usability evaluations is not always possible or is ethically questionable. On the other hand, neglecting the perspectives of such groups can lead to UI designs that fail to be inclusive and adaptable. In this paper, we outline an initial idea to utilize artificial intelligence methods to simulate mobile user interface interactions of such user groups. The goal is to support software developers and designers with tools that show them how users of certain user groups might interact with a user interface under development and show potential issues before actual, more expensive usability evaluations are conducted. We present a study that employs synthetic representations of user interactions with UI elements based on a small sample of real interactions. This synthetic data was then used to train a classification model predicting whether real user interactions were from younger or elderly persons. The good performance of this model provides evidence that synthetic user interface interactions might be accurate enough to feed into imitation learning approaches, which, in turn, could be the foundation for the desired tool support.

Reflexion Modelling is a successful method used in industry for Software Architectural Consistency Checking (SACC). However, it includes a mapping step that is manual and tedious, especially for large complex systems. Various studies have shown how to successfully automate the interactive mapping of the basic units of the software’s codebase, i.e. its classes, to its architecture modules. However, their inherent drawback is that the effort required by an architect to review the mapping recommendations produced, whether during the mapping occurs or at the end of mapping, can be considerable. Subsequent studies have attempted to reduce this effort by use of a hierarchical mapping approach. These studies have demonstrated a reduction in the review effort required by a software architect; however, the gain in effort reduction occurred at the price of a lower recall and precision than similar non-hierarchical mapping approaches. In this study, we present an integrated approach of automated code-to-architecture mapping that draws from hierarchical (package mapping) and non-hierarchical (class mapping) techniques to keep effort minimal for an architect with marginal loss in recall and precision. Using the harmonic mean of f1-scores and effort reduction, our results show that with our integrated approach, we could achieve 0.90 on average, compared to 0.87 for the other two methods. 

InMap is an interactive and iterative information retrieval-based automated mapping algorithm that produces code-to-architecture mapping recommendations. In its original form, InMap requires an architect to provide feedback for each code-to-architecture mapping recommendation in a given set produced (complete feedback). However, architects may delay/defer deciding on some of the mapping recommendations provided. This leads us to ask, how would InMap perform if only a subset of the recommendations provided (partial feedback) or only a single recommendation (real-time feedback) is reviewed by the architect Through carefully designed mapping experiments, we show that an architect giving partial or real-time feedback does not harm the recall and precision of the recommendations produced by InMap. On the contrary, we observed from the results of the systems tested a net increase of 2-5% (depending on the approach). This shows that in addition to InMap’s original complete feedback approach, the two new approaches of collecting feedback presented in this paper, i.e. partial and real-time, create flexibility in how software architecture consistency checking tool developers may choose to collect mapping feedback and how architects may opt-to provide feedback, with no harm to the recall and precision of the results. 

Software architecture consistency checking (SACC) is a popular method to detect architecture degradation. Most SACC techniques require software engineers to manually map a subset of entities of a system’s implementation onto elements of its intended software architecture. Manually creating such a "seed mapping"for complex systems is a time-consuming activity.The objective of this paper is to investigate if creating seed mappings semi-automatically based on mapping recommendations for training automatic, machine learning-based mappers can reduce the effort for this task.To this end, we applied InMap, a highly accurate, interactive code-to-architecture mapping approach, to create seed mappings for five open source system with known architectures and mappings. Three different machine learning-based mappers were trained with these seed mappings and analysed regarding their predictive performance. We then compared the manual effort involved in using the combination of InMap and the most accurate automatic mapper and the manual effort of mapping the systems solely with InMap.The results suggest that InMap, with a minor adaption, can be used to seed an accurate mapper based on Naive Bayes. A full mapping with only InMap though turns out to involve slightly less manual effort on average; this is, however, not consistent across all systems. These results give reason to assume that more advanced ways of combining automatic mappers with InMap may further reduce that effort. 

Automating the mapping of a system’s code to its architecture is important in improving the adoption of successful Software Architecture Consistency Checking (SACC) methods like Reflexion Modelling. InMap is an interactive and iterative code-to-architecture mapping recommendation approach that achieves a rather decent recall and precision of 0.97 and 0.82 respectively, using minimal architecture documentation to apply natural language techniques to a software’s codebase. Nevertheless, InMap like most other automated recommendations techniques maps to architectural modules, low-level source code units like classes. For large complex systems, this can still hinder adoption due to the review effort required by a software architect when accepting or rejecting the recommendations. In this paper, we present a hierarchical package mapping technique that provides recommendations for higher-level source code units, i.e. packages. It utilizes InMap’s information retrieval capabilities to recommend mappings between the software’s packages and its architectural modules. We show that using our proposed technique we are able to reduce the recommendation review effort required by an architect, by 95% on average, for the six systems tested, and still achieve a code coverage of 75%. 

A mapping between a system's implementation and its software architecture is mandatory in many architecture consistency checking techniques. Creating such a mapping manually is a non-trivial task for most complex software systems. Machine learning-based text classification may be an highly effective tool for automating this task. How to make use of this tool most effectively has not been thoroughly investigated yet.

This article presents a comparative analysis of three classifiers applied to map the implementations of five open-source systems to their architectures. The performance of the classifiers is evaluated for different extraction and preprocessing settings as well as different training set sizes.

The results suggest that Logical Regression and Support Vector Machines both outperform Naive Bayes unless information about coarse-grained implementation structures cannot be exploited. Moreover, initial manual mappings of more than 15% of all source code files, or 10 files per module, do not seem to lead to a significantly better classification.

Context: High-quality software is essential to the progressing digitalisation of the Swedish healthcare sector. Developing software with the desired high quality is far from trivial due to the sophisticated requirements of the domain.

Problem: Studies on healthcare digitalisation challenges in Sweden and other countries, however, largely focus on the perceptions of healthcare professionals and patients and less on opinions of IT professionals.

Method: In this exploratory study, we conducted semi-structured interviews with nine IT professionals about observed challenges in developing software for the Swedish healthcare sector. A qualitative analysis was performed to identify common themes.

Results: We identified the prevalent challenges to be related to data integrity, privacy and security, rules and regulations, engineering usability, and software testing.

Conclusion: The results suggest that further research is required regarding agile methods, efficient requirement engineering, and testing in eHealth as well as in privacy and usability engineering.