Video game companies often have to choose among thousands of ideas to decide which ones to turn into video games. Despite the huge amount of money at stake, this process known as green-lighting in the game industry is largely a guesswork based on experts’ experience and intuitions. In this paper, through a case study with a prominent Swedish gaming company we identify the key factors influencing green-lighting decisions and develops a typology for evaluating game concepts more effectively. These factors can be used to build a data-driven model that integrates domain knowledge with AI/ML techniques, including natural language processing (NLP), to predict a game’s potential return on investment (ROI) during the funding request process. Further, the typology provides a structured framework for decision-makers, helping to reduce risk and ensure investments are aligned with market demand. 

Context: Ensuring smartphone interfaces are usable and accessible is essential for elderly users, particularly those with motor impairments, who face challenges with touchscreen interactions. Problem: Hand tremors and limited motor control can hinder touchscreen accuracy and efficiency. Meanwhile, recruiting elderly participants for usability studies can be challenging, often resulting in limited interaction data. Objectives: This study aimed to investigate elderly users’ smartphone interaction patterns, identify key challenges, and generate synthetic data to address data scarcity for usability research. Method: A custom-designed mobile app collected interaction data from 51 elderly participants performing tapping, dragging, and tracing tasks. Hand steadiness was assessed using accelerometer data. Gaussian Process Regression (GPR) and Long Short-Term Memory (LSTM) models were used to generate synthetic datasets replicating user interaction patterns. Results: Users with shaky hands struggled with precision tasks, especially involving smaller GUI elements, while larger elements improved performance. Continuous control was also found to be challenging in tracing tasks. Synthetic datasets successfully replicated spatial, temporal, and distributional metrics, demonstrating potential utility in future usability evaluation research. Conclusions: Inclusive GUI designs and adaptive features can improve accessibility for the elderly with limited motor control. Synthetic data can offer a potential solution for further usability evaluation research in building AI-driven design evaluation tools, reducing reliance on resource-intensive participant recruitment in earlier prototypes. Future work should examine diverse tasks and scenarios and involve people with severe motor impairments. 

Context: Usability is a major factor in the acceptance of digital health (DH) solutions. Problem: Despite its importance, usability experts have expressed concerns about the insufficient attention given to usability evaluation in practice, indicating potential efficiency problems of common evaluation methods in the healthcare domain. Objectives: This research paper aimed to analyse industrial usability evaluation practices in digital health to identify potential threats to the efficiency of their application. Method: To this end, we conducted an online survey of 144 usability experts experienced in usability evaluations for digital health applications. The survey questions aimed to explore the prevalence of techniques applied, and the participants’ familiarity and perceptions regarding tools and techniques. Results: The prevalently applied techniques might impose efficiency problems in common scenarios in digital health. Participant recruitment is considered timeconsuming and selecting the most appropriate evaluation method for a given context is perceived difficult. The results highlight a lack of utilisation of tools automating aspects of usability evaluation. Conclusions: A more widespread adoption of tools for automating usability evaluation activities seems desirable as well as guidelines for selecting evaluation techniques in a given context. We furthermore recommend to explore AI-based solutions to address the problem of involving targeted user groups that are difficult to access for usability evaluations.

Context: Digital Health (DH) is widely considered essential for sustainable future healthcare systems. Software quality, particularly usability, is crucial for the success and adoption of most DH products. However, concerns about the effectiveness and efficiency of usability evaluation of DH products have been raised. Objective: This article aims to analyse the prevalence and application contexts of usability evaluation methods in DH and to highlight potential issues related to their effectiveness and efficiency. Method: A systematic literature review of usability evaluation studies, published by (academic) practitioners between 2016 and April 2023, was conducted. 610 primary articles were identified and analysed, utilising five major scientific databases. Results: Our findings show a preference for inquiry (85%) and testing (63%) methods, with inspection used less frequently (17%). The published studies employed methods like questionnaires (75%); notably the SUS (49%), semi-structured interviews (25%), and heuristic evaluations (73%), with percentages based on their group. Data collection mainly involved the use of participant feedback (45%), audio/video recordings (44%), and system logs (20%), with both qualitative and quantitative data analyses prevalent in studies. However, several usability characteristics such as accessibility, memorability, and operability were found to be largely overlooked, and automation tools or platforms were not widely used. Among the systems evaluated were mHealth applications (70%), telehealth platforms (36%), health information technology (HIT) solutions (29%), personalized medicine (Per. Med.) (17%), wearable devices (12%), and digital therapeutics (DTx) interventions (6%), with the participation of general users, patients, healthcare providers, and informal caregivers varying based on the health condition studied. Furthermore, insights and experiences gathered from 24 articles underscored the importance of a mixed-method approach in usability evaluations, the limitations of traditional methods, the necessity for sector-specific customisation, and the potential benefits of remote usability studies. Moreover, while eye-tracking emerged as a promising evaluation technique, careful execution and interpretation are crucial to avoid data misinterpretation. Conclusion: The study’s findings showed that employing a combination of inquiry and testing-based methods is prevalent for evaluating DH platforms. Despite an array of DH systems, method distribution remained consistent across platforms and targeted user groups. The study also underlines the importance of involving target user groups in the process. Potentially affected cognitive abilities of participants and potential user groups of interest have to be taken into account when choosing evaluation methods, and methods might therefore need to be tailored. Complementary inspection methods might be particularly useful when recruiting representative participants is difficult. Several potential paths for future research are outlined, such as exploring novel technologies like artificial intelligence, for improved automation tool support in the usability evaluation process. 

Effective usability evaluation of user interface (UI) designs is essential. Particularly in digital healthcare, frequently involving relevant user groups in usability evaluations is not always possible or is ethically questionable. On the other hand, neglecting the perspectives of such groups can lead to UI designs that fail to be inclusive and adaptable. In this paper, we outline an initial idea to utilize artificial intelligence methods to simulate mobile user interface interactions of such user groups. The goal is to support software developers and designers with tools that show them how users of certain user groups might interact with a user interface under development and show potential issues before actual, more expensive usability evaluations are conducted. We present a study that employs synthetic representations of user interactions with UI elements based on a small sample of real interactions. This synthetic data was then used to train a classification model predicting whether real user interactions were from younger or elderly persons. The good performance of this model provides evidence that synthetic user interface interactions might be accurate enough to feed into imitation learning approaches, which, in turn, could be the foundation for the desired tool support.

Reflexion Modelling is a successful method used in industry for Software Architectural Consistency Checking (SACC). However, it includes a mapping step that is manual and tedious, especially for large complex systems. Various studies have shown how to successfully automate the interactive mapping of the basic units of the software’s codebase, i.e. its classes, to its architecture modules. However, their inherent drawback is that the effort required by an architect to review the mapping recommendations produced, whether during the mapping occurs or at the end of mapping, can be considerable. Subsequent studies have attempted to reduce this effort by use of a hierarchical mapping approach. These studies have demonstrated a reduction in the review effort required by a software architect; however, the gain in effort reduction occurred at the price of a lower recall and precision than similar non-hierarchical mapping approaches. In this study, we present an integrated approach of automated code-to-architecture mapping that draws from hierarchical (package mapping) and non-hierarchical (class mapping) techniques to keep effort minimal for an architect with marginal loss in recall and precision. Using the harmonic mean of f1-scores and effort reduction, our results show that with our integrated approach, we could achieve 0.90 on average, compared to 0.87 for the other two methods. 

Software architecture consistency checking (SACC) is a popular method to detect architecture degradation. Most SACC techniques require software engineers to manually map a subset of entities of a system's implementation onto elements of its intended software architecture. Manually creating such a "seed mapping" for complex systems is a time-consuming activity.The objective of this paper is to investigate if creating seed mappings semi-automatically based on mapping recommendations for training automatic, machine learning-based mappers can reduce the effort for this task.To this end, we applied InMap, a highly accurate, interactive code-to-architecture mapping approach, to create seed mappings for five opensource systems with known architectures and mappings. Three different machine learning-based mappers were trained with these seed mappings and analysed regarding their predictive performance. We then compared the manual effort involved in using the combination of InMap and the most accurate automatic mapper and the manual effort of mapping the systems solely with InMap.The results suggest that InMap, with a minor adaption, can be used to seed an accurate mapper at a size of 20--25% based on Naive Bayes. A full mapping with only InMap though turns out to involve slightly less manual effort on average; this is, however, not consistent across all systems. These results give evidence that more advanced ways of combining automatic mappers with InMap may further reduce that effort.

InMap is an interactive and iterative information retrieval-based automated mapping algorithm that produces code-to-architecture mapping recommendations. In its original form, InMap requires an architect to provide feedback for each code-to-architecture mapping recommendation in a given set produced (complete feedback). However, architects may delay/defer deciding on some of the mapping recommendations provided. This leads us to ask, how would InMap perform if only a subset of the recommendations provided (partial feedback) or only a single recommendation (real-time feedback) is reviewed by the architect Through carefully designed mapping experiments, we show that an architect giving partial or real-time feedback does not harm the recall and precision of the recommendations produced by InMap. On the contrary, we observed from the results of the systems tested a net increase of 2-5% (depending on the approach). This shows that in addition to InMap’s original complete feedback approach, the two new approaches of collecting feedback presented in this paper, i.e. partial and real-time, create flexibility in how software architecture consistency checking tool developers may choose to collect mapping feedback and how architects may opt-to provide feedback, with no harm to the recall and precision of the results. 

Software architecture consistency checking (SACC) is a popular method to detect architecture degradation. Most SACC techniques require software engineers to manually map a subset of entities of a system’s implementation onto elements of its intended software architecture. Manually creating such a "seed mapping"for complex systems is a time-consuming activity.The objective of this paper is to investigate if creating seed mappings semi-automatically based on mapping recommendations for training automatic, machine learning-based mappers can reduce the effort for this task.To this end, we applied InMap, a highly accurate, interactive code-to-architecture mapping approach, to create seed mappings for five open source system with known architectures and mappings. Three different machine learning-based mappers were trained with these seed mappings and analysed regarding their predictive performance. We then compared the manual effort involved in using the combination of InMap and the most accurate automatic mapper and the manual effort of mapping the systems solely with InMap.The results suggest that InMap, with a minor adaption, can be used to seed an accurate mapper based on Naive Bayes. A full mapping with only InMap though turns out to involve slightly less manual effort on average; this is, however, not consistent across all systems. These results give reason to assume that more advanced ways of combining automatic mappers with InMap may further reduce that effort.