Shared decision-making (SDM), the collaborative process involving active participation from both patients and providers in health care treatment decisions, reflects an important paradigm shift in medicine toward patient-centered care [1,2]. SDM facilitates information exchange and discussion of treatment options that involve the best scientific evidence and consider patient preferences [3,4]. The readiness for using SDM may be enhanced through its accessibility to individuals with limited health literacy or those with disabilities [5]. In the context of rehabilitation, SDM typically occurs during goal setting by selecting and agreeing upon behavioral objectives that patients, caregivers, and the rehabilitation team work together to achieve [6]. The development of mutual trust, 2-way communication, and sharing of power are conditions that influence patients’ capacity and confidence to participate in SDM in musculoskeletal physiotherapy [7] and in the treatment of depression [8]. As a result, SDM assists patients in making individualized care decisions, and health care providers can feel confident in the presented and prescribed options [3,4]. SDM is important to increase satisfaction with care among both patients and providers, may improve individuals’ quality of life and clinical outcomes, and fosters a better patient-provider relationship [9]. Furthermore, SDM encourages patient participation in their rehabilitation, supporting self-efficacy, empowerment, and ownership over the decisions [6].

Despite the listed benefits, it has been difficult to implement SDM in clinical practice because of barriers such as time constraints, accessibility to information and effective SDM tools, and limited technical and organizational resources [3]. It has been reported that only 10% of face-to-face clinical consultations involve SDM [10,11]. Advances in digital health technologies (eHealth) have resulted in tools that can bridge this SDM gap by allowing increased access to shared information and support for patient-provider communication [12]. Accessible, cost-effective, web-based decision-making is supported by use across various platforms such as the internet, tablets, or smartphone apps [13,14]. Such SDM technologies include patient decision aids that clarify options and values for personalized decision support, leading to reduced decisional conflict and increased participation in treatment choices that are consistent with the patient’s values [13]. Patient portals reflect another technology that can support SDM, providing patients with secure access to their health information profile and communication with their care provider [15-17].

Although studies have been conducted to introduce and investigate the acceptance of rehabilitation technologies, research into the usability of technology systems is limited [18,19]. A technology system in rehabilitation is defined as an environmental factor that incorporates aspects of the physical and social environments that may affect communicative participation [20]. Technology systems need to be evaluated in terms of their usability to maximize their acceptance and benefits. The International Organization for Standardization (ISO) 9241 defined usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [21]. Evaluation of usability is key to guiding the development of efficient and effective technologies that end users will readily adopt by providing information about how a user uses the technology system and the challenges they find while interacting with a system’s interface [22]. Different usability models have been proposed for evaluating software usability. Gupta et al [23] proposed a comprehensive hierarchal usability model with a detailed taxonomy, including 7 usability parameters: efficiency, effectiveness, satisfaction, memorability, security, universality, and productivity. Evaluating these usability parameters throughout the design process can allow for continuous improvement of ease of use and can predict the user’s acceptance or rejection of the product [24]. Therefore, including input from individuals who will use the technology (in the case of SDM technologies, clinicians, patients, and caregivers) through usability testing is a necessary component in designing relevant, understandable, and usable technologies.

The field of rehabilitation science is defined as a multidimensional person-centered process targeting body functions, activities and participation, and the interaction with the environment aiming at optimizing functioning among persons with health conditions experiencing disability [25]. It has seen a recent rise in the development and implementation of technologies aimed at supporting SDM between clinicians, patients, and their caregivers [26]. However, it is unclear how user input or usability testing is integrated into the design process of these rehabilitation health technologies, including how usability is conceptualized, what measures are used, and at what stage of design usability is evaluated. To date, few studies, and no systematic or scoping reviews that we are aware of, have addressed how usability is measured among rehabilitation technologies supporting SDM. Given the complexity of usability, a thorough examination of the methodologies used to measure usability in this context is required to comprehensively map what has been done and inform future research efforts. A greater understanding of how the parameters of usability are measured will guide future usability testing to inform further development of SDM technologies designed to enhance patient-centered care in rehabilitation. Therefore, this scoping review was conducted to provide knowledge about the methods and measures used to determine the usability of rehabilitation technologies aimed at supporting SDM at different phases of technology development and implementation.

This scoping review aimed to answer the following research questions: (1) which methods and measures have been used to produce knowledge about the usability of rehabilitation technologies aimed at supporting SDM at the different phases of development and implementation? (2) Which parameters of usability have been measured and reported in studies focusing on rehabilitation technologies aimed at supporting SDM?

The eligibility criteria for this scoping review are outlined in Textbox 1.

The search strategy was developed in collaboration with a health science librarian. As health system issues often change with models of care delivery, the economic climate, and the environment [29], we decided to narrow the scope of the search (2005 to 2020). The following electronic databases were searched in both English and French: Ovid MEDLINE, Embase, CINAHL, and PsycINFO. A combination of Medical Subject Heading terms, subject headings, and keywords was used and covered five concepts: (1) usability OR user* friendl* OR eas* to use OR useful* OR user* perspective* OR patient* perspective* OR client* perspective* OR user* experience* AND (2) rehabilitation OR telerehabilitation OR tele rehabilitation OR disabled OR disabilit* OR physical limitation* OR mental limitation* OR psycho* limitation* OR adaptation* OR mobility OR occupational therap* OR physiotherap* OR physical therap* OR speech languag* pathol* OR speech therap* OR language therap* OR communication disorder* AND (3) think* aloud OR focus group* OR interview* OR Wizard* OR Empathy map* OR Persona* OR Questionnaire* OR instrument* OR scale* OR tool OR tools OR measurement* OR survey* OR drama OR deliberation* OR evaluation* OR assessment* OR video confrontation* OR photo voice* AND (4) technolog* OR gerontotechnolog* OR smart* OR intelligen* OR ambient assisted living OR virtual reality OR virtual rehabilitation OR telemonitoring OR telehealth OR telemedicine OR telerehabilitation OR ehealth OR tele monitoring OR tele health OR tele medicine OR tele rehabilitation OR e health or sensor* OR biosensor* OR mobile app* OR product* OR internet OR web OR computer* OR software* OR device* OR self-help OR wheelchair* OR wheelchair* OR communication aid* AND (5) shared decision making OR Decision-Making OR patient-provider communication OR decision aid OR decision support. This was followed by hand searches of the reference lists of the included studies (the search strategy for Ovid MEDLINE is presented in Multimedia Appendix 1).

All identified studies were uploaded into EndNote X9.1 (Clarivate Analytics), and duplicates were removed. In total, 2 independent reviewers conducted the selection of abstracts starting with a pilot phase involving the examination of the first 10 titles and abstracts to screen and decide on retention of the abstract based on the inclusion criteria. Interrater agreements were assessed using the κ statistic [30]. Interrater agreement of <75% resulted in a clarification of the eligibility criteria and a revision if needed. The process was repeated twice between the reviewers until an agreement of 75% was reached, which is evidence of excellent agreement [30]. Finally, all eligible studies and those classified as unclear (ie, requiring further information to make a final decision regarding their retention) were independently reviewed as full-text articles. Disagreements at this stage were resolved through consensus. The PRISMA-ScR flow diagram [28] was used to guide the selection process.

In total, 2 reviewers independently extracted data from the included articles to avoid missing relevant information. The data extracted included information corresponding to study design, rehabilitation technology intervention used (ie, setting, content, and detail of the type of user interface), population studied (participant demographics and target conditions), characteristics of the measures, and the development stage.

Descriptive statistics were used to describe the characteristics of the included studies, study design, characteristics of the study population, and geographical location. Findings were categorized based on study designs, parameters of usability, types of technologies, stage of development of the technology, and usability evaluation methodologies.

Types of SDM technologies and usability evaluations were mapped to parameters of usability based on a comprehensive hierarchal usability model presented by Gupta et al [23]. The usability parameters include efficiency, defined as “enables user to produce desired results with respect to investment of resources”; effectiveness, defined as “a measure of software product with which user can accomplish specified tasks and desired results with completeness and certainty”; satisfaction, defined as “a measure of responses, feelings of user when users are using the software i.e., freedom from discomfort, likeability”; memorability, defined as “the property of software product that enables the user to remember the elements and the functionality of the system product”; security, defined as “the degree to which risks and damages to people or other resources i.e. hardware and software can be avoided”; universality, defined as “the accommodation of different cultural backgrounds of diverse users with software product and practical utility of software product”; and productivity, defined as “the amount of useful output with the software product” [28] (Textbox 2).

The usability evaluation methodologies were mapped based on the framework by Jacobsen [31]. The categories of the usability evaluation methods included (1) empirical methods, based on users’ experience with the technology in a systematic way; (2) inspection methods, conducted by experts who examined usability-related aspects of a user interface without involving any users; and (3) inquiry methods, based on the information about users’ needs, likes, and understanding of the technology through interviews or focus groups, observation, and verbal or written questions [31].

This scoping review is part of an initiative (Réseau provincial de recherche en adaptation-réadaptation–RS6 Technologies de readaptation [Quebec Rehabilitation Research Network]; [6]) to create an interactive directory of methodological tools for measures of the usability of rehabilitation technologies. Stakeholder consultations with members of the Réseau provincial de recherche en adaptation-réadaptation–RS6 group were held at the beginning of the process (requesting feedback to refine the research question for data extraction and synthesis), during the study (validating the data extraction and deciding on the best way to align the information with stakeholders’ needs), and when the final results were available (knowledge mobilization).

A total of 430 studies were identified from electronic searches, and a total of 19 were identified through hand sorting reference lists. We excluded 57.2% (257/449) of the studies at the title and abstract stage, resulting in 192 full-text articles. Of these 192 studies, 154 (80.2%) were excluded at the full-text stage, resulting in 38 (19.8%) studies. The search strategy was updated in November 2020 and followed the PRISMA-ScR flowchart of the selection process. Reasons for exclusion of studies are provided in Figure 1. Interrater agreement reached ≥75%, which is evidence of excellent agreement. Disagreements were resolved through consensus.

The characteristics of the included studies are presented in Multimedia Appendix 2 [32-69]. Overall, the 38 included studies were published between 2008 and 2020 as peer-reviewed studies. Studies were published in the United States (17/38, 43%), Europe (14/38, 37%), Canada (5/38, 13%), and Asia (2/38, 5%). The study designs of the included studies were mixed methods (20/38, 53%), qualitative (12/38, 31%), and quantitative (6/38, 16%).

Multimedia Appendix 2 presents the characteristics of the included participants. The number of participants across all the included studies was 2138, with age ranging between 18 and 86 years. Participants of usability evaluations included patients (38/38, 100%); clinicians (32/38, 84%); caregivers or family (12/38, 32%); and others (6/38, 16%), including case managers, drug advisory committees, computer scientists, behavioral scientists, communication scientists, clinical administrators, service providers, and social service providers. The target end users of the developed SDM technologies were mainly patients and clinicians (24/38, 63%). The recruitment methods and settings varied across the included studies, including hospitals (24/38, 63%), the community (10/38, 26%), and universities (4/38, 11%).

Table 1 presents usability definitions and parameters provided by the authors across the included studies. Notably, only 50% (19/38) of the included studies provided an a priori definition of usability or listed parameters of usability. Usability parameters were categorized as effectiveness (9/38, 23%), efficiency (8/38, 21%), memorability (11/38, 29%), satisfaction (14/38, 37%), security (5/38, 13%), universality (4/38, 10%), and productivity (10/38, 26%) based on Gupta et al [23].

Table 2 presents the type of SDM technologies that were used across the included studies. Technologies for SDM included clinical decision support systems (9/33, 27%), mobile health apps (9/33, 27%), and web-based aids (15/33, 46%). The SDM context was mainly between clinicians and patients (32/36, 89%). The types of technology for SDM were mapped to usability parameters, including effectiveness (10/38, 26%), efficiency (11/38, 29%), memorability (20/38, 53%), satisfaction (27/38, 71%), security (5/38, 13%), universality (4/38, 10%), and productivity (16/38, 42%) based on Gupta et al [23]. The most common SDM technologies evaluated for usability were web-based aids. Satisfaction was the most common usability parameter mapped to types of SDM technologies.

The usability evaluation methods were categorized, based on the framework by Jacobsen [31], into (1) empirical (think-aloud protocol, 14/38, 36%; user tracking, 3/38, 8%; performance measures, 4/38, 10%; field test, 2/38, 5%; video recording, 1/38, 2%; and screen capture, 2/38, 5%), (2) inspection (cognitive walk-through, 1/38, 2% and Near live clinical situation, 1/38, 2%), and (3) inquiry (focus groups, 3/38, 8%; workshops, 2/38, 5%; semistructured interviews, 16/38, 42%; structured interviews, 1/38, 2%; questionnaires, 24/38, 63%; observations, 5/38, 13%; and comments, 3/38, 8%; Table 3). An important point to emphasize is the frequency with which researchers used 1 (13/38, 34%), 2 (15/38, 39%), 3 (7/38, 18%), 4 (2/38, 5%), and 6 (1/38, 2%) methods from the framework by Jacobsen [31], presented in Figure 2 [32-69]. Most (28/38, 73%) used 1 or 2 methods of evaluation. Usability was assessed during development (18/38, 47%), preimplementation (13/38, 34%), or implementation (7/38, 18%) through a variety of measures, including usability questionnaires (15/38, 39%), tailored tools developed by the authors (17/38, 45%), and acceptance and satisfaction questionnaires (6/38, 16%). The usability evaluation parameters identified by the authors were mapped to the usability parameters explained by Gupta et al [23], including effectiveness (13/38, 34%), efficiency (12/38, 31%), memorability (13/38, 34%), productivity (2/38, 5%), security (2/38, 5%), and satisfaction (32/38, 84% Figure 3 and Table 4).

The frameworks and theoretical models reported by the authors during the development, implementation, and evaluation of the technologies to support SDM reflected 5 categories: technology design (15/38, 39%), behavior change (21/38, 21%), analysis (9/38, 24%), SDM framework (8/38, 21%), and not reported (9/38, 24%; Figure 4). Notably, 24% (9/38) of the studies did not report using a framework or model during any stage of their research. Authors most commonly reported using a model or framework as a foundation to inform the design of their respective SDM technologies. User-centered design (9/15, 60%) was the most frequently used technology design framework.

This scoping review was conducted to provide knowledge about how usability is evaluated when developing or implementing rehabilitation technologies aimed at supporting SDM. The first research question examined the methods and measures used in the context of SDM at different phases of technology development and implementation. Our findings revealed 14 reported methods that can help in evaluating the overall functionalities of the system and whether it fulfills the users’ requirements [75] and can be effective for identifying issues with a system [76]. The most frequent reported methods included think-aloud protocols (14/38, 36%), semistructured interviews (16/38, 42%), and questionnaires (24/38, 63%; Table 3). There was a total of 30 usability measures reported (Table 4), with the System Usability Scale being the most frequently used among the included studies. We operationalized the different types of methods used through the model by Jacobsen [31], reflecting empirical methods (based on users’ experience with the technology in a systematic way), inspection methods (conducted by experts who examine usability-related aspects of a user interface without involving any users), and inquiry methods (based on the information about users’ needs, likes, and understanding of the technology through interviews or focus groups, observation, or comments). Notably, the reported methods were predominantly classified as inquiry and empirical (Figure 2).

The second research question examined the parameters of usability that were measured and reported. We found that the methods used to evaluate different parameters of usability varied according to the a priori framing of usability, demonstrated by the variations in the definitions of usability described by the authors (Table 1). There was an evolution in the definition of usability across the included studies, with more recent studies (published since 2016) using the unified definition proposed by the ISO [43,46,48,57,61,64,65,67]. The usability parameters of the definitions were categorized based on the proposed comprehensive hierarchal model by Gupta et al [23] as effectiveness (9/38, 23%), efficiency (8/38, 21%), memorability (11/38, 29%), satisfaction (14/38, 37%), security (5/38, 13%), universality (4/38, 10%), and productivity (10/38, 26%). These are consistent with the 3 constructs of the ISO standards, which are effectiveness, efficiency, and satisfaction, and allows for a more detailed categorization of usability parameters.

Although the ISO standards [21] and the usability model by Gupta et al [23] provide dimensions that could be considered as primary usability parameters, there remain challenges with measuring usability that emerged in this review. On the surface, usability is a simple concept. In fact, simplicity is at the heart of usability; however, measuring usability is not simple. Paradoxically, the ISO definition of usability is complex. Usability is about the person’s experience; however, that experience is influenced by many aspects, such as a person’s behavior and social network and the complexity of the technological functionalities. Usability may be viewed as a feature of the technology or an emergent property of the interaction between the user, the system, and contextual factors. Evaluating usability through these lens leads to using inspection, empirical, or inquiry methods [31]. These can be applied at different stages of development of a technology (ie, in a developmental laboratory, in preimplementation, or during implementation), as described by the included studies (Table 2).

This review revealed that evaluating usability requires a comprehensive approach with several methods to cover multiple usability parameters. Most articles included in this review (36/38, 95%) focused on inquiry methods, relying heavily on questionnaires and semistructured interviews to evaluate usability, and the most frequent empirical method was think-aloud protocols (Figure 2). Although a comprehensive approach is suggested for accurate usability evaluation, this was largely not shown in the included articles. Rather, 73% (28/38) of the included studies only used 1 or 2 methods in total to evaluate usability. Only 2% (1/38) of the studies, conducted by Span et al [59], incorporated multiple methods that covered all 3 dimensions—inquiry, inspection, and empirical [31]. However, some of the included studies (2/38, 5%) described different usability evaluations for the same technology at different stages of development in separate articles (eg, “Take Charge, Get Cured” in the developmental [34] and preimplementation [51] stages). It is believed that the combination of inspection, empirical, and inquiry methods can provide more accurate and complete results in finding usability problems as there is no exact method considered to be the best for usability evaluation [77]. Matera et al [78] developed a systematic usability evaluation framework to address this challenge. They posited that usability can be reliably evaluated by systematically combining evaluation methods [78]. Recent reviews of usability not specific to SDM in software [79], mobile health [80], eHealth [81], user experience [82], and web development [83] mirrored the results of this review in that few studies used a combination of evaluation methods.

However, the lack of reported inspection methods demonstrated in this review may partially be explained by the inherent nature of SDM technologies for rehabilitation rather than a lack of comprehensive evaluation. Very few examples of inspection methods were demonstrated across the included studies, with only 2% (1/38) using cognitive walk-throughs and an additional 2% (1/38) using “near live” clinical situations. Critically, inspection methods refer to evaluations conducted by specific usability experts [31], not by the end users of the technology (eg, patients and clinicians). As the purpose of technology to support SDM in rehabilitation is to improve patient-centered care, the consideration of end users in the development—and, consequently, the usability evaluations—is crucial to ensure that the technology will be understood and adopted by the target population. Therefore, we propose that a comprehensive approach for evaluating the usability of rehabilitation technologies aimed at supporting SDM could focus on empirical and inquiry methods to prioritize the input of the patient and clinician end users.

Although questionnaires were found to be the most common method used overall, the identified measures of usability in the included studies demonstrated limitations in comprehensiveness, largely mapping to the parameters of satisfaction and memorability (Figure 3). The emphasis on the parameter of satisfaction (demonstrated in 32/38, 84% of measures) may reflect the importance of this parameter when developing technologies for SDM in rehabilitation (eg, the importance of evaluating the usefulness, user-friendliness, and ease of use). However, this may also reflect key missing areas in usability evaluation. Critically, the parameters of usability described by the authors in their a priori definitions of usability were not found to be consistent with the parameters of the measures that were used. Therefore, although individuals may be conceptualizing usability in a comprehensive manner, the measurement itself was not comprehensive. For example, there was a demonstrated lack of measurement of the parameters of effectiveness and efficiency, which were both described in the definition of usability in 34% (13/38) of the included studies, although both were only found to be used in 23% (9/38) of usability measures.

This review uncovered the need for inclusion of theoretical models or frameworks during various stages of SDM usability studies to guide which usability parameter to measure. Theoretical models and frameworks were infrequently reported (Figure 4). Most studies in this review (27/38, 71%) reported using 1 model or framework, whereas some (10/38, 26%) integrated 2. Only 2% (1/38) of the studies, carried out by Bauerle Bass et al [34], exhibited an in-depth application of models and frameworks as underpinnings to their research. The most common (9/38, 24%) and perhaps the most beneficial framework, user-centered design, served as the foundation for designing an SDM technology [21,36,42].

The importance of using theoretical models and frameworks during the development, implementation, and analysis of technologies and evaluation of usability is demonstrated through the implications of poor usability [18,84,85], which discourages users from using the technology systems. Moreover, if the technology systems are not user-friendly, then they can increase the problems experienced by users. Solutions to systems failing to meet the users’ needs include understanding user feedback [86], usability evaluations [75], involving users in the early stages of development [87], and including professionals such as providers [88]. There is a need for flexibility and for friendly, simple, and self-explanatory interfaces that allow users to interact with the system [89]. For the systems to be effective, it is important to assess a system that is easy to use on a daily basis. This would increase the ability of the patients to control their diseases and allow their daily lives to be more satisfying [76]. The technology systems need to be designed for a particular type of user and need to be easy to use to create acceptance. The usability of the technology system is vital as it has a high degree of influence over the success of the system. Thus, the system needs to be designed to provide a friendly environment for the user to develop a positive attitude toward using it and lead to its successful adoption.

It is envisioned that the involvement of end users in the development of SDM technologies will continue to grow and that more applications of existing technology, such as mobile phones, websites, or applications, will be used to benefit individuals with disabilities. We also anticipate that more companies may show an interest in this market, potentially promoting frequent use of SDM technologies in rehabilitation care. However, there are challenges in the development of SDM technologies, such as tailoring to individuals’ capabilities and properly addressing the emotional state of individuals with disabilities or cognitive impairments during everyday tasks. It will be critical to develop these technologies in a way that meets individual variations in needs and abilities of individuals with disabilities so that they really help maintain autonomy, provide meaningful activities, and promote decision-making [18,84,85].

An important area for this growing field will be how to effectively integrate end-user input throughout all stages of development of such SDM technologies, including effective usability testing. An additional challenge for the field of rehabilitation care in supporting SDM technologies would be in integrating the technology into the built environment, such as a client-server system, and into routine care [86]. There is a clear need for new methods of rapid SDM technology appraisal and evaluation to inform deployment to overcome the barriers that will be faced because of the expected further integration of SDM technologies within the built environment.

We did not assess the quality of the included articles, consistent with the scoping review methodology [27,90]. Therefore, we included studies with different designs and different quality levels, which allowed for a broad exploration of measures and methods used to evaluate the usability of SDM technologies. In our results, we focused mainly on general usability measures and did not report the psychometric properties and clinical utility of these measures. Future work needs to evaluate the psychometric properties and clinical utility of usability measures through a systematic review methodology with a quality assessment of the included articles. Another limitation was that we did not include gray literature as this scoping review aimed to examine the reported measures and methods used in peer-reviewed rehabilitation literature on SDM technologies. It could be an area of interest for future work to examine what methods and measures are used in gray literature.

The results of this scoping review highlight the importance as well as the complexity of usability evaluation. Although various methods and measures were shown to be used to evaluate the usability of technologies to support SDM in rehabilitation, very few evaluations used in the included studies adequately spanned the selected usability parameters. This review identified gaps in usability evaluation as most studies relied solely on questionnaires rather than a combination of inspection and empirical methods and most questionnaires simply focused on the usability parameter of satisfaction. We recommend for individuals to adopt a comprehensive approach to usability evaluation of SDM technologies, starting with a clear definition of how usability is conceptualized to guide the structure of the evaluation. In addition, we recommend the use of multiple usability evaluation methods categorized as inspection (eg, questionnaires, focus groups, and interviews) or empirical (eg, think-aloud protocols) to capture a more complete picture of end-user needs and interpretations. The selected methods should span a variety of parameters of usability, not just satisfaction (eg, effectiveness, efficiency, memorability, security, universality, and productivity). The consideration of end users (such as patients and clinicians) is of particular importance for the development of technologies to support SDM as the process of SDM itself aims to improve patient-centered care and integrate both patient and clinician voices into their rehabilitation care.