Background: The field of rehabilitation has seen a recent rise in technologies to support shared decision-making (SDM). Usability testing during the design process of SDM technologies is needed to optimize adoption and realize potential benefits. There is variability in how usability is defined and measured. Given the complexity of usability, a thorough examination of the methodologies used to measure usability to develop the SDM technologies used in rehabilitation care is needed.
Objective: This scoping review aims to answer the following research questions: which methods and measures have been used to produce knowledge about the usability of rehabilitation technologies aimed at supporting SDM at the different phases of development and implementation? Which parameters of usability have been measured and reported?
Methods: This review followed the Arksey and O’Malley framework. An electronic search was performed in the Ovid MEDLINE, Embase, CINAHL, and PsycINFO databases from January 2005 up to November 2020. In total, 2 independent reviewers screened all retrieved titles, abstracts, and full texts according to the inclusion criteria and extracted the data. The International Organization for Standardization framework was used to define the scope of usability (effectiveness, efficiency, and satisfaction). The characteristics of the studies were outlined in a descriptive summary. Findings were categorized based on usability parameters, technology interventions, and measures of usability.
Results: A total of 38 articles were included. The most common SDM technologies were web-based aids (15/33, 46%). The usability of SDM technologies was assessed during development, preimplementation, or implementation, using 14 different methods. The most frequent methods were questionnaires (24/38, 63%) and semistructured interviews (16/38, 42%). Satisfaction (27/38, 71%) was the most common usability parameter mapped to types of SDM technologies and usability evaluation methods. User-centered design (9/15, 60%) was the most frequently used technology design framework.
Conclusions: The results from this scoping review highlight the importance and the complexity of usability evaluation. Although various methods and measures were shown to be used to evaluate the usability of technologies to support SDM in rehabilitation, very few evaluations used in the included studies were found to adequately span the selected usability domains. This review identified gaps in usability evaluation, as most studies (24/38, 63%) relied solely on questionnaires rather than multiple methods, and most questionnaires simply focused on the usability parameter of satisfaction. The consideration of end users (such as patients and clinicians) is of particular importance for the development of technologies to support SDM, as the process of SDM itself aims to improve patient-centered care and integrate both patient and clinician voices into their rehabilitation care.
Shared decision-making (SDM), the collaborative process involving active participation from both patients and providers in health care treatment decisions, reflects an important paradigm shift in medicine toward patient-centered care [, ]. SDM facilitates information exchange and discussion of treatment options that involve the best scientific evidence and consider patient preferences [ , ]. The readiness for using SDM may be enhanced through its accessibility to individuals with limited health literacy or those with disabilities [ ]. In the context of rehabilitation, SDM typically occurs during goal setting by selecting and agreeing upon behavioral objectives that patients, caregivers, and the rehabilitation team work together to achieve [ ]. The development of mutual trust, 2-way communication, and sharing of power are conditions that influence patients’ capacity and confidence to participate in SDM in musculoskeletal physiotherapy [ ] and in the treatment of depression [ ]. As a result, SDM assists patients in making individualized care decisions, and health care providers can feel confident in the presented and prescribed options [ , ]. SDM is important to increase satisfaction with care among both patients and providers, may improve individuals’ quality of life and clinical outcomes, and fosters a better patient-provider relationship [ ]. Furthermore, SDM encourages patient participation in their rehabilitation, supporting self-efficacy, empowerment, and ownership over the decisions [ ].
Despite the listed benefits, it has been difficult to implement SDM in clinical practice because of barriers such as time constraints, accessibility to information and effective SDM tools, and limited technical and organizational resources . It has been reported that only 10% of face-to-face clinical consultations involve SDM [ , ]. Advances in digital health technologies (eHealth) have resulted in tools that can bridge this SDM gap by allowing increased access to shared information and support for patient-provider communication [ ]. Accessible, cost-effective, web-based decision-making is supported by use across various platforms such as the internet, tablets, or smartphone apps [ , ]. Such SDM technologies include patient decision aids that clarify options and values for personalized decision support, leading to reduced decisional conflict and increased participation in treatment choices that are consistent with the patient’s values [ ]. Patient portals reflect another technology that can support SDM, providing patients with secure access to their health information profile and communication with their care provider [ - ].
Although studies have been conducted to introduce and investigate the acceptance of rehabilitation technologies, research into the usability of technology systems is limited [, ]. A technology system in rehabilitation is defined as an environmental factor that incorporates aspects of the physical and social environments that may affect communicative participation [ ]. Technology systems need to be evaluated in terms of their usability to maximize their acceptance and benefits. The International Organization for Standardization (ISO) 9241 defined usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [ ]. Evaluation of usability is key to guiding the development of efficient and effective technologies that end users will readily adopt by providing information about how a user uses the technology system and the challenges they find while interacting with a system’s interface [ ]. Different usability models have been proposed for evaluating software usability. Gupta et al [ ] proposed a comprehensive hierarchal usability model with a detailed taxonomy, including 7 usability parameters: efficiency, effectiveness, satisfaction, memorability, security, universality, and productivity. Evaluating these usability parameters throughout the design process can allow for continuous improvement of ease of use and can predict the user’s acceptance or rejection of the product [ ]. Therefore, including input from individuals who will use the technology (in the case of SDM technologies, clinicians, patients, and caregivers) through usability testing is a necessary component in designing relevant, understandable, and usable technologies.
The field of rehabilitation science is defined as a multidimensional person-centered process targeting body functions, activities and participation, and the interaction with the environment aiming at optimizing functioning among persons with health conditions experiencing disability . It has seen a recent rise in the development and implementation of technologies aimed at supporting SDM between clinicians, patients, and their caregivers [ ]. However, it is unclear how user input or usability testing is integrated into the design process of these rehabilitation health technologies, including how usability is conceptualized, what measures are used, and at what stage of design usability is evaluated. To date, few studies, and no systematic or scoping reviews that we are aware of, have addressed how usability is measured among rehabilitation technologies supporting SDM. Given the complexity of usability, a thorough examination of the methodologies used to measure usability in this context is required to comprehensively map what has been done and inform future research efforts. A greater understanding of how the parameters of usability are measured will guide future usability testing to inform further development of SDM technologies designed to enhance patient-centered care in rehabilitation. Therefore, this scoping review was conducted to provide knowledge about the methods and measures used to determine the usability of rehabilitation technologies aimed at supporting SDM at different phases of technology development and implementation.
This scoping review followed the methodology described by Arksey and O’Malley  and was reported according to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines [ ] ( ).
Identifying the Research Questions
This scoping review aimed to answer the following research questions: (1) which methods and measures have been used to produce knowledge about the usability of rehabilitation technologies aimed at supporting SDM at the different phases of development and implementation? (2) Which parameters of usability have been measured and reported in studies focusing on rehabilitation technologies aimed at supporting SDM?
The eligibility criteria for this scoping review are outlined in.
Inclusion and exclusion criteria.
- Articles published in peer-reviewed journals, including quantitative (randomized controlled trials or nonrandomized controlled trials), qualitative, and mixed methods studies
- Articles including different groups of people, such as health care practitioners and individuals seeking rehabilitation services (ie, patients and their caregivers) or case managers
- Articles that focused on the usability of technology in making decisions
- Articles reporting a clear objective to evaluate the usability of shared decision-making (SDM) technologies in rehabilitation
- Nonstructured reviews, protocols, descriptive reviews, nonhuman studies, and gray literature
- Articles not focusing on or measuring the usability of technologies in SDM and groups not related to the health care sector (ie, students)
The search strategy was developed in collaboration with a health science librarian. As health system issues often change with models of care delivery, the economic climate, and the environment , we decided to narrow the scope of the search (2005 to 2020). The following electronic databases were searched in both English and French: Ovid MEDLINE, Embase, CINAHL, and PsycINFO. A combination of Medical Subject Heading terms, subject headings, and keywords was used and covered five concepts: (1) usability OR user* friendl* OR eas* to use OR useful* OR user* perspective* OR patient* perspective* OR client* perspective* OR user* experience* AND (2) rehabilitation OR telerehabilitation OR tele rehabilitation OR disabled OR disabilit* OR physical limitation* OR mental limitation* OR psycho* limitation* OR adaptation* OR mobility OR occupational therap* OR physiotherap* OR physical therap* OR speech languag* pathol* OR speech therap* OR language therap* OR communication disorder* AND (3) think* aloud OR focus group* OR interview* OR Wizard* OR Empathy map* OR Persona* OR Questionnaire* OR instrument* OR scale* OR tool OR tools OR measurement* OR survey* OR drama OR deliberation* OR evaluation* OR assessment* OR video confrontation* OR photo voice* AND (4) technolog* OR gerontotechnolog* OR smart* OR intelligen* OR ambient assisted living OR virtual reality OR virtual rehabilitation OR telemonitoring OR telehealth OR telemedicine OR telerehabilitation OR ehealth OR tele monitoring OR tele health OR tele medicine OR tele rehabilitation OR e health or sensor* OR biosensor* OR mobile app* OR product* OR internet OR web OR computer* OR software* OR device* OR self-help OR wheelchair* OR wheelchair* OR communication aid* AND (5) shared decision making OR Decision-Making OR patient-provider communication OR decision aid OR decision support. This was followed by hand searches of the reference lists of the included studies (the search strategy for Ovid MEDLINE is presented in ).
All identified studies were uploaded into EndNote X9.1 (Clarivate Analytics), and duplicates were removed. In total, 2 independent reviewers conducted the selection of abstracts starting with a pilot phase involving the examination of the first 10 titles and abstracts to screen and decide on retention of the abstract based on the inclusion criteria. Interrater agreements were assessed using the κ statistic . Interrater agreement of <75% resulted in a clarification of the eligibility criteria and a revision if needed. The process was repeated twice between the reviewers until an agreement of 75% was reached, which is evidence of excellent agreement [ ]. Finally, all eligible studies and those classified as unclear (ie, requiring further information to make a final decision regarding their retention) were independently reviewed as full-text articles. Disagreements at this stage were resolved through consensus. The PRISMA-ScR flow diagram [ ] was used to guide the selection process.
In total, 2 reviewers independently extracted data from the included articles to avoid missing relevant information. The data extracted included information corresponding to study design, rehabilitation technology intervention used (ie, setting, content, and detail of the type of user interface), population studied (participant demographics and target conditions), characteristics of the measures, and the development stage.
Descriptive statistics were used to describe the characteristics of the included studies, study design, characteristics of the study population, and geographical location. Findings were categorized based on study designs, parameters of usability, types of technologies, stage of development of the technology, and usability evaluation methodologies.
Types of SDM technologies and usability evaluations were mapped to parameters of usability based on a comprehensive hierarchal usability model presented by Gupta et al . The usability parameters include efficiency, defined as “enables user to produce desired results with respect to investment of resources”; effectiveness, defined as “a measure of software product with which user can accomplish specified tasks and desired results with completeness and certainty”; satisfaction, defined as “a measure of responses, feelings of user when users are using the software i.e., freedom from discomfort, likeability”; memorability, defined as “the property of software product that enables the user to remember the elements and the functionality of the system product”; security, defined as “the degree to which risks and damages to people or other resources i.e. hardware and software can be avoided”; universality, defined as “the accommodation of different cultural backgrounds of diverse users with software product and practical utility of software product”; and productivity, defined as “the amount of useful output with the software product” [ ] ( ).
The usability evaluation methodologies were mapped based on the framework by Jacobsen . The categories of the usability evaluation methods included (1) empirical methods, based on users’ experience with the technology in a systematic way; (2) inspection methods, conducted by experts who examined usability-related aspects of a user interface without involving any users; and (3) inquiry methods, based on the information about users’ needs, likes, and understanding of the technology through interviews or focus groups, observation, and verbal or written questions [ ].
Usability parameters based on a comprehensive hierarchal usability model presented by Gupta et al .
- User effort
- Task accomplishment
- Memorability of structure
- Consistency of structure
- Error tolerance
- Cultural universality
- Useful user task output
Consulting and Translating Knowledge
This scoping review is part of an initiative (Réseau provincial de recherche en adaptation-réadaptation–RS6 Technologies de readaptation [Quebec Rehabilitation Research Network]; ) to create an interactive directory of methodological tools for measures of the usability of rehabilitation technologies. Stakeholder consultations with members of the Réseau provincial de recherche en adaptation-réadaptation–RS6 group were held at the beginning of the process (requesting feedback to refine the research question for data extraction and synthesis), during the study (validating the data extraction and deciding on the best way to align the information with stakeholders’ needs), and when the final results were available (knowledge mobilization).
A total of 430 studies were identified from electronic searches, and a total of 19 were identified through hand sorting reference lists. We excluded 57.2% (257/449) of the studies at the title and abstract stage, resulting in 192 full-text articles. Of these 192 studies, 154 (80.2%) were excluded at the full-text stage, resulting in 38 (19.8%) studies. The search strategy was updated in November 2020 and followed the PRISMA-ScR flowchart of the selection process. Reasons for exclusion of studies are provided in. Interrater agreement reached ≥75%, which is evidence of excellent agreement. Disagreements were resolved through consensus.
Characteristics of the Included Studies
The characteristics of the included studies are presented in[ - ]. Overall, the 38 included studies were published between 2008 and 2020 as peer-reviewed studies. Studies were published in the United States (17/38, 43%), Europe (14/38, 37%), Canada (5/38, 13%), and Asia (2/38, 5%). The study designs of the included studies were mixed methods (20/38, 53%), qualitative (12/38, 31%), and quantitative (6/38, 16%).
Characteristics of the Included Participants
presents the characteristics of the included participants. The number of participants across all the included studies was 2138, with age ranging between 18 and 86 years. Participants of usability evaluations included patients (38/38, 100%); clinicians (32/38, 84%); caregivers or family (12/38, 32%); and others (6/38, 16%), including case managers, drug advisory committees, computer scientists, behavioral scientists, communication scientists, clinical administrators, service providers, and social service providers. The target end users of the developed SDM technologies were mainly patients and clinicians (24/38, 63%). The recruitment methods and settings varied across the included studies, including hospitals (24/38, 63%), the community (10/38, 26%), and universities (4/38, 11%).
Usability Definitions and Parameters
presents usability definitions and parameters provided by the authors across the included studies. Notably, only 50% (19/38) of the included studies provided an a priori definition of usability or listed parameters of usability. Usability parameters were categorized as effectiveness (9/38, 23%), efficiency (8/38, 21%), memorability (11/38, 29%), satisfaction (14/38, 37%), security (5/38, 13%), universality (4/38, 10%), and productivity (10/38, 26%) based on Gupta et al [ ].
|Study||Definition of usabilitya||Usability parametersa||Gupta et al  framework|
|Bauerle Bass et al , 2018||User testing was completed to assess the extent to which the tool was understandable, how easily it could be navigated, and its relevance to patients taking HCVb+methadone.|
|Berry et al , 2015||Usability testing is the evaluation of information systems through testing by representative users, enabling evaluation of social acceptability, practicality, and usability of a technology.|
|Bogza et al , 2020||NRc|
|Chrimes et al , 2014||Refers to commentary on the perceived effectiveness, efficiency, and ease of use, or lack thereof, of the ADAPTd Toolkit.|
|Cox et al , 2015||Usability describes the quality of a user’s experience with software or an IT considering their own needs, values, abilities, and limitations.|
|Cuypers et al , 2019||NR|
|Danial-Saad et al , 2016||Usability is defined by the ISOe 9241 as the “extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.”|
|De Vito Dabbs et al , 2009||The measure of the ease with which a system can be learned and used, including its safety, effectiveness, and efficiency.|
|Fleisher et al , 2008||Whether patients found the tools easy to use and navigate, as well as the readability and usefulness of the physician report. Usability protocol based on NCIf guidelines.|
|Fu et al , 2020||Usability is defined by the ISO 9241-11 as the extent to which a product can be used by a specific person in a specific context to achieve realistic goals of effectiveness, efficiency, and satisfaction.|
|Goud et al , 2008||NR|
|Grim et al , 2017||Usability refers to commentary. Understandability and usefulness are 2 major constructs when talking about usability. Understandability refers to the extent to which the descriptive texts and items are comprehensible. Usefulness refers to commentary on the extent to which the features in the decision aid are perceived as supporting decision-making processes on the perceived effectiveness, efficiency, and ease of use, or lack thereof, of the decision aid.|
|Kallen et al , 2012||Usability was considered an incorporation of system effectiveness, efficiency, and user satisfaction. Usability was defined in the context of the assessment and review of tasks assigned to study participants.|
|Li et al , 2013||A usability issue was defined as (1) when a participant was not able to advance to the next step because of the decision aid design or a programming error or (2) when a participant was distracted by a particular design or content of the web tool.|
|Rochette et al , 2008||The term “usability” is defined as the effectiveness, efficiency, and satisfaction with which users can achieve tasks in a particular environment. High usability means that a system is easy to learn and remember, efficient, visually pleasing, and fun to use and enables quick recovery from errors.|
|Span et al , 2014||NR|
|Støme et al , 2019||NR|
|Van Maurik et al , 2019||Clinicians were asked to complete the SUSg after using the tool.|
|Williams et al , 2016||NR|
|Zafeiridi et al , 2020||Usability is measured as the user-friendliness (eg, ease to learn) and perceived usefulness in addressing users’ needs.|
aAs defined by the authors.
bHCV: hepatitis C virus.
cNR: not reported.
dADAPT: Avoiding Diabetes Through Action Plan Targeting.
eISO: International Organization for Standardization.
fNCI: National Cancer Institute.
gSUS: System Usability Scale.
Technology for SDM
presents the type of SDM technologies that were used across the included studies. Technologies for SDM included clinical decision support systems (9/33, 27%), mobile health apps (9/33, 27%), and web-based aids (15/33, 46%). The SDM context was mainly between clinicians and patients (32/36, 89%). The types of technology for SDM were mapped to usability parameters, including effectiveness (10/38, 26%), efficiency (11/38, 29%), memorability (20/38, 53%), satisfaction (27/38, 71%), security (5/38, 13%), universality (4/38, 10%), and productivity (16/38, 42%) based on Gupta et al [ ]. The most common SDM technologies evaluated for usability were web-based aids. Satisfaction was the most common usability parameter mapped to types of SDM technologies.
|Study||Title of developed technology||Technology overview||Stage of development of technology intervention||Framework followed or guidelines by the authors||Description of SDM context or type of decision-making||Usability parameters measured||Gupta et al  framework|
|Anderson et al , 2014||STOPa Tool||Web-based user interface for adaptive clinical decision support integrated into electronic health record||Preimplementation||Framework based on usability engineering||SDM between patients and clinicians for self-management and secondary stroke prevention|
|Barrio et al , 2017||SIDEALb||Mobile app||Developmental laboratory||MIc was the main source of guidance throughout the development process.||SDM between patients and clinicians related to self-management of alcohol dependence|
|Bauerle Bass et al , 2018||“Take Charge, Get Cured”||mHealthd decision support tool||Development||Model of illness self-regulation, information-communication theory, and formative evaluation framework||SDM between patients and clinicians related to initiating hepatitis C treatment|
|Berry et al , 2015||P3Pe||Web-based decision aid||Preimplementation||NRf||SDM between patients and clinicians about prostate cancer management options|
|Bogza et al , 2020||Web-based decision aids||—g||Development||User-centered approach; Center for eHealth and Wellbeing Research guidelines||SDM between patients and clinicians|
|Burns and Pickens , 2017||NR||Technology-based CDSSh for app-based assessments||Preimplementation||NR||SDM between providers, client, and family for home evaluation and modifications|
|Canally et al , 2015||NR||GUIsi||Developmental laboratory||NR||Shared decision support system that integrated biophysiological information obtained through multiple nonintrusive monitoring for home care|
|Chrimes et al , 2014||ADAPTj||Clinical decision support tool integrating evidence-based shared goal-setting components into electronic health record||Developmental laboratory||ADAPT framework||SDM between patients and clinicians for behavior changes to manage prediabetes|
|Cox et al , 2015||eCODESk||Web-based decision aid integrated into data entry and management system||Developmental laboratory||NR||SDM between clinicians and surrogate decision makers of patients receiving prolonged mechanical ventilation|
|Cuypers et al , 2019||Web-based decision aid system||—||Development||On the basis of existing evidence-based Canadian decision aid, developed by Feldman-Stewart et al [- ]||SDM between patients and clinicians||NR||NR|
|Danial-Saad et al , 2016||OSCARl||Interactive CDSS||Development laboratory||LUCIDm framework||Server-client system to recommend and select optimal pointing device|
|De Vito Dabbs et al , 2009||Pocket PATHn||IHTo through handheld computer device||Preimplementation||User-centered design||SDM between patients of lung transplant and their transplant team about self-monitoring of critical values|
|Fleisher et al , 2008||CONNECTp||Interactive web-based communication aid||Preimplementation||C-SHIPq model||SDM between patients and clinicians about treatment decisions supported through communication skill development modules|
|Flynn et al , 2015||COMPASSr prototype||User interface with decision analytical model developed on iPad mobile device||Developmental laboratory||Decision analytic model predictions developed from S-TPIs||SDM between clinicians and patients about patient-specific treatment options for acute ischemic stroke and personalized information to patients|
|Fu et al , 2020||Mobile apps||—||Testing||Nielsen heuristics||Unclear|
|Goud et al , 2008||CARDSSt||Guideline-based computerized decision support systems||Implementation||Clinical guidelines||SDM between clinicians and patients for patient-specific care for cardiac rehabilitation and patient management|
|Grim et al , 2017||NR||Interactive web-based software||Preimplementation||The team followed published evidence on the consensus guidelines for development of decision aids and SDM.||SDM between patients and clinicians about care in psychiatric services|
|Holch et al , 2017||e-RAPIDu||Integrated electronic platform for patient self-report||Preimplementation||Translational research model||SDM between patients and clinicians for management of events during cancer treatment|
|Jameie et al , 2019||Cardiac telerehabilitation platform||—||Development||BACPRv||SDM between patients and clinicians||NR||NR|
|Jessop et al , 2020||“Take Charge, Get Cured”||mHealth treatment decision support tool embedded in Articulate 360 app||Preimplementation||NR||SDM between patients and physicians about hepatitis C treatment||NR||—|
|Kallen et al , 2012||PROw-based Palliative and Hospice Care Management System—prototype||Electronic PRO system||Implementation||User-centered design approach||SDM between patients in palliative care and treating physician or nurse|
|Li et al , 2013||ANSWERx||Web-based decision aid with educational modules||Preimplementation||The International Patient Decision Aid Standards and the Jabaja-Weiss edutainment decision aid model||SDM between patients and clinicians about using methotrexate|
|Murphy et al , 2020||CP-PDAy||Web-based algorithmic intervention||Development||International Patient Decision Aid Standards criteria checklist, SUNDAEz checklist, and the EQUATORaa CONSORTab checklist||SDM between patients and clinicians about postprostatectomy care regarding continence product choice|
|Rochette et al , 2008||StrokEngine-Family||Stroke rehabilitation layperson website||Implementation||NR||SDM between patients and clinicians|
|Setiawan et al , 2019||iMHereac 2.0||Adaptive mHealth system with mobile app modules (client app, caregiver app, web-based clinician portal, back-end server, and 2-way communication protocol)||Development||User-centered design||Monitoring and support of self-management for people with chronic conditions and disabilities and allowing for personalized and adaptive treatment strategies|
|Schön et al , 2018||Digital interactive decision support tool||—||Development||The decision support tool is based on the theoretical framework of SDM.||SDM between patients and clinicians|
|Snyder et al , 2009||PatientViewpoint prototype||Web-based system to collect PROs linked with electronic medical record||Preimplementation||NR||SDM between patients and clinicians for cancer management|
|Span et al , 2014||DecideGuide||Interactive web tool||Developmental laboratory||NR||SDM in dementia care networks between patients, care managers, and informal caregivers|
|Span et al , 2018||DecideGuide||Interactive web tool||Preimplementation||The 5 phases of the CeHResad road map||SDM made by care network of people with dementia (patients, care managers, and informal caregivers)|
|Støme et al , 2019||Vett interactive mobile app||—||Implementation||NR||Unclear|
|Tony et al , 2011||EVIDEMae decision support framework||MCDAaf and HTAag||Developmental laboratory||EVIDEM framework||SDM between patients and clinicians to appraise health care interventions|
|Toth-Pal et al , 2008||EviBase||CDSS through internet-based application||Implementation||Clinical guidelines (1 Swedish and 2 European)||SDM between clinicians and patients through integration of individual patient data with guidelines for management of chronic heart failure|
|Tsai et al , 2019||MagicPlan||Mobile app with laser distance measurer||Preimplementation||NR||Clinical home evaluations with virtual floor plan for DMEah recommendations|
|Van Maurik et al , 2019||Web-based diagnostic support tool named ADappt||—||Development||NR||SDM between patients and clinicians|
|Welch et al , 2015||MedMinder||Cellular pillbox monitoring device||Implementation||NR||SDM between clinicians and patients related to treatment and adherence support|
|Williams et al , 2016||NR||Clinical decision support on mHealth app||Developmental laboratory||User-centered design approach (user interface and user experience design)||SDM between clinicians and patients for patient-specific recommendations for cardiovascular disease|
|Zafeiridi et al , 2018||CAREGIVERSPRO-MMDai||Web-based platform||Development||User-centered design||Social network for sharing information, tips, and support across peers and health professionals|
|Zheng et al , 2017||NR||mHealth app with PROs||Preimplementation||User-centered design principles||SDM between patients and clinicians for knee arthritis treatment|
aSTOP: Self-Management to Prevent Stroke.
bSIDEAL: Soporte Innovador al paciente con Dependencia del Alcohol, Innovative Support to the Alcohol Dependent Patient.
cMI: motivational interviewing.
dmHealth: mobile health.
eP3P: The Personal Patient Profile-Prostate.
fNR: not reported.
gData not available.
hCDSS: clinical decision support system.
iGUI: graphical user interface.
jADAPT: Avoiding Diabetes Through Action Plan Targeting.
keCODES: Electronic Collaborative Decision Support.
lOSCAR: Ontology-Supported Computerized Assistive Technology Recommender.
mLUCID: logical user-centered interaction design.
nPATH: Personal Assistant for Tracking Health.
oIHT: interactive health technology.
pCONNECT: web-based communication aid.
qC-SHIP: Cognitive-Social Health Information Processing.
rCOMPASS: Computerized Decision Aid for Stroke Thrombolysis.
sS-TPI: Stroke-Thrombolytic Predictive Instrument.
tCARDSS: Cardiac Rehabilitation Decision Support System.
ue-RAPID: Electronic Patient Self-Reporting of Adverse-Events: Patient Information and Advice.
vBACPR: British Association for Cardiovascular Prevention and Rehabilitation.
wPRO: patient-reported outcome.
xANSWER: Animated, Self-Serve, Web-Based Research Tool.
yCP-PDA: Continence Product Patient Decision Aid.
zSUNDAE: Standards for Universal Reporting of Patient Decision Aid Evaluations.
aaEQUATOR: Enhancing the Quality and Transparency of Health Research.
abCONSORT: Consolidated Standards of Reporting Trials.
aciMHere: Interactive Mobile Health and Rehabilitation.
adCeHRes: Center for eHealth Research and Disease Management.
aeEVIDEM: Evidence and Value: Impact on Decision-Making.
afMCDA: multicriteria decision analysis.
agHTA: health technology assessment.
ahDME: durable medical equipment.
aiCAREGIVERSPRO-MMD: Caregivers Patient-Reported Outcome-Mild Mental Disorder.
Usability Evaluation Methods
The usability evaluation methods were categorized, based on the framework by Jacobsen , into (1) empirical (think-aloud protocol, 14/38, 36%; user tracking, 3/38, 8%; performance measures, 4/38, 10%; field test, 2/38, 5%; video recording, 1/38, 2%; and screen capture, 2/38, 5%), (2) inspection (cognitive walk-through, 1/38, 2% and Near live clinical situation, 1/38, 2%), and (3) inquiry (focus groups, 3/38, 8%; workshops, 2/38, 5%; semistructured interviews, 16/38, 42%; structured interviews, 1/38, 2%; questionnaires, 24/38, 63%; observations, 5/38, 13%; and comments, 3/38, 8%; ). An important point to emphasize is the frequency with which researchers used 1 (13/38, 34%), 2 (15/38, 39%), 3 (7/38, 18%), 4 (2/38, 5%), and 6 (1/38, 2%) methods from the framework by Jacobsen [ ], presented in [ - ]. Most (28/38, 73%) used 1 or 2 methods of evaluation. Usability was assessed during development (18/38, 47%), preimplementation (13/38, 34%), or implementation (7/38, 18%) through a variety of measures, including usability questionnaires (15/38, 39%), tailored tools developed by the authors (17/38, 45%), and acceptance and satisfaction questionnaires (6/38, 16%). The usability evaluation parameters identified by the authors were mapped to the usability parameters explained by Gupta et al [ ], including effectiveness (13/38, 34%), efficiency (12/38, 31%), memorability (13/38, 34%), productivity (2/38, 5%), security (2/38, 5%), and satisfaction (32/38, 84% and ).
|Study||Method||Jacobsen  framework||Details|
|Anderson et al , 2014|
|Barrio et al , 2017|
|Bauerle Bass et al , 2018|
|Berry et al , 2015|
|Bogza et al , 2020|
|Burns and Pickens , 2017|
|Canally et al , 2015|
|Chrimes et al , 2014|
|Cox et al , 2015|
|Cuypers et al , 2019|
|De Vito Dabbs et al , 2009|
|Danial-Saad et al , 2016|
|Fleisher et al , 2008|
|Flynn et al , 2015|
|Fu et al , 2020|
|Goud et al , 2008|
|Grim et al , 2017|
|Holch et al , 2017|
|Jameie et al , 2019|
|Jessop et al , 2020|
|Kallen et al , 2012|
|Li et al , 2013|
|Murphy et al , 2020|
|Rochette et al , 2008|
|Schön et al , 2018|
|Setiawan et al , 2019|
|Snyder et al , 2009|
|Span et al , 2014|
|Span et al , 2018|
|Støme et al , 2019|
|Tony et al , 2011|
|Toth-Pal et al , 2008|
|Tsai et al , 2019|
|Van Maurik et al , 2019|
|Welch et al , 2015|
|Williams et al , 2016|
|Zafeiridi et al , 2018|
|Zheng et al , 2017|
aUSE: Usefulness, Satisfaction and Ease of Use.
bSUS: System Usability Scale.
cASQ: After-Scenario Questionnaire.
dPSSUQ: Post-Study System Usability Questionnaire.
eCSUQ: Computer System Usability Questionnaire.
fPrepDM: Preparation for Decision Making.
gPRO: patient-reported outcome.
hNR: not reported.
|Usability measures||Type of scale||Items||Usability evaluation parameters identified by the authors||Gupta et al  framework|
|Acceptability questionnaire ||5-point Likert scale|
|ASQa ||7-point Likert scale|
|Clinical decision support report questionnaire ||5-point Likert scale|
|Feasibility of encodes ||—e|
|IBM CSUQf ||7-point Likert scale|
|LQg ||6-point Likert scale|
|Measures of accessibility and satisfaction ||5-point Likert scale|
|Mobile app usability concept questions ||5-point Likert scale|
|Nielsen heuristic checklist ||Likert scale|
|Open-ended questionnaire for usability ||—|
|Patient satisfaction with diabetes telehealth program questionnaire ||5-point Likert scale|
|Perceived general helpfulness and value ||3-point Likert scale|
|Perceived helpfulness ||10-point Likert scale|
|Perceived usefulness ||10-point Likert scale|
|PSSUQk [, ]||7-point Likert scale|
|PrepDMl scale ||5-point Likert scale|
|Remote home monitoring device usability questionnaire ||5-point Likert scale|
|Survey for perception of easiness and usability of the 6 interfaces ||5-point Likert scale|
|Survey satisfaction questionnaire ||Likert scale|
|Telehealth nurse satisfaction questionnaire ||5-point Likert scale|
|Tool developed by the authors focused on description ||5-point Likert scale|
|Usability questionnaire ||100-point Likert scale|
|Usability questionnaire ||10-point Likert scale|
|Usability scale (SUS) questionnaire [, , , - , , , , ]||5-point Likert scale|
|USEp questionnaire ||7-point Likert scale|
|Usefulness and relevance survey ||7-point Likert scale|
|User acceptance and satisfaction scale ||5-point Likert scale|
|User-friendliness measured with an instrument based on the CeHResq assessment of design quality ||—|
|User-specific evaluation questionnaire for clinicians ||—|
|User-specific evaluation questionnaire for patients and caregivers ||—|
aASQ: After-Scenario Questionnaire.
bBG: blood glucose.
cBP: blood pressure.
dEMR: electronic medical record.
eData not available.
fCSUQ: Computer System Usability Questionnaire.
gLQ: learnability questionnaire.
hRHM: remote home monitoring.
iT2D: type 2 diabetes.
jHCV: hepatitis C virus.
kPSSUQ: Post-Study System Usability Questionnaire.
lPrepDM: Preparation for Decision-Making.
mDSM: diabetes self-management.
nPCP: primary care provider.
oCDS: clinical decision support.
pUSE: Usefulness, Satisfaction, and Ease of Use.
qCeHRes: Center for eHealth Research and Disease Management.
Frameworks and Theoretical Models
The frameworks and theoretical models reported by the authors during the development, implementation, and evaluation of the technologies to support SDM reflected 5 categories: technology design (15/38, 39%), behavior change (21/38, 21%), analysis (9/38, 24%), SDM framework (8/38, 21%), and not reported (9/38, 24%;). Notably, 24% (9/38) of the studies did not report using a framework or model during any stage of their research. Authors most commonly reported using a model or framework as a foundation to inform the design of their respective SDM technologies. User-centered design (9/15, 60%) was the most frequently used technology design framework.
This scoping review was conducted to provide knowledge about how usability is evaluated when developing or implementing rehabilitation technologies aimed at supporting SDM. The first research question examined the methods and measures used in the context of SDM at different phases of technology development and implementation. Our findings revealed 14 reported methods that can help in evaluating the overall functionalities of the system and whether it fulfills the users’ requirements  and can be effective for identifying issues with a system [ ]. The most frequent reported methods included think-aloud protocols (14/38, 36%), semistructured interviews (16/38, 42%), and questionnaires (24/38, 63%; ). There was a total of 30 usability measures reported ( ), with the System Usability Scale being the most frequently used among the included studies. We operationalized the different types of methods used through the model by Jacobsen [ ], reflecting empirical methods (based on users’ experience with the technology in a systematic way), inspection methods (conducted by experts who examine usability-related aspects of a user interface without involving any users), and inquiry methods (based on the information about users’ needs, likes, and understanding of the technology through interviews or focus groups, observation, or comments). Notably, the reported methods were predominantly classified as inquiry and empirical ( ).
The second research question examined the parameters of usability that were measured and reported. We found that the methods used to evaluate different parameters of usability varied according to the a priori framing of usability, demonstrated by the variations in the definitions of usability described by the authors (). There was an evolution in the definition of usability across the included studies, with more recent studies (published since 2016) using the unified definition proposed by the ISO [ , , , , , , , ]. The usability parameters of the definitions were categorized based on the proposed comprehensive hierarchal model by Gupta et al [ ] as effectiveness (9/38, 23%), efficiency (8/38, 21%), memorability (11/38, 29%), satisfaction (14/38, 37%), security (5/38, 13%), universality (4/38, 10%), and productivity (10/38, 26%). These are consistent with the 3 constructs of the ISO standards, which are effectiveness, efficiency, and satisfaction, and allows for a more detailed categorization of usability parameters.
Although the ISO standards  and the usability model by Gupta et al [ ] provide dimensions that could be considered as primary usability parameters, there remain challenges with measuring usability that emerged in this review. On the surface, usability is a simple concept. In fact, simplicity is at the heart of usability; however, measuring usability is not simple. Paradoxically, the ISO definition of usability is complex. Usability is about the person’s experience; however, that experience is influenced by many aspects, such as a person’s behavior and social network and the complexity of the technological functionalities. Usability may be viewed as a feature of the technology or an emergent property of the interaction between the user, the system, and contextual factors. Evaluating usability through these lens leads to using inspection, empirical, or inquiry methods [ ]. These can be applied at different stages of development of a technology (ie, in a developmental laboratory, in preimplementation, or during implementation), as described by the included studies ( ).
This review revealed that evaluating usability requires a comprehensive approach with several methods to cover multiple usability parameters. Most articles included in this review (36/38, 95%) focused on inquiry methods, relying heavily on questionnaires and semistructured interviews to evaluate usability, and the most frequent empirical method was think-aloud protocols (). Although a comprehensive approach is suggested for accurate usability evaluation, this was largely not shown in the included articles. Rather, 73% (28/38) of the included studies only used 1 or 2 methods in total to evaluate usability. Only 2% (1/38) of the studies, conducted by Span et al [ ], incorporated multiple methods that covered all 3 dimensions—inquiry, inspection, and empirical [ ]. However, some of the included studies (2/38, 5%) described different usability evaluations for the same technology at different stages of development in separate articles (eg, “Take Charge, Get Cured” in the developmental [ ] and preimplementation [ ] stages). It is believed that the combination of inspection, empirical, and inquiry methods can provide more accurate and complete results in finding usability problems as there is no exact method considered to be the best for usability evaluation [ ]. Matera et al [ ] developed a systematic usability evaluation framework to address this challenge. They posited that usability can be reliably evaluated by systematically combining evaluation methods [ ]. Recent reviews of usability not specific to SDM in software [ ], mobile health [ ], eHealth [ ], user experience [ ], and web development [ ] mirrored the results of this review in that few studies used a combination of evaluation methods.
However, the lack of reported inspection methods demonstrated in this review may partially be explained by the inherent nature of SDM technologies for rehabilitation rather than a lack of comprehensive evaluation. Very few examples of inspection methods were demonstrated across the included studies, with only 2% (1/38) using cognitive walk-throughs and an additional 2% (1/38) using “near live” clinical situations. Critically, inspection methods refer to evaluations conducted by specific usability experts , not by the end users of the technology (eg, patients and clinicians). As the purpose of technology to support SDM in rehabilitation is to improve patient-centered care, the consideration of end users in the development—and, consequently, the usability evaluations—is crucial to ensure that the technology will be understood and adopted by the target population. Therefore, we propose that a comprehensive approach for evaluating the usability of rehabilitation technologies aimed at supporting SDM could focus on empirical and inquiry methods to prioritize the input of the patient and clinician end users.
Although questionnaires were found to be the most common method used overall, the identified measures of usability in the included studies demonstrated limitations in comprehensiveness, largely mapping to the parameters of satisfaction and memorability (). The emphasis on the parameter of satisfaction (demonstrated in 32/38, 84% of measures) may reflect the importance of this parameter when developing technologies for SDM in rehabilitation (eg, the importance of evaluating the usefulness, user-friendliness, and ease of use). However, this may also reflect key missing areas in usability evaluation. Critically, the parameters of usability described by the authors in their a priori definitions of usability were not found to be consistent with the parameters of the measures that were used. Therefore, although individuals may be conceptualizing usability in a comprehensive manner, the measurement itself was not comprehensive. For example, there was a demonstrated lack of measurement of the parameters of effectiveness and efficiency, which were both described in the definition of usability in 34% (13/38) of the included studies, although both were only found to be used in 23% (9/38) of usability measures.
This review uncovered the need for inclusion of theoretical models or frameworks during various stages of SDM usability studies to guide which usability parameter to measure. Theoretical models and frameworks were infrequently reported (). Most studies in this review (27/38, 71%) reported using 1 model or framework, whereas some (10/38, 26%) integrated 2. Only 2% (1/38) of the studies, carried out by Bauerle Bass et al [ ], exhibited an in-depth application of models and frameworks as underpinnings to their research. The most common (9/38, 24%) and perhaps the most beneficial framework, user-centered design, served as the foundation for designing an SDM technology [ , , ].
The importance of using theoretical models and frameworks during the development, implementation, and analysis of technologies and evaluation of usability is demonstrated through the implications of poor usability [, , ], which discourages users from using the technology systems. Moreover, if the technology systems are not user-friendly, then they can increase the problems experienced by users. Solutions to systems failing to meet the users’ needs include understanding user feedback [ ], usability evaluations [ ], involving users in the early stages of development [ ], and including professionals such as providers [ ]. There is a need for flexibility and for friendly, simple, and self-explanatory interfaces that allow users to interact with the system [ ]. For the systems to be effective, it is important to assess a system that is easy to use on a daily basis. This would increase the ability of the patients to control their diseases and allow their daily lives to be more satisfying [ ]. The technology systems need to be designed for a particular type of user and need to be easy to use to create acceptance. The usability of the technology system is vital as it has a high degree of influence over the success of the system. Thus, the system needs to be designed to provide a friendly environment for the user to develop a positive attitude toward using it and lead to its successful adoption.
It is envisioned that the involvement of end users in the development of SDM technologies will continue to grow and that more applications of existing technology, such as mobile phones, websites, or applications, will be used to benefit individuals with disabilities. We also anticipate that more companies may show an interest in this market, potentially promoting frequent use of SDM technologies in rehabilitation care. However, there are challenges in the development of SDM technologies, such as tailoring to individuals’ capabilities and properly addressing the emotional state of individuals with disabilities or cognitive impairments during everyday tasks. It will be critical to develop these technologies in a way that meets individual variations in needs and abilities of individuals with disabilities so that they really help maintain autonomy, provide meaningful activities, and promote decision-making [, , ].
An important area for this growing field will be how to effectively integrate end-user input throughout all stages of development of such SDM technologies, including effective usability testing. An additional challenge for the field of rehabilitation care in supporting SDM technologies would be in integrating the technology into the built environment, such as a client-server system, and into routine care . There is a clear need for new methods of rapid SDM technology appraisal and evaluation to inform deployment to overcome the barriers that will be faced because of the expected further integration of SDM technologies within the built environment.
We did not assess the quality of the included articles, consistent with the scoping review methodology [, ]. Therefore, we included studies with different designs and different quality levels, which allowed for a broad exploration of measures and methods used to evaluate the usability of SDM technologies. In our results, we focused mainly on general usability measures and did not report the psychometric properties and clinical utility of these measures. Future work needs to evaluate the psychometric properties and clinical utility of usability measures through a systematic review methodology with a quality assessment of the included articles. Another limitation was that we did not include gray literature as this scoping review aimed to examine the reported measures and methods used in peer-reviewed rehabilitation literature on SDM technologies. It could be an area of interest for future work to examine what methods and measures are used in gray literature.
The results of this scoping review highlight the importance as well as the complexity of usability evaluation. Although various methods and measures were shown to be used to evaluate the usability of technologies to support SDM in rehabilitation, very few evaluations used in the included studies adequately spanned the selected usability parameters. This review identified gaps in usability evaluation as most studies relied solely on questionnaires rather than a combination of inspection and empirical methods and most questionnaires simply focused on the usability parameter of satisfaction. We recommend for individuals to adopt a comprehensive approach to usability evaluation of SDM technologies, starting with a clear definition of how usability is conceptualized to guide the structure of the evaluation. In addition, we recommend the use of multiple usability evaluation methods categorized as inspection (eg, questionnaires, focus groups, and interviews) or empirical (eg, think-aloud protocols) to capture a more complete picture of end-user needs and interpretations. The selected methods should span a variety of parameters of usability, not just satisfaction (eg, effectiveness, efficiency, memorability, security, universality, and productivity). The consideration of end users (such as patients and clinicians) is of particular importance for the development of technologies to support SDM as the process of SDM itself aims to improve patient-centered care and integrate both patient and clinician voices into their rehabilitation care.
RA is supported by Princess Nourah Bint Abdulrahman University, Riyadh, Saudi Arabia. SA and CA are supported by career awards from the Fonds de recherche du Québec–Santé. The funding agency had no role in the design of the study, in the collection and analysis of the data, or in the decision to publish the results. The authors would also like to acknowledge the network investigators from Réseau provincial de recherche en adaptation-réadaptation that provided guidance for this work, particularly Geneviève Lessard, PhD. This study was funded by the Réseau provincial de recherche en adaptation-réadaptation.
All authors contributed to the design of this study, provided critical insights, and contributed to the final written manuscript.
Conflicts of Interest
Ovid MEDLINE search strategy.DOCX File , 15 KB
Characteristics of the included studies and participants.DOCX File , 24 KB
- Barry MJ, Edgman-Levitan S. Shared decision making--pinnacle of patient-centered care. N Engl J Med 2012 Mar 01;366(9):780-781 [CrossRef] [Medline]
- Sandman L, Munthe C. Shared decision making, paternalism and patient choice. Health Care Anal 2010 Mar;18(1):60-84 [CrossRef] [Medline]
- Légaré F, Witteman HO. Shared decision making: examining key elements and barriers to adoption into routine clinical practice. Health Aff (Millwood) 2013 Feb;32(2):276-284 [CrossRef] [Medline]
- Davis S, MacKay L. Moving beyond the rhetoric of shared decision-making: designing personal health record technology with young adults with type 1 diabetes. Can J Diabetes 2020 Jul;44(5):434-441 [CrossRef] [Medline]
- Zisman-Ilani Y, Gorbenko KO, Shern D, Elwyn G. Comparing digital vs paper decision aids about the use of antipsychotic medication: client, clinician, caregiver and administrator perspectives. Int J Pers Cent Med 2017 Jul 13;7(1):21-30 [https://www.proquest.com/openview/6a1f4c9ab441af2d185f21d72b09dc9c/1?pq-origsite=gscholar&cbl=5455939] [CrossRef]
- Rose A, Rosewilliam S, Soundy A. Shared decision making within goal setting in rehabilitation settings: a systematic review. Patient Educ Couns 2017 Jan;100(1):65-75 [CrossRef] [Medline]
- Grenfell J, Soundy A. People's Experience of Shared Decision Making in Musculoskeletal Physiotherapy: A Systematic Review and Thematic Synthesis. Behav Sci (Basel) 2022 Jan 12;12(1):12 [https://www.mdpi.com/resolver?pii=bs12010012] [CrossRef] [Medline]
- Matthews EB, Savoy M, Paranjape A, Washington D, Hackney T, Galis D, et al. Acceptability of health information exchange and patient portal use in depression care among underrepresented patients. J Gen Intern Med 2022 Nov;37(15):3947-3955 [https://europepmc.org/abstract/MED/35132548] [CrossRef] [Medline]
- Zisman-Ilani Y, Roe D, Elwyn G, Kupermintz H, Patya N, Peleg I, et al. Shared decision making for psychiatric rehabilitation services before discharge from psychiatric hospitals. Health Commun 2019 May;34(6):631-637 [CrossRef] [Medline]
- Shay LA, Lafata JE. Where is the evidence? A systematic review of shared decision making and patient outcomes. Med Decis Making 2015 Jan;35(1):114-131 [https://europepmc.org/abstract/MED/25351843] [CrossRef] [Medline]
- Wilson SR, Strub P, Buist AS, Knowles SB, Lavori PW, Lapidus J, Better Outcomes of Asthma Treatment (BOAT) Study Group. Shared treatment decision making improves adherence and outcomes in poorly controlled asthma. Am J Respir Crit Care Med 2010 Mar 15;181(6):566-577 [https://europepmc.org/abstract/MED/20019345] [CrossRef] [Medline]
- Hartasanchez SA, Heen AF, Kunneman M, García-Bautista A, Hargraves IG, Prokop LJ, et al. Remote shared decision making through telemedicine: a systematic review of the literature. Patient Educ Couns 2022 Feb;105(2):356-365 [CrossRef] [Medline]
- Safran Naimark J, Madar Z, Shahar DR. The impact of a web-based app (eBalance) in promoting healthy lifestyles: randomized controlled trial. J Med Internet Res 2015 Mar 02;17(3):e56 [https://www.jmir.org/2015/3/e56/] [CrossRef] [Medline]
- Solomon M, Wagner SL, Goes J. Effects of a web-based intervention for adults with chronic conditions on patient activation: online randomized controlled trial. J Med Internet Res 2012 Feb 21;14(1):e32 [https://www.jmir.org/2012/1/e32/] [CrossRef] [Medline]
- Antonio MG, Petrovskaya O, Lau F. The state of evidence in patient portals: umbrella review. J Med Internet Res 2020 Nov 11;22(11):e23851 [https://www.jmir.org/2020/11/e23851/] [CrossRef] [Medline]
- Seljelid B, Varsi C, Solberg Nes L, Øystese KA, Børøsund E. Feasibility of a digital patient-provider communication intervention to support shared decision-making in chronic health care, involveme: pilot study. JMIR Form Res 2022 Apr 07;6(4):e34738 [https://formative.jmir.org/2022/4/e34738/] [CrossRef] [Medline]
- Davis S, Roudsari A, Raworth R, Courtney KL, MacKay L. Shared decision-making using personal health record technology: a scoping review at the crossroads. J Am Med Inform Assoc 2017 Jul 01;24(4):857-866 [https://europepmc.org/abstract/MED/28158573] [CrossRef] [Medline]
- Kao H, Wei C, Yu M, Liang T, Wu W, Wu YJ. Integrating a mobile health applications for self-management to enhance telecare system. Telemat Inform 2018 Jul;35(4):815-825 [https://www.sciencedirect.com/science/article/abs/pii/S0736585316307080] [CrossRef]
- Taylor L, Capling H, Portnoy JM. Administering a telemedicine program. Curr Allergy Asthma Rep 2018 Sep 15;18(11):57 [CrossRef] [Medline]
- Howe TL, Worrall LE, Hickson LM. What is an aphasia-friendly environment? Aphasiology 2010 Aug 18;18(11):1015-1037 [https://espace.library.uq.edu.au/view/UQ:70606] [CrossRef]
- Jokela T, Iivari N, Matero J, Karukka M. The standard of user-centered design and the standard definition of usability: analyzing ISO 13407 against ISO 9241-11. In: Proceedings of the Latin American Conference on Human-Computer Interaction. 2003 Presented at: CLIHC '03; August 17-20, 2003; Rio de Janeiro, Brazil p. 53-60 URL: https://dl.acm.org/doi/abs/10.1145/944519.944525 [CrossRef]
- Holzinger A. Usability engineering methods for software developers. Commun ACM 2005 Jan;48(1):71-74 [https://dl.acm.org/doi/10.1145/1039539.1039541] [CrossRef]
- Gupta D, Ahlawat AK, Sagar K. Usability prediction and ranking of SDLC models using fuzzy hierarchical usability model. Open Eng 2017 Jun 24;7(1):161-168 [https://www.degruyter.com/document/doi/10.1515/eng-2017-0021/html?lang=en] [CrossRef]
- Schultheis MT, Rebimbas J, Mourant R, Millis SR. Examining the usability of a virtual reality driving simulator. Assist Technol 2007;19(1):1-10 [CrossRef] [Medline]
- Negrini S, Meyer T, Arienti C, Kiekens C, Pollock A, Selb M, 3rd Cochrane Rehabilitation Methodology Meeting participants. The 3rd Cochrane rehabilitation methodology meeting: "rehabilitation definition for scientific research purposes". Eur J Phys Rehabil Med 2020 Oct;56(5):658-660 [CrossRef] [Medline]
- Damman OC, Jani A, de Jong BA, Becker A, Metz MJ, de Bruijne MC, et al. The use of PROMs and shared decision-making in medical encounters with patients: an opportunity to deliver value-based health care to patients. J Eval Clin Pract 2020 Apr;26(2):524-540 [https://europepmc.org/abstract/MED/31840346] [CrossRef] [Medline]
- Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005;8(1):19-32 [https://www.tandfonline.com/doi/abs/10.1080/1364557032000119616] [CrossRef]
- Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018 Oct 02;169(7):467-473 [https://www.acpjournals.org/doi/abs/10.7326/M18-0850?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub 0pubmed] [CrossRef] [Medline]
- Gammon D, Berntsen GK, Koricho AT, Sygna K, Ruland C. The chronic care model and technological research and innovation: a scoping review at the crossroads. J Med Internet Res 2015 Feb 06;17(2):e25 [https://www.jmir.org/2015/2/e25/] [CrossRef] [Medline]
- Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Hoboken, NJ, USA: Wiley-Blackwell; 2008.
- Jacobsen NE. Usability evaluation methods: the reliability and usage of cognitive walkthrough and usability test. Department of Psychology, University of Copenhagen. 1999 Oct 05. URL: https://rauterberg.employee.id.tue.nl/lecturenotes/0H420/uem%5b1999%5d.pdf [accessed 2023-06-21]
- Anderson JA, Godwin KM, Saleem JJ, Russell S, Robinson JJ, Kimmel B. Accessibility, usability, and usefulness of a web-based clinical decision support tool to enhance provider-patient communication around self-management TO prevent (STOP) stroke. Health Informatics J 2014 Dec;20(4):261-274 [https://journals.sagepub.com/doi/10.1177/1460458213493195?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub 0pubmed] [CrossRef] [Medline]
- Barrio P, Ortega L, López H, Gual A. Self-management and shared decision-making in alcohol dependence via a mobile app: a pilot study. Int J Behav Med 2017 Oct;24(5):722-727 [CrossRef] [Medline]
- Bauerle Bass S, Jessop A, Gashat M, Maurer L, Alhajji M, Forry J. Take charge, get cured: the development and user testing of a culturally targeted mHealth decision tool on HCV treatment initiation for methadone patients. Patient Educ Couns 2018 Nov;101(11):1995-2004 [CrossRef] [Medline]
- Berry DL, Halpenny B, Bosco JL, Bruyere Jr J, Sanda MG. Usability evaluation and adaptation of the e-health personal patient profile-prostate decision aid for Spanish-speaking Latino men. BMC Med Inform Decis Mak 2015 Jul 24;15:56 [https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-015-0180-4] [CrossRef] [Medline]
- Bogza LM, Patry-Lebeau C, Farmanova E, Witteman HO, Elliott J, Stolee P, et al. User-centered design and evaluation of a web-based decision aid for older adults living with mild cognitive impairment and their health care providers: mixed methods study. J Med Internet Res 2020 Aug 19;22(8):e17406 [https://www.jmir.org/2020/8/e17406/] [CrossRef] [Medline]
- Burns SP, Pickens ND. Embedding technology into inter-professional best practices in home safety evaluation. Disabil Rehabil Assist Technol 2017 Aug;12(6):585-591 [CrossRef] [Medline]
- Canally C, Doherty S, Doran DM, Goubran RA. Using integrated bio-physiotherapy informatics in home health-care settings: a qualitative analysis of a point-of-care decision support system. Health Informatics J 2015 Jun;21(2):149-158 [https://journals.sagepub.com/doi/10.1177/1460458213511346?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub 0pubmed] [CrossRef] [Medline]
- Chrimes D, Kitos NR, Kushniruk A, Mann DM. Usability testing of avoiding diabetes thru action plan targeting (ADAPT) decision support for integrating care-based counseling of pre-diabetes in an electronic health record. Int J Med Inform 2014 Sep;83(9):636-647 [https://europepmc.org/abstract/MED/24981988] [CrossRef] [Medline]
- Cox CE, Wysham NG, Walton B, Jones D, Cass B, Tobin M, et al. Development and usability testing of a web-based decision aid for families of patients receiving prolonged mechanical ventilation. Ann Intensive Care 2015 Mar 25;5:6 [https://europepmc.org/abstract/MED/25852965] [CrossRef] [Medline]
- Cuypers M, Lamers RE, Kil PJ, The R, Karssen K, van de Poll-Franse LV, et al. A global, incremental development method for a web-based prostate cancer treatment decision aid and usability testing in a Dutch clinical setting. Health Informatics J 2019 Sep;25(3):701-714 [https://journals.sagepub.com/doi/abs/10.1177/1460458217720393?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub 0pubmed] [CrossRef] [Medline]
- De Vito Dabbs A, Myers BA, Mc Curry KR, Dunbar-Jacob J, Hawkins RP, Begey A, et al. User-centered design and interactive health technologies for patients. Comput Inform Nurs 2009 May;27(3):175-183 [https://europepmc.org/abstract/MED/19411947] [CrossRef] [Medline]
- Danial-Saad A, Kuflik T, Weiss PL, Schreuer N. Usability of clinical decision support system as a facilitator for learning the assistive technology adaptation process. Disabil Rehabil Assist Technol 2016;11(3):188-194 [CrossRef] [Medline]
- Fleisher L, Buzaglo J, Collins M, Millard J, Miller SM, Egleston BL, et al. Using health communication best practices to develop a web-based provider-patient communication aid: the CONNECT study. Patient Educ Couns 2008 Jun;71(3):378-387 [https://europepmc.org/abstract/MED/18417312] [CrossRef] [Medline]
- Flynn AJ, Friedman CP, Boisvert P, Landis-Lewis Z, Lagoze C. The knowledge object reference ontology (KORO): a formalism to support management and sharing of computable biomedical knowledge for learning health systems. Learn Health Syst 2018 Apr 16;2(2):e10054 [https://europepmc.org/abstract/MED/31245583] [CrossRef] [Medline]
- Fu H, Rizvi RF, Wyman JF, Adam TJ. Usability evaluation of four top-rated commercially available diabetes apps for adults with type 2 diabetes. Comput Inform Nurs 2020 Jun;38(6):274-280 [https://europepmc.org/abstract/MED/31904594] [CrossRef] [Medline]
- Goud R, Jaspers MW, Hasman A, Peek N. Subjective usability of the CARDSS guideline-based decision support system. Stud Health Technol Inform 2008;136:193-198 [Medline]
- Grim K, Rosenberg D, Svedberg P, Schön UK. Development and usability testing of a web-based decision support for users and health professionals in psychiatric services. Psychiatr Rehabil J 2017 Sep;40(3):293-302 [CrossRef] [Medline]
- Holch P, Warrington L, Bamforth LC, Keding A, Ziegler LE, Absolom K, et al. Development of an integrated electronic platform for patient self-report and management of adverse events during cancer treatment. Ann Oncol 2017 Sep 01;28(9):2305-2311 [https://linkinghub.elsevier.com/retrieve/pii/S0923-7534(19)35259-7] [CrossRef] [Medline]
- Jameie S, Haybar H, Aslani A, Saadat M. Development and usability evaluation of web-based telerehabilitation platform for patients after myocardial infarction. Stud Health Technol Inform 2019;261:68-74 [Medline]
- Jessop AB, Bass SB, Brajuha J, Alhajji M, Burke M, Gashat MT, et al. "Take charge, get cured": pilot testing a targeted mHealth treatment decision support tool for methadone patients with hepatitis C virus for acceptability and promise of efficacy. J Subst Abuse Treat 2020 Feb;109:23-33 [CrossRef] [Medline]
- Kallen MA, Yang D, Haas N. A technical solution to improving palliative and hospice care. Support Care Cancer 2012 Jan;20(1):167-174 [CrossRef] [Medline]
- Li LC, Adam PM, Townsend AF, Lacaille D, Yousefi C, Stacey D, et al. Usability testing of ANSWER: a web-based methotrexate decision aid for patients with rheumatoid arthritis. BMC Med Inform Decis Mak 2013 Dec 01;13:131 [https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/1472-6947-13-131] [CrossRef] [Medline]
- Murphy C, de Laine C, Macaulay M, Fader M. Development and randomised controlled trial of a continence product patient decision aid for men postradical prostatectomy. J Clin Nurs 2020 Jul;29(13-14):2251-2259 [CrossRef] [Medline]
- Rochette A, Korner-Bitensky N, Tremblay V, Kloda L. Stroke rehabilitation information for clients and families: assessing the quality of the strokengine-family website. Disabil Rehabil 2008;30(19):1506-1512 [CrossRef] [Medline]
- Schön UK, Grim K, Wallin L, Rosenberg D, Svedberg P. Psychiatric service staff perceptions of implementing a shared decision-making tool: a process evaluation study. Int J Qual Stud Health Well-being 2018 Dec;13(1):1421352 [https://europepmc.org/abstract/MED/29405889] [CrossRef] [Medline]
- Setiawan IM, Zhou L, Alfikri Z, Saptono A, Fairman AD, Dicianno BE, et al. An adaptive mobile health system to support self-management for persons with chronic conditions and disabilities: usability and feasibility studies. JMIR Form Res 2019 Apr 25;3(2):e12982 [https://formative.jmir.org/2019/2/e12982/] [CrossRef] [Medline]
- Snyder CF, Jensen R, Courtin SO, Wu AW, Website for Outpatient QOL Assessment Research Network. PatientViewpoint: a website for patient-reported outcomes assessment. Qual Life Res 2009 Sep;18(7):793-800 [https://europepmc.org/abstract/MED/19544089] [CrossRef] [Medline]
- Span M, Hettinga M, Groen-van de Ven L, Jukema J, Janssen R, Vernooij-Dassen M, et al. Involving people with dementia in developing an interactive web tool for shared decision-making: experiences with a participatory design approach. Disabil Rehabil 2018 Jun;40(12):1410-1420 [CrossRef] [Medline]
- Span M, Smits C, Jukema J, Groen-van de Ven L, Janssen R, Vernooij-Dassen M, et al. An interactive web tool to facilitate shared decision-making in dementia: design issues perceived by caregivers and patients. Int J Adv Life Sci 2014;6(3-4):107-121 [https://research.vumc.nl/ws/portalfiles/portal/695378/chapter+4+determining+design.pdf] [CrossRef]
- Støme LN, Pripp AH, Kværner JS, Kvaerner KJ. Acceptability, usability and utility of a personalised application in promoting behavioural change in patients with osteoarthritis: a feasibility study in Norway. BMJ Open 2019 Jan 28;9(1):e021608 [https://bmjopen.bmj.com/lookup/pmidlookup?view=long&pmid=30696666] [CrossRef] [Medline]
- Tony M, Wagner M, Khoury H, Rindress D, Papastavros T, Oh P, et al. Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Serv Res 2011 Nov 30;11:329 [https://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-11-329] [CrossRef] [Medline]
- Toth-Pal E, Wårdh I, Strender LE, Nilsson G. Implementing a clinical decision-support system in practice: a qualitative analysis of influencing attitudes and characteristics among general practitioners. Inform Health Soc Care 2008 Mar;33(1):39-54 [CrossRef] [Medline]
- Tsai YL, Huang JJ, Pu SW, Chen HP, Hsu SC, Chang JY, et al. Usability assessment of a cable-driven exoskeletal robot for hand rehabilitation. Front Neurorobot 2019 Feb 13;13:3 [https://europepmc.org/abstract/MED/30814945] [CrossRef] [Medline]
- van Maurik IS, Visser LN, Pel-Littel RE, van Buchem MM, Zwan MD, Kunneman M, et al. Development and usability of ADappt: web-based tool to support clinicians, patients, and caregivers in the diagnosis of mild cognitive impairment and Alzheimer disease. JMIR Form Res 2019 Jul 08;3(3):e13417 [https://formative.jmir.org/2019/3/e13417/] [CrossRef] [Medline]
- Welch G, Balder A, Zagarins S. Telehealth program for type 2 diabetes: usability, satisfaction, and clinical usefulness in an urban community health center. Telemed J E Health 2015 May;21(5):395-403 [CrossRef] [Medline]
- Williams PA, Furberg RD, Bagwell JE, LaBresh KA. Usability testing and adaptation of the pediatric cardiovascular risk reduction clinical decision support tool. JMIR Hum Factors 2016 Jun 21;3(1):e17 [https://humanfactors.jmir.org/2016/1/e17/] [CrossRef] [Medline]
- Zafeiridi P, Paulson K, Dunn R, Wolverson E, White C, Thorpe JA, et al. A web-based platform for people with memory problems and their caregivers (CAREGIVERSPRO-MMD): mixed-methods evaluation of usability. JMIR Form Res 2018 Mar 12;2(1):e4 [https://formative.jmir.org/2018/1/e4/] [CrossRef] [Medline]
- Zheng H, Tulu B, Choi W, Franklin P. Using mHealth app to support treatment decision-making for knee arthritis: patient perspective. EGEMS (Wash DC) 2017 Apr 20;5(2):7 [https://europepmc.org/abstract/MED/29930969] [CrossRef] [Medline]
- Feldman-Stewart D, Brundage MD, Nickel JC, MacKillop WJ. The information required by patients with early-stage prostate cancer in choosing their treatment. BJU Int 2001 Feb;87(3):218-223 [CrossRef] [Medline]
- Feldman-Stewart D, Brundage MD, Van Manen L, Svenson O. Patient-focussed decision-making in early-stage prostate cancer: insights from a cognitively based decision aid. Health Expect 2004 Jun;7(2):126-141 [https://europepmc.org/abstract/MED/15117387] [CrossRef] [Medline]
- Feldman-Stewart D, Brennenstuhl S, Brundage MD, Roques T. An explicit values clarification task: development and validation. Patient Educ Couns 2006 Nov;63(3):350-356 [CrossRef] [Medline]
- Feldman-Stewart D, Brundage MD, Zotov V. Further insight into the perception of quantitative information: judgments of gist in treatment decisions. Med Decis Making 2007 Jan;27(1):34-43 [CrossRef] [Medline]
- Feldman-Stewart D, Tong C, Siemens R, Alibhai S, Pickles T, Robinson J, et al. The impact of explicit values clarification exercises in a patient decision aid emerges after the decision is actually made: evidence from a randomized controlled trial. Med Decis Making 2012 Jul;32(4):616-626 [CrossRef] [Medline]
- van Osch M, Rövekamp A, Bergman-Agteres SN, Wijsman LW, Ooms SJ, Mooijaart SP, et al. User preferences and usability of iVitality: optimizing an innovative online research platform for home-based health monitoring. Patient Prefer Adherence 2015 Jun 30;9:857-867 [https://europepmc.org/abstract/MED/26170635] [CrossRef] [Medline]
- Lilholt PH, Jensen MH, Hejlesen OK. Heuristic evaluation of a telehealth system from the Danish TeleCare north trial. Int J Med Inform 2015 May;84(5):319-326 [CrossRef] [Medline]
- Paz F, Paz FA, Villanueva D, Pow-Sang JA. Heuristic evaluation as a complement to usability testing: a case study in web domain. In: Proceedings of the 12th International Conference on Information Technology-New Generations. 2015 Presented at: ITNG '15; April 13-15, 2015; Las Vegas, NV, USA p. 546-551 URL: https://ieeexplore.ieee.org/document/7113530 [CrossRef]
- Matera M, Costabile MF, Garzotto F, Paolini P. SUE inspection: an effective method for systematic usability evaluation of hypermedia. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 2002 Jan;32(1):93-103 [https://ieeexplore.ieee.org/document/995532] [CrossRef]
- Sagar K, Saha A. A systematic review of software usability studies. Int J Inf Technol 2017:1-24 [https://link.springer.com/article/10.1007/s41870-017-0048-1] [CrossRef]
- Al-Saadi TA, Aljarrah TM, Alhashemi AM, Hussain A. A systematic review of usability challenges and testing in mobile health. Int J Account Financ Report 2015 Jul 12;5(2):1 [https://www.macrothink.org/journal/index.php/ijafr/article/view/8004/6547] [CrossRef]
- Maramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform 2019 Jun;126:95-104 [CrossRef] [Medline]
- Maia CL, Furtado ES. A systematic review about user experience evaluation. In: Proceedings of the 5th International Conference on Design, User Experience, and Usability: Design Thinking and Methods. 2016 Presented at: DUXU '16; July 17–22, 2016; Toronto, Canada p. 445-455 URL: https://link.springer.com/chapter/10.1007/978-3-319-40409-7_42 [CrossRef]
- Insfran E, Fernandez A. A systematic review of usability evaluation in web development. In: Proceedings of the 2008 International Workshops on Web Information Systems Engineering. 2008 Presented at: WISE '08; September 1-4, 2008; Auckland, New Zealand p. 81-91 URL: https://link.springer.com/chapter/10.1007/978-3-540-85200-1_10 [CrossRef]
- Cook EJ, Randhawa G, Guppy A, Sharp C, Barton G, Bateman A, et al. Exploring factors that impact the decision to use assistive telecare: perspectives of family care-givers of older people in the United Kingdom. Ageing Soc 2018;38(9):1912-1932 [https://www.cambridge.org/core/journals/ageing-and-society/article/abs/exploring-factors-that-impact-the-decision-to-use-assistive-telecare-perspectives-of-family-caregivers-of-older-people-in-the-united-kingdom/43BA74B852CD71A1964936D57B6B16CD] [CrossRef]
- Martínez García MA, Fernández Rosales MS, López Domínguez E, Hernández Velázquez Y, Domínguez Isidro S. Telemonitoring system for patients with chronic kidney disease undergoing peritoneal dialysis: usability assessment based on a case study. PLoS One 2018 Nov 06;13(11):e0206600 [https://dx.plos.org/10.1371/journal.pone.0206600] [CrossRef] [Medline]
- Saeed N, Manzoor M, Khosravi P. An exploration of usability issues in telecare monitoring systems and possible solutions: a systematic literature review. Disabil Rehabil Assist Technol 2020 Apr;15(3):271-281 [CrossRef] [Medline]
- Sánchez-Morillo D, Crespo M, León A, Crespo Foix LF. A novel multimodal tool for telemonitoring patients with COPD. Inform Health Soc Care 2015 Jan;40(1):1-22 [CrossRef] [Medline]
- Klack L, Ziefle M, Wilkowska W, Kluge J. Telemedical versus conventional heart patient monitoring: a survey study with German physicians. Int J Technol Assess Health Care 2013 Oct;29(4):378-383 [CrossRef] [Medline]
- Prescher S, Bourke AK, Koehler F, Martins A, Ferreira HS, Sousa TB, et al. Ubiquitous ambient assisted living solution to promote safer independent living in older adults suffering from co-morbidity. In: Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2012 Presented at: EMBC' 12; August 28 to September 1, 2012; San Diego, CA, USA p. 5118-5121 URL: https://ieeexplore.ieee.org/document/6347145 [CrossRef]
- Daudt HM, van Mossel C, Scott SJ. Enhancing the scoping study methodology: a large, inter-professional team's experience with Arksey and O'Malley's framework. BMC Med Res Methodol 2013 Mar 23;13:48 [https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-13-48] [CrossRef] [Medline]
|ISO: International Organization for Standardization|
|PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews|
|SDM: shared decision-making|
Edited by T Leung; submitted 26.07.22; peer-reviewed by W van Harten, Y Zisman-Ilani; comments to author 20.12.22; revised version received 27.02.23; accepted 31.03.23; published 15.08.23Copyright
©Rehab Alhasani, Nicole George, Dennis Radman, Claudine Auger, Sara Ahmed. Originally published in JMIR Rehabilitation and Assistive Technology (https://rehab.jmir.org), 15.08.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Rehabilitation and Assistive Technology, is properly cited. The complete bibliographic information, a link to the original publication on https://rehab.jmir.org/, as well as this copyright and license information must be included.