The initiative to develop an app was taken by the head of shoulder surgery at the University Hospital Salzburg (Salzburger Landeskliniken, Universitätsklinikum Salzburg, SALK), Department of Orthopedics and Trauma Surgery of the Paracelsus Medical University, Salzburg, which is a level one trauma center. The app was developed and tested at the Department of Multimedia Technology at the Salzburg University of Applied Sciences (SUAS). The study was conducted at the educational facility of the Department of Physiotherapy of the SUAS, which is located at the main facility of the SALK.

The app for frozen shoulder patients was developed in a co-creation process, which included a training mode with detailed instructions on exercise conduct, an exercise calendar, and a mobile phone sensor–based mobility measurement (see Multimedia Appendix 1). The exercises were demonstrated by means of a three-dimensional (3D) avatar, which performed the exercises as intended.

The Unity3D game engine was used to implement the app. The exercises were first recorded with a 3D capturing system (OptiTrack) and on video. The OptiTrack recordings and videos were used by a 3D modeler to create accurate animations of the exercises. Several interface concepts were tested and evaluated by the authors and their colleagues (see Acknowledgments). App development was an iterative process of analysis, conceptualizing, and prototyping in a focus group. This prototype was evaluated in a focus group consisting of 8 potential patients typical for the target group and 5 physiotherapists. The final prototype for the pilot study is explained in detail in the following sections.

The main screen of the frozen shoulder app for the patients in the pilot study had 4 buttons to access four functions (see Figure 1):

The training mode included instructions for four exercises (selected by a team of physiotherapists and the physician), which are shown in Figure 2. In the first exercise, the shoulders are moved up and down (see first screen of Figure 2). In the second exercise, the affected arm is mobilized on a table (see second screen of Figure 2). For the third exercise, the patient is lying down and laterally moving the affected arm, while the other arm is used for support (see third screen of Figure 2). The fourth exercise involves the use of doorframe for external rotational stretching (see fourth screen of Figure 2). The app recommends three sets with 20 repetitions for each exercise.

Mobility assessment is useful for monitoring the progress of the effect of the treatment of frozen shoulder. For mobility assessment, two options were implemented, which can be freely chosen by the patient for each mobility assessment (see Figure 3). One mode employs manual input of the range of motion with a slider, whereas the other employs the built-in sensors of modern mobile phones. Mobility is assessed in four ways: lateral arm lift, frontal arm lift, lateral external rotation, and back rotation/scratch (see Figure 4). For sensor-based measurements, the patient uses a wrist band to attach the mobile phone to the upper arm for the lateral and frontal arm lift and on the forearm to the lateral external and back rotation. Then, the user presses the “measurement” button in an arbitrary position (see Figure 5). The patient moves the arm in a neutral (hanging) position. After 3 seconds, the measurement starts (as indicated by an audible beep) and the patient moves the arm as far as possible without any pain in the measured plane. The measurement is automatically stopped if the user moves back to the initial position. The maximum angle to the neutral position is automatically computed without any user input. After reaching the maximum position, the patient can move his or her arm into any comfortable position and examine the measurement, which is also illustrated on the avatar (see Figure 6). The patient can always repeat the measurement by pressing the “retry” button. By pressing the “ok” button, the measurement is saved. The recommendation was to conduct mobility measurement once a week. The overview screen shows a monthly calendar with a progress overview. A smiley on a day indicates that the training was carried out. Measurement results are visualized as bar charts in percent of maximum possible range of motion. Figure 7 shows the results of an overview screen of a patient included in the pilot study.

The information screen gives a brief definition of frozen shoulder and mentions the common treatment options, pain medication, and mobilization exercises. Furthermore, the most important functions of the app are briefly explained and contact information for the physician who supervised the study is given.

For the usability and acceptance evaluation of the app, several standardized questionnaires were employed that included selected parts (intention to use, perceived usefulness, perceived ease of use) of the revised Technology Acceptance Model (TAM-2) [22,23], the System Usability Scale (SUS) [24], and the USE (Usability, Satisfaction, and Ease of use) questionnaire, which were employed in a previous study [25]. For the interpretation of SUS scores, refer to the study by Bangor et al (2008) [26].

The app automatically collected usage data, namely the time and date when the app was started and ended, the time and date and interaction type with the app (button push), and the results of the mobility measurement.

The duration of a single training set (20 repetitions of single exercise) were computed on the basis of these data.

Furthermore, each patient was asked what they liked and what they disliked about the app. Questions on technical aspects of user interactions were asked as well, that is, whether they viewed the exercise from different angles and distances, whether they read the instructions, and whether they listened to the audio instructions. These questions were contrived by the human-computer interaction (HCI) expert and tested for understandability by the other authors.

Questions on further improvements in the overall conduct of the study and whether the initial personal instructions about how to use the app were necessary were asked.

In the second meeting, the physiotherapist reassessed the correctness of the exercises. The patients performed the conducted exercises under supervision of a single physiotherapist, and correctness was rated on a scale of 1 to 5:

The perceived pain on the first meeting (introduction of the exercises and the app) and the second meeting (interviews and evaluations) was recorded. The pain was recorded on a numeric rating scale (NRS), where 0 indicated no pain and 10 indicated the highest level of pain. Minimum pain levels (Did you experience even pain free episodes in the last days?), maximum pain (What was the worst pain you had in the last days?), and current pain levels at the time of the interview were assessed (What is your level of pain right now?). The occurrence of nightly pain was recorded as well.

In addition to the mobility assessment in the app, the ability to perform two movement tasks was assessed qualitatively by one physiotherapist at the start and at the end of the study:

The physiotherapist explained and demonstrated the movement and recorded the ability of the patient to perform the task (“Able,” “Hardly able,” “Unable”).

The results of the usability questionnaires are summarized in Table 2. TAM-2 answers were given on 5-point Likert scale, from 1 (negative/disagree) to 5 (positive/strongly agree). The TAM-2 results are summarized in Table 2. The users (5 patients and the partner of 1 patient) showed strong intention to further use the app (4.2 on an average); only one patient reported that she/he did not like regular usage of mobile phones at all and she/he would not like to use such apps. The users considered the app useful. The average score for perceived usefulness was 3.9. The users considered the app easy to use. The average score for perceived ease of use was 4.3.

The questions of the USE questionnaire were rated on a 5-level Likert scale as well. The users considered the app easy to learn. The average score for ease of learning was 4.2. The users were satisfied with the app. The average score for satisfaction was 4.7.

The app achieved an average SUS score of 88 (on a 0 to 100 scale), which indicates a very usable system [26].

All patients reported that they used the app. The patient statements were verified by the log files of the app; the overview screens of the patients are shown in Figures 8-12. A green smiley refers to a training session. A blue bar plot represents the result of a mobility assessment (the higher the bar, the more mobile the patient’s shoulder joint). The patients performed the training on every day of the study (green smileys), but one patient started later (see Figure 11). All patients except one assessed their mobility at least once a week during the study (a bar plot in the Calendar represents a mobility assessment).

In the further analysis of the quantitative usage data, the first day (instruction day) and the last day (end of study) were excluded in order to omit the instruction and reporting usage cases of the app. Especially interesting is a closer investigation of the mobility measurements with the app. Figure 13 shows all measurement results per patient; each circle visualizes one distinct mobility measurement. We excluded one patient (PID 02), who did not record any mobility measurements after the first meeting. One patient (PID 03) repeated the mobility measurements multiple times until she/he was satisfied. Overall, 139 single mobility assessments were successfully completed, 32 mobility assessments were interrupted (eg, by pressing the “home” button, an incoming call), and for 21 mobility assessments, the slider was not touched at all.

Another interesting question is how the training mode of the app was used. Namely, did the patients just quickly mark the exercises as done, or did they use the training mode to guide them through the exercises?

The avatar executes a single repetition of an exercise within 3.5 seconds, that is, 70 seconds for a set of 20 repetitions, and the time for the relaxation phase between sets was not specified. The minimum plausible time for a set when using the app during exercising was 20 seconds, as a single repetition of one exercise requires at least one second based on practical tests by the research team. The maximum plausible time for using the app during exercising was set to 200 seconds, that is, about 3 minutes for 20 repetitions and a relaxation phase.

Our analysis shows that for more than half of the time (624 out of 1145), the patients used the app during training and did not just tick off the exercises. Figure 14 shows a histogram of the duration of a single set of an exercise. Many durations of a single set are close to zero; in these cases, the patients used the app often just to tick off exercises. A smaller peak at 125 seconds can be observed, which corresponds to the recommended set time (70 seconds) and less than a minute of relaxation between the sets.

The patients were instructed to conduct the four exercises with three sets each on a daily basis (4 patients for 20 study days and 1 patient for 9 study days), that is, for perfect compliance 1068 exercise sets were expected. As 1260 exercise sets were recorded, training compliance was excellent. Of these 1260 exercise sets, 78 sets had durations of above 200 seconds and were therefore excluded. 37 exercise sets were interrupted (eg, by pressing the “home” button, turning off the phone, or receiving a call) and were therefore excluded as well.

Four hundred and sixty sets had durations shorter than 7 seconds, that is, in these cases it was concluded that the app was only used to mark the exercises as completed. In addition, 61 set-durations were too long for just checking the exercise sets as done, and too short to properly conduct the exercise set. An explanation could be that the patients showed the exercises to someone.

Figure 15 shows the set duration per patient and day of study. One patient (PID 05) stopped to use the app during training, and started to use the app only for confirmation after a week.

Figure 16 shows the usage patterns of the app over time. Green bars illustrate the sets that have likely been completed using the app during the exercise (set durations between 20 and 200 seconds), whereas blue bars illustrate the percentage of sets that used the app just to tick off the exercises (confirmation, set durations below 7 seconds). Gray bars (label “unknown”) refer to set durations above 7 seconds and below 19 seconds. Overall, compliance stayed high during the study. For perfect compliance, each patient had to perform 12 exercise sets per day, that is, 48 exercises for the 4 patients of the first 11 days of the study and 60 exercise sets for the 5 patients for the rest of the study.

In the following, the feedback to each question about technical aspects of the app is summarized. In order to preserve the privacy of the patients, all information is given as generally as possible.

All the patients reported that no improvements of the exercise instructions were necessary. All of them thought that they conducted the exercises correctly (4, 4, 5, 5, 5; with 1 having no memory how to perform the exercise and 5 being totally correct). Four of the 5 patients could participate in the second meeting in person; one patient was ill and was interviewed by telephone. Thus, the correct conduct was only assessed for 4 patients. The assessment of the physiotherapist confirmed the correctness of the conduct of the exercises. Only minor differences to the optimal exercise conduct were present (see Table 3 for detailed comments).

The results for the grades of pain are summarized in Table 4. Decreased pain levels are colored in green; increased pain levels are colored in red and with a horizontal stripe pattern. Minimum pain levels (NRS min in Table 4) increased slightly for Patient PID 02 (from 0 to 1.5). Patient PID 05 had a decrease in minimum pain from 3.5 to 2. Maximum pain levels (NRS max in Table 4) were reduced in 4 patients (decreased by 2.5, 0.5, 1 and 0.5, respectively) and increased by 0.5 in patient PID 05. All patients reported reduced current pain levels. Pain during the night remained constant for all patients (three were affected by nightly pain, two did not).

Additionally, two movement tasks were tested, namely moving the hand to the neck and moving the hand to the lower back (Table 5). For one patient, an improvement for the first movement was recorded (from hardly possible to possible), and for another patient, an improvement for the second movement was noticeable (from not possible to hardly possible).

Professional tracking systems are capable of precisely tracking patient motion during exercises and use these data to provide feedback. A Vicon tracking system was used to implement a prototype for physiotherapy at home [44,38].

Virtual reality (VR) HMDs offer the efficient simulation of training environments. VR systems were used to simulate situations of everyday life (eg, a virtual kitchen) where patients with cognitive disabilities could relearn daily living skills [45]. VR exer-game, in which the user controls the avatar movement with an ergometer, was proposed as well [46]. However, as compared with a mobile phone app, a VR system is not as suitable for home exercising and wide deployment, as it requires expensive hardware to be installed at the home of the patient.

Augmented reality (AR) systems with HMDs (such as the Microsoft HoloLens) allow to graphically overlay the visual perception with additional information, which would be well-suited to provide patients with feedback on exercise performance. The design of AR games for upper extremity motor dysfunctions was investigated [47] and in a follow-up study, an AR game for an HMD system was evaluated [39]. However, as compared with a mobile phone, AR HMDs are expensive and not widely available at the moment.

Off-the-shelf game console hardware has been proposed to support physiotherapy. The accuracy of Microsoft’s Kinect body tracking for rehabilitation purposes was quantitatively assessed [48]. Kinect-based systems for physiotherapy have been proposed [40,49]. A Kinect-based system for shoulder impingement therapy was presented as well [36]. The Nintendo Wii system includes a game controller that allows pointing at screen positions and contains an accelerometer. Rehabilitation of cerebral palsy with a system running on the Nintendo Wii was investigated [33]. Off-the-shelf Nintendo Wii Fit games were employed and evaluated with respect to the retention of motor skills of patients with Parkinson disease [32].

However, as compared with mobile phones, even gaming consoles are not as widely deployed, especially for individuals in the age group of 40 to70 years. Furthermore, the small movements of the exercises for frozen shoulder are hard to track with off-the-shelf hardware. Even recordings of our exercises with a professional motion capturing system (OptiTrack) required manual corrections by a 3D animator.

Accelerometers and gyroscopes, that is, inertial measurement units (IMUs), have been widely used in previous studies on technology-assisted rehabilitation. An IMU-sensor–based system to deliver balance and strength exercises to the elderly was proposed [50]. Knee rehabilitation supported by IMUs was proposed [35,51]. A cap with an IMU (Sense-Cap) to monitor balance exercises was proposed and evaluated [30]. A more complex IMU-based system to provide motion guidance was also proposed [52]. Compared with our system, additional hardware (IMUs) needs to be distributed to the patients.

Mobile phone apps for general health and fitness have moved from research to practice. The application of mobile phone apps in medical and rehabilitation contexts is currently heavily researched.

Early studies [41,53] proposed a context-aware and user-adaptive mobile system for fitness training. A 3D avatar was used as a mobile trainer and to show the exercises. It was pointed out that the use of a 3D avatar allowed the user to perform the exercises more accurately.

The use of conversational interfaces for health and fitness companions was discussed [54]. User-tailored activity coaching systems were reviewed [55]. Mobile phone apps were investigated for physiotherapy [43]. A reminder app for stroke patients was proposed [56]. A mobile phone app to encourage activity in patients with chronic obstructive pulmonary disease was evaluated [25].

There are a large number of commercial fitness and training apps. In these apps, exercises are presented using animated videos (no view point change is possible). None of the commercially available mobile phone applications use an interactive 3D avatar, which our system offers.

Physiotherapy over video communication was discussed and evaluated [57]. It was highlighted that information of bodily cues is limited in two-dimensional videos.

Compared with most of the previous contributions from academia, which have mainly focused on special not widely available hardware (especially in the age group of 40 to 70), our proposal only requires a standard mobile phone.

Previous studies show, that new technology is hardly accepted by many elderly patients [58] and especially, hardware that has to be installed at home is problematic [31].

Although our app is not the first app to target MSDs, it is the first that specifically tackles frozen shoulder and presents an evaluation on the basis of a pilot study.