The full details of the “Strengthen Your Ankle” study have been described elsewhere [3,4,13]. In brief, 220 sports participants who experienced an ankle sprain during the past 2 months were included in this RCT. Participants were randomly assigned to either the app or booklet intervention group and were instructed to follow the embedded 8 week “Strengthen Your Ankle” NMT prevention program using either the app or the printed booklet.

The RE-AIM framework describes five dimensions to evaluate the practical feasibility of an intervention: “Reach,” “Effectiveness,” “Adoption,” “Implementation,” and “Maintenance” [16]. The dimensions “Reach” and “Adoption” are out of scope when describing the feasibility of an intervention within a controlled trial. As such, for the this study, we focused on the dimensions “Effectiveness,” “Implementation,” and “Maintenance.”

The “Effectiveness” dimension describes the clinical impact of the studied intervention. The short- and long-term effectiveness of the app compared with the booklet for preventing ankle sprain recurrences were assessed in a RCT. The full methods and results of this trial have been published elsewhere [3,4,13]. In order to put the outcomes of the “Implementation” and “Maintenance” dimensions in context, we will briefly summarize the “Effectiveness” outcomes.

Implementation concerns the participants’ use of the intervention strategies. In this study, we quantified use as compliance with the 8-week NMT program in each of the study groups, measured as a percentage of the total program completed. In addition, the participants’ attitudes and perceptions toward the delivery of the NMT programs were assessed.

During the 8 weeks of the NMT program, participants received a weekly online questionnaire. The questionnaire registered what percentage of the program was executed during the week, the amount of difficulty that was experienced while conducting each of the exercises, and the reason for a possible lack of compliance. For each of the 6 different exercises, participants indicated what percentage of the exercises they performed each week. Additionally, using a 5-point Likert scale, participants were asked how they perceived the exercises. When participants failed to complete the questionnaire, reminders were sent by email. The details on the questionnaire have been published previously [3].

After the 8-week training period, a more extensive evaluation questionnaire was completed, including closed and free-text questions on the subjectively-experienced value of the NMT program delivery mode, a subjective evaluation of the program, and the perceived disadvantages and advantages of the allocated intervention delivery mode. To measure satisfaction, all remaining participants (75 in the app group and 88 in the booklet group) were asked to give a 0-10 score for the app or booklet. An unpaired t test was performed to examine the difference in scores between the two groups.

“Maintenance” describes the long-term effectiveness of the intervention strategies. For this study, this dimension was defined as the percentage of participants still conducting the NMT program combined with the advantages the participants perceived related to the app or paper booklet use for intervention delivery.

After 12 months, semistructured interviews were conducted with individual participants to assess the perceived advantages of using the app over the paper booklet. All study participants were asked if they were willing to participate in a semistructured interview concerning the NMT program; 27% (32/119) of the remaining participants, evenly divided over the two study groups, responded positively. The interviews were structured using a preselected topic list on the individual experiences with the NMT program either through the booklet or app. All interviews were conducted and transcribed by one researcher (MA). Interviews were conducted via telephone until saturation was reached, that is, when interviews did not lead to new themes or information, within both study groups, resulting in 16 semistructured interviews with 8 randomly selected participants in the booklet group and 8 randomly selected participants in the app group. Multimedia Appendix 1 shows the question guide for the semistructured interviews, aimed at process evaluation, after finishing the 12-month intervention.

Due to dropout during follow-up (n=57 after 8 weeks and a further n=44 after 12 months), sample sizes differed between questionnaires. The reasons for dropout were unknown. The participants’ answers on the 5-point Likert scales regarding attitudes and perceptions toward the program, as registered during the 8-week program, were averaged for each participant over the available follow-up moments. Independent sample t tests with assumed equal variances were conducted to assess for differences in the average Likert responses between the two study groups. The significance level was evaluated at P=.05. SPSS (version 22.0) and was used for all statistical analyses.

All semistructured interviews were audiorecorded and transcribed verbatim. In transcriptions, any personal information or information that was deducible to an individual was anonymized. Verbatim-transcribed interviews were thematically analyzed and fragmented on the basis of topical similarity using Atlas.ti [19]. Open, inductive coding was used line by line on the transcripts of the interviews and these codes were converged into subthemes [20]. Peer debriefing was used as an external check to the research process. This method of analysis was used after each interview and ended when no new codes arose and saturation was reached [19]. The final step in the analysis process was to submerge the subthemes to a limited number of main themes [19].

Previous studies that looked at the effectiveness of the “Strengthen Your Ankle” program provided further details on the (cost)-effectiveness of the program in the short and long term [3,4]. In short, during the 8 weeks of the NMT, there were 93 self-reported recurrent ankle sprains, which resulted in injury incidence densities of 25.3 per 1000 hours of sport (95% CI 18.0 to 32.7) in the app group and 25.6 per 1000 hours of sport (95% CI 18.3 to 32.9) in the booklet group. There was no significant difference in the incidence densities of self-reported recurrences (HR [hazard ratio] 3.07; 95% CI 0.62 to 15.20) [1].

During the 12-month follow-up, there were 139 recurrent ankle injuries, resulting in injury incidence densities of 15.59 per 1000 hours of sport (95% CI 11.94 to 19.24) in the app group and 15.84 (95% 12.10 to 19.58) in the booklet group. Over the long term, this difference in injury density was not significant (HR 1.06; 0.76 to 1.49) [4].

The first study in this larger research project looked at compliance during the 8 weeks of the NMT intervention. It was shown that the average compliance to the exercise scheme was 73.3% (95% CI 67.7% to 78.1%) in the app group and 76.7% (95% CI 71.9 to 82.3%) in the booklet group. No significant difference in compliance was found between the groups [3].

The weekly questionnaires (Table 1) showed that participants in both the app and booklet groups gave comparable scores with regard to simplicity, usefulness, and subjective evaluation of the exercises. Table 1 shows the averaged responses of the participants over the 8 weeks.

After the 8-week intervention period, 35 participants using the app and 22 participants using the booklet discontinued the study for unknown reasons. The remaining 75 users of the app found this method of NMT program delivery more user friendly, easier, fun to use, and less annoying and thought that the videos were more helpful than the booklet (Table 2). The latter question should be interpreted with caution because online videos were available for the booklet users (n=88), but many of the participants stated that they were not aware of this possibility.

Therefore, the answers of 53 of the booklet users were “neutral” when asked if the online videos were of help; this was in comparison with 5% (4/75) in the app group. Some participants failed to answer all the questions, the number of missing responses can be found in Table 2. Additional questions specifically related to possible improvements in the app, and not the booklet, (Multimedia Appendix 2) indicated that participants desired feedback after the exercises (44/75, 59%) and wanted the ability to postpone a training session (41/75, 55%). Overall, a t test showed that the users of the app were significantly more satisfied with the app (score 1 out of 10 with 10 referring to the highest score, mean±SD) compared with booklet users; 7.7 (SD 0.99) versus 7.1 (SD 1.23) P=.006.

At the end of the 12-month follow-up period, an additional 44 participants discontinued the study. These participants were asked if they were still doing (part of the) NMT program. Only 23% (28/122) of all participants still in the study responded affirmatively. We did not ask what amount of the program they were still doing.

Two main themes arose from the semistructured interviews that related to the design of the app and possible additional benefits of the app. Fourteen out of 16 participants stated that an app would provide an additional benefit compared with a booklet. The main reasons given were that most of the participants always had their mobile phones with them and that the app provided visual support and had a reminder function. The two participants who did not feel that the app offered any benefit found the exercises too easy, which made the app redundant.

Errors in navigation and explanation, the lack of feedback and music, and lack of explanation of the purpose of the exercises were the main disadvantages experienced by the app users. The greatest perceived disadvantages of the booklet were the big size when folded out, small font, lack of robustness, and errors in explanation. Table 3 shows the individual responses during the semistructured interviews to illustrate the flavor of the original data and demonstrate the prevalence of the themes, as suggested by King [21].

Previous studies [3,4] have shown that using an app or a booklet with a special NMT program to prevent recurrent ankle sprains has resulted in comparable injury densities during both short- (8 weeks) and long-term (12 months) follow-ups and comparable compliance rates with the program. During the execution of the program during the first 8 weeks, the app and booklet were given comparable scores for simplicity, usefulness, and liking of the exercises. After the 12-month follow-up, the users of the app were significantly more satisfied with the app compared with the users of the booklet. The users of the app evaluated the app as more patient friendly, easier to use, and less annoying and thought that the videos were helpful. With the help of semistructured interviews, 14 out of 16 participants agreed that an app would be of additional benefit over a booklet, mainly due to use of instructional videos, phone portability, and the agenda function. Further suggestions for improving the app that were mentioned by various participants were the ability to postpone exercises and the provision of exercise feedback.

Interventions for preventing sport injuries require high participant compliance [3] Therefore, ways to increase compliance are a focus of many intervention studies [3]. The “Strengthen Your Ankle” program was developed in 2009. Since then, the program has been studied intensively [3,4,9-11,22]. It was shown that (1) the program was effective in reducing recurrent ankle sprains for those with high compliance [10], (2) the use of either the app or a booklet produced nonsignificant differences in injury densities in both the short and long term [3,4], and (3) both methods had comparable cost-effectiveness of implementation [23].

Over the years, compliance with the “Strengthen Your Ankle” program in RCTs has steadily increased from 23% [9] to 45% [10] and 75% [3], likely as a result of annual updates, increased acknowledgment of the usefulness of the program by the target population, and improvements in the program content. However, the reach of the target population still requires substantial attention. In 2011, the annual number of downloads of the “Strengthen Your Ankle” app reached 25,781, which corresponds to a low percentage (25,781/911,576, 2.6%) of potential users [18]. Some studies have looked at the use of apps in injury prevention over the last decade. What can be concluded from those studies is that numerous apps seek to prevent (re)injury. However, the scientific evidence supporting these app-based programs is nonexistent or scarce [22,24]. A recent review found that out of 18 apps concerned with preventing sports and physical activity-related injuries, only four included evidence regarding efficacy [22]. In addition to the app that is the focus of this study, one of those four apps dealt with ankle injury prevention using NMT. No information is available on the use or compliance of the other app [22].

This study aimed to explore user experiences with the NMT program, as well as with the app and booklet as delivery methods, by means of semistructured interviews. The information gathered can be used to further improve the methods of delivery and, thus, increase future reach and compliance. The interviews and questionnaires showed that the app and booklet can be successfully used to prevent recurrent ankle sprains and that both show high user satisfaction. Future updates may include options for feedback or postponement of exercises, an explanation of the use of specific exercises, and possibly music; these additions could further improve user perceptions of the program and hence increase compliance.

A limitation of this study, and that of previous studies on the “Strengthen Your Ankle” program, is the mismatch between compliance and adherence. Although both constructs have been used interchangeably, they are not synonymous. Adherence refers to a situation where a clinician or researcher develops a program in cooperation with the participant. The participant attempts to follow the program as best as possible, taking personal preferences and constraints into consideration. Adherence can be seen as what happens in real-life conditions when individuals with an ankle sprain try to follow the program; compliance is studied in clinical settings. The extent to which the participant obeys the program instructions is measured by compliance rates [12,24]. Research, ideally performed in a more or less controlled setting, implicitly focuses on compliance, rather than on adherence. However, the “Strengthen Your Ankle” program is meant to increase adherence for all individuals at risk for an ankle sprain, not only for those who participate in the studies involved. This study has tried to explore the barriers and opportunities that participants experienced while using the training program via an app or booklet within a controlled study setting. However, because the interviews were held after follow-up, that is, months after the participants had finished the 8 weeks of the training program, we expected to gain insight as to program performance in real-life situations.

A further limitation of this study is the possibility of selection bias for the semistructured interviews. It is possible that only those participants that carried a strong negative or positive view of the program agreed to participate because the invitation for the interviews was made only after termination of the 12-month follow-up. Additionally, the (single) interviewer did not structure the interviews and continued to question the participants when needed. This may have affected the validity of the data analyses. However, it is recognized that this characteristic is inherent to the flexible nature of thematic analysis and does not threaten the depth of analysis [5].

With the use of semistructured interviews and online questionnaires, we were able to evaluate users’ opinions on both the app and booklet. The users of the app were significantly more satisfied with the app although there was no significant difference in the perceived simplicity, usefulness, and liking of the exercise during the 8 weeks of the NMT program. In the interviews, users acknowledged the need for improvements. Future updates should take the users’ suggestions into account because adherence with the NMT program remains an ongoing challenge.