This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Rehabilitation and Assistive Technology, is properly cited. The complete bibliographic information, a link to the original publication on http://rehab.jmir.org/, as well as this copyright and license information must be included.
The “Strengthen Your Ankle” neuromuscular training program has been thoroughly studied over the past 8 years. This process evaluation is a part of a randomized controlled trial that examined both the short- and long-term effectiveness of this particular program. Although it was shown previously that the program, available both in a printed booklet and as a mobile app, is able to effectively reduce the number of recurrent ankle sprains, participants’ compliance with the program is an ongoing challenge.
This process evaluation explored participants’ opinions regarding both the methods of delivery, using RE-AIM (Reach Effectiveness Adoption Implementation Maintenance) Framework to identify barriers and challenges to program compliance. Although Reach, Effectiveness, and Adaptation were the focus of a previous study, this paper focuses on the implementation and maintenance phases.
Semistructured interviews and online questionnaires were analyzed using qualitative content analysis. Fisher exact, chi-square, and
While there were no significant differences in the perceived simplicity, usefulness, and liking of the exercise during the 8 weeks of the neuromuscular training program, semistructured interviews showed that 14 of 16 participants agreed that an app would be of additional benefits over a booklet. After the 12-month follow-up, when asked how they evaluated the overall use of the app or the booklet, the users of the app gave a mean score of 7.7 (SD 0.99) versus a mean score 7.1 (SD 1.23) for the users of the booklet. This difference in mean score was significant (
Although both the app and booklet showed a high user satisfaction, the users of the app were significantly more satisfied. Semistructured questionnaires allowed users to address issues they would like to improve in future updates. Including a possibility for feedback and postponement of exercises, an explanation of the use of specific exercises and possibly music were identified as features that might further improve the contentment of the program, probably leading to increased compliance.
Netherlands Trial Register NTR4027; http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=4027 (Archived by Webcite at http://www.webcitation.org/70MTo9dMV)
Injuries, due to participation in sports and physical activities, are prevalent. Internationally, ankle sprains are one of the most common musculoskeletal injuries [
One of the many available interventions that has been shown to be effective in reducing the risk of recurrent ankle sprains, as well as protecting against secondary complaints, is neuromuscular training (NMT) [
As with other preventive interventions, the translation of the evidence on ankle sprain prevention through NMT to the real-world context of sports remains a challenge, by which effective ankle sprain prevention in the community is lagging [
Although developed for use in health care settings, the RE-AIM framework has been previously used to evaluate the success of introducing strategies for sports injury prevention within a practical sports context [
The full details of the “Strengthen Your Ankle” study have been described elsewhere [
The RE-AIM framework describes five dimensions to evaluate the practical feasibility of an intervention: “Reach,” “Effectiveness,” “Adoption,” “Implementation,” and “Maintenance” [
The “Effectiveness” dimension describes the clinical impact of the studied intervention. The short- and long-term effectiveness of the app compared with the booklet for preventing ankle sprain recurrences were assessed in a RCT. The full methods and results of this trial have been published elsewhere [
Implementation concerns the participants’ use of the intervention strategies. In this study, we quantified use as compliance with the 8-week NMT program in each of the study groups, measured as a percentage of the total program completed. In addition, the participants’ attitudes and perceptions toward the delivery of the NMT programs were assessed.
During the 8 weeks of the NMT program, participants received a weekly online questionnaire. The questionnaire registered what percentage of the program was executed during the week, the amount of difficulty that was experienced while conducting each of the exercises, and the reason for a possible lack of compliance. For each of the 6 different exercises, participants indicated what percentage of the exercises they performed each week. Additionally, using a 5-point Likert scale, participants were asked how they perceived the exercises. When participants failed to complete the questionnaire, reminders were sent by email. The details on the questionnaire have been published previously [
After the 8-week training period, a more extensive evaluation questionnaire was completed, including closed and free-text questions on the subjectively-experienced value of the NMT program delivery mode, a subjective evaluation of the program, and the perceived disadvantages and advantages of the allocated intervention delivery mode. To measure satisfaction, all remaining participants (75 in the app group and 88 in the booklet group) were asked to give a 0-10 score for the app or booklet. An unpaired
“Maintenance” describes the long-term effectiveness of the intervention strategies. For this study, this dimension was defined as the percentage of participants still conducting the NMT program combined with the advantages the participants perceived related to the app or paper booklet use for intervention delivery.
After 12 months, semistructured interviews were conducted with individual participants to assess the perceived advantages of using the app over the paper booklet. All study participants were asked if they were willing to participate in a semistructured interview concerning the NMT program; 27% (32/119) of the remaining participants, evenly divided over the two study groups, responded positively. The interviews were structured using a preselected topic list on the individual experiences with the NMT program either through the booklet or app. All interviews were conducted and transcribed by one researcher (MA). Interviews were conducted via telephone until saturation was reached, that is, when interviews did not lead to new themes or information, within both study groups, resulting in 16 semistructured interviews with 8 randomly selected participants in the booklet group and 8 randomly selected participants in the app group.
Due to dropout during follow-up (n=57 after 8 weeks and a further n=44 after 12 months), sample sizes differed between questionnaires. The reasons for dropout were unknown. The participants’ answers on the 5-point Likert scales regarding attitudes and perceptions toward the program, as registered during the 8-week program, were averaged for each participant over the available follow-up moments. Independent sample
All semistructured interviews were audiorecorded and transcribed verbatim. In transcriptions, any personal information or information that was deducible to an individual was anonymized. Verbatim-transcribed interviews were thematically analyzed and fragmented on the basis of topical similarity using Atlas.ti [
Previous studies that looked at the effectiveness of the “Strengthen Your Ankle” program provided further details on the (cost)-effectiveness of the program in the short and long term [
During the 12-month follow-up, there were 139 recurrent ankle injuries, resulting in injury incidence densities of 15.59 per 1000 hours of sport (95% CI 11.94 to 19.24) in the app group and 15.84 (95% 12.10 to 19.58) in the booklet group. Over the long term, this difference in injury density was not significant (HR 1.06; 0.76 to 1.49) [
The first study in this larger research project looked at compliance during the 8 weeks of the NMT intervention. It was shown that the average compliance to the exercise scheme was 73.3% (95% CI 67.7% to 78.1%) in the app group and 76.7% (95% CI 71.9 to 82.3%) in the booklet group. No significant difference in compliance was found between the groups [
The weekly questionnaires (
After the 8-week intervention period, 35 participants using the app and 22 participants using the booklet discontinued the study for unknown reasons. The remaining 75 users of the app found this method of NMT program delivery more user friendly, easier, fun to use, and less annoying and thought that the videos were more helpful than the booklet (
Participants’ attitudes and perceptions toward the allocated delivery of the NMT program during the 8-week intervention period.
Participants’ opinions and method of delivery | Mean (SD)a | Mean differenceb (95% CI) | ||
0.03 (−0.19 to 0.25) | .79 | |||
App | 3.79 (0.86) | |||
Booklet | 3.76 (0.78) | |||
−0.16 (−0.36 to 0.05) | .13 | |||
App | 2.25 (0.82) | |||
Booklet | 2.41 (0.71) | |||
0.05 (−0.16 to 0.26) | .65 | |||
App | 3.72 (0.85) | |||
Booklet | 3.67 (0.75) | |||
−0.01 (−0.25 to 0.23) | .96 | |||
App | 3.30 (0.94) | |||
Booklet | 3.30 (0.87) | |||
−0.04 (−0.22 to 0.14) | .64 | |||
App | 3.94 (0.68) | |||
Booklet | 3.98 (0.67) | |||
0.09 (−0.14 to 0.32) | .47 | |||
App | 3.42 (0.87) | |||
Booklet | 3.33 (0.88) | |||
−0.09 (−0.35 to 0.17) | .49 | |||
App | 3.29 (0.99) | |||
Booklet | 3.38 (0.97) | |||
−0.15 (−0.32 to −0.01) | .07 | |||
App | 2.00 (0.58) | |||
Booklet | 2.16 (0.67) | |||
−0.02 (−0.21 to 0.17) | .84 | |||
App | 3.87 (0.75) | |||
Booklet | 3.89 (0.66) | |||
−0.06 (−0.24 to 0.11) | .49 | |||
App | 2.34 (0.68) | |||
Booklet | 2.41 (0.64) | |||
App | 3.42 (0.88) | 0.12 (−0.11 to 0.35) | .32 | |
Booklet | 3.31 (0.84) | |||
0.07 (−0.13 to 0.26) | .50 | |||
App | 2.66 (0.77) | |||
Booklet | 2.59 (0.71) |
aScores present means (SD) of 5-point Likert scales (1=strongly agree; 5=strongly disagree).
bDifferences in scores between groups were analyzed by independent
The subjectively-experienced value of the NMT program and perceived disadvantages and advantages of the allocated intervention delivery mode assessed directly after the 8-week intervention.
Participants’ opinions and method of delivery | Mean (SD)a | Mean differenceb (95% CI) | ||
−0.43 (−0.75 to −0.11) | .009 | |||
App | 1.85 (0.98) | |||
Booklet | 2.28 (1.10) | |||
−0.40 (−0.69 to −0.11) | .008 | |||
App | 1.84 (0.92) | |||
Booklet | 2.24 (0.97) | |||
−0.06 (−0.35 to 0.23) | .68 | |||
App | 2.12 (0.90) | |||
Booklet | 2.18 (0.97 | |||
−0.29 (−0.59 to 0.01) | .06 | |||
App | 2.13 (0.95) | |||
Booklet | 2.42 (1.01) | |||
−0.29 (−0.59 to 0.01) | .06 | |||
App | 2.19 (0.95) | |||
Booklet | 2.48 (0.97) | |||
−0.29 (−0.62 to 0.03) | .07 | |||
App | 2.08 (1.03) | |||
Booklet | 2.38 (1.04) | |||
0.47 (0.12 to 0.81) | .008 | |||
App | 4.09 (1.09) | |||
Booklet | 3.63 (1.13) | |||
−0.18 (−0.48 to 0.12) | .23 | |||
App | 2.25 (0.95) | |||
Booklet | 2.44 (0.97) | |||
−0.99 (−1.31 to −0.68) | < .001 | |||
App | 1.96 (1.07) | |||
Booklet | 2.95 (0.96) | |||
−0.07 (−0.35 to 0.21) | .64 | |||
App | 2.08 (0.98) | |||
Booklet | 2.15 (0.84) | |||
0.08 (−0.23 to 0.38) | .62 | |||
App | 2.12 (1.10) | |||
Booklet | 2.05 (0.87) | |||
−0.05 (−0.36 to 0.26) | .73 | |||
App | 3.48 (1.03) | |||
Booklet | 3.53 (0.97) | |||
−0.36 (−0.65 to −0.07) | .02 | |||
App | 2.09 (0.94) | |||
Booklet | 2.45 (0.95) | |||
−0.37 (−0.66 to −0.08) | .01 | |||
App | 2.68 (0.94) | |||
Booklet | 3.06 (0.93) | |||
−0.14 (−0.39 to 0.11) | 0.26 | |||
App | 2.20 (0.74) | |||
Booklet | 2.34 (0.84) | |||
−0.17 (−0.42 to 0.09) | 0.13 | |||
App | 2.23 (0.84) | |||
Booklet | 2.40 (0.870) | |||
−0.22 (−0.52 to 0.10) | 0.17 | |||
App | 2.26 (1.07) | |||
Booklet | 2.47 (0.91) |
aScores present means (SD) of 5-point Likert scales (1=strongly agree; 5=strongly disagree).
bDifferences in scores between groups were analyzed through independent
Therefore, the answers of 53 of the booklet users were “neutral” when asked if the online videos were of help; this was in comparison with 5% (4/75) in the app group. Some participants failed to answer all the questions, the number of missing responses can be found in
At the end of the 12-month follow-up period, an additional 44 participants discontinued the study. These participants were asked if they were still doing (part of the) NMT program. Only 23% (28/122) of all participants still in the study responded affirmatively. We did not ask what amount of the program they were still doing.
Two main themes arose from the semistructured interviews that related to the design of the app and possible additional benefits of the app. Fourteen out of 16 participants stated that an app would provide an additional benefit compared with a booklet. The main reasons given were that most of the participants always had their mobile phones with them and that the app provided visual support and had a reminder function. The two participants who did not feel that the app offered any benefit found the exercises too easy, which made the app redundant.
Errors in navigation and explanation, the lack of feedback and music, and lack of explanation of the purpose of the exercises were the main disadvantages experienced by the app users. The greatest perceived disadvantages of the booklet were the big size when folded out, small font, lack of robustness, and errors in explanation.
Individual responses from semistructured interviews.
Method of delivery and respondent | Added benefit of the app? | Reason given | Pros (+) and suggestions for improvement (−) for the app | |
R1 | Yes | You always have your phone with you |
+ Easy to use |
|
R2 | No | The exercises are so easy, you don’t need an app | + Videos with instructions |
|
R3 | Yes | You always have your phone with you |
+ Tick off done exercises |
|
R4 | Yes | The app gives visual support | + Easy to use |
|
R5 | Yes | You always have your phone with you | + Easy to use |
|
R6 | Yes | The app is smaller and thus easier to use | + Easier navigation | |
R7 | Yes | The app gives visual support |
+ Videos with instructions |
|
R8 | Yes | You always have your phone with you | + Videos with instructions |
|
R9 | Yes | You always have your phone with you | − Show why you need to do an exercise. |
|
R10 | No | The exercises are so easy, you don’t need an app | − Stopwatch function | |
R11 | Yes | The app gives visual support | − Show why you need to do an exercise | |
R12 | Yes | You always have your phone with you | + Reminder to do the exercise |
|
R13 | Yes | The app gives visual support | + Videos with instructions |
|
R14 | Yes | You always have your phone with you |
− Direct translation of the app to a booklet | |
R15 | Yes | You always have your phone with you | − More variation in the exercises | |
R16 | Yes | An agenda function would be easy | − Direct translation of the app to a booklet |
Previous studies [
Interventions for preventing sport injuries require high participant compliance [
Over the years, compliance with the “Strengthen Your Ankle” program in RCTs has steadily increased from 23% [
This study aimed to explore user experiences with the NMT program, as well as with the app and booklet as delivery methods, by means of semistructured interviews. The information gathered can be used to further improve the methods of delivery and, thus, increase future reach and compliance. The interviews and questionnaires showed that the app and booklet can be successfully used to prevent recurrent ankle sprains and that both show high user satisfaction. Future updates may include options for feedback or postponement of exercises, an explanation of the use of specific exercises, and possibly music; these additions could further improve user perceptions of the program and hence increase compliance.
A limitation of this study, and that of previous studies on the “Strengthen Your Ankle” program, is the mismatch between compliance and adherence. Although both constructs have been used interchangeably, they are not synonymous. Adherence refers to a situation where a clinician or researcher develops a program in cooperation with the participant. The participant attempts to follow the program as best as possible, taking personal preferences and constraints into consideration. Adherence can be seen as what happens in real-life conditions when individuals with an ankle sprain try to follow the program; compliance is studied in clinical settings. The extent to which the participant obeys the program instructions is measured by compliance rates [
A further limitation of this study is the possibility of selection bias for the semistructured interviews. It is possible that only those participants that carried a strong negative or positive view of the program agreed to participate because the invitation for the interviews was made only after termination of the 12-month follow-up. Additionally, the (single) interviewer did not structure the interviews and continued to question the participants when needed. This may have affected the validity of the data analyses. However, it is recognized that this characteristic is inherent to the flexible nature of thematic analysis and does not threaten the depth of analysis [
With the use of semistructured interviews and online questionnaires, we were able to evaluate users’ opinions on both the app and booklet. The users of the app were significantly more satisfied with the app although there was no significant difference in the perceived simplicity, usefulness, and liking of the exercise during the 8 weeks of the NMT program. In the interviews, users acknowledged the need for improvements. Future updates should take the users’ suggestions into account because adherence with the NMT program remains an ongoing challenge.
Question guide for the process evaluation using semi-structured interviews after finishing the 12-month intervention period.
Responses to process evaluation of the neuromuscular training program after the intervention period.
exercise science and sports medicine
hazard ratio
neuromuscular training
randomized controlled trial
Zilveren Kruis Achmea
This study was funded by the Netherlands Organisation for Health Research and Development (ZonMw) grant number 525001003, Balance boards were provided by Disporta and Booklets were provided by VeiligheidNL.
We would like to thank the following partners for their collaboration and recruitment of study participants: Royal Dutch Society for Physical Therapy (KNGF), Dutch Society for Physical Therapy in Sports (NVFS), Dutch College of General Practitioners (NHG), Dutch Sports Medicine Society (VSG), Dutch Olympic Committee (NOC*NSF), Zilveren Kruis Achmea (ZKA), and Disporta.
None declared.