This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Rehabilitation and Assistive Technology, is properly cited. The complete bibliographic information, a link to the original publication on https://rehab.jmir.org/, as well as this copyright and license information must be included.
Objective measures specifically assessing selective voluntary motor control are scarce. Therefore, we have developed an interval-scaled assessment based on accelerometers.
This study provided a preliminary evaluation of the validity and reliability of this novel gamelike assessment measuring lower limb selective voluntary motor control in children with cerebral palsy (CP).
Children with CP and their neurologically intact peers were recruited for this psychometric evaluation of the assessgame. The participants played the assessgame and steered an avatar by selective hip, knee, or ankle joint movements captured with accelerometers. The assessgame’s scores provide information about the accuracy of the selective movement of the target joint and the amplitude and frequency of involuntary movements occurring in uninvolved joints. We established discriminative validity by comparing the assessgame scores of the children with CP with those of the neurologically intact children, concurrent validity by correlations with clinical scores and therapists’ opinions, and relative and absolute test-retest reliability.
We included 20 children with CP (mean age 12 years and 5 months, SD 3 years and 4 months; Gross Motor Function Classification System levels I to IV) and 31 neurologically intact children (mean age 11 years and 1 month, SD 3 years and 6 months). The assessgame could distinguish between the children with CP and neurologically intact children. The correlations between the assessgame’s involuntary movement score and the therapist’s rating of the occurrence of involuntary movements during the game were moderate (Spearman ρ=0.56;
The assessgame proved reliable and showed discriminative validity in this preliminary evaluation. Concurrent validity was moderate with the therapist’s opinion but relatively poor with the Selective Control Assessment of the Lower Extremity. We assume that the assessment’s gamelike character demanded various other motor control aspects that are less considered in current clinical assessments.
A loss of selective voluntary motor control (SVMC) is a common negative sign in patients with lesions of the upper motor neuron, for instance, children with cerebral palsy (CP) [
Despite the well-known importance of lower limb SVMC for motor activities, only a few tools are used to measure it [
The focus of this study was to perform a similar preliminary investigation of several psychometric properties of the assessgame for the lower limbs in children with CP. As a gold standard is lacking for measuring SVMC, we evaluated the discriminative and concurrent validity. In line with the findings for the upper limbs [
For concurrent validity, we expected moderate to high correlations (0.50≤|ρ|≤0.70) between the assessgame outcomes and the SCALE. In addition, we expected low correlations (|ρ|<0.50) with the Gross Motor Function Classification System (GMFCS), as it is not a specific measure of SVMC. We expected high positive correlations (|ρ|>0.70) of the assessgame scores with a therapist’s rating of movement selectivity during the game to internally validate the analysis algorithm.
Finally, we investigated the test-retest reliability of the assessgame. We considered test-retest reliability as good when ICCs exceeded 0.75 and absolute measurement errors were acceptable.
Inpatients and outpatients of the Swiss Children’s Rehab of the University Children’s Hospital Zurich were recruited by convenience sampling from June 2017 to March 2018. Inclusion criteria comprised a clinical diagnosis of predominantly spastic CP (ie, unilateral or bilateral spastic CP or mixed CP with distinct spastic components), an age between 6 and 18 years, and the ability to follow simple verbal instructions. Children with a primarily dystonic or ataxic impairment, those with an unstable situation regarding their tonus-regulating medications, or those who had a botulinum toxin injection within the last 6 months or any surgical correction of the lower extremity within the last year were excluded.
For establishing discriminative validity, neurologically intact children aged between 6 and 18 years were recruited. Only children without any medical history of neurological or orthopedic diagnosis within the lower extremity were included. In addition, we recruited neurologically intact adults because the algorithm for the accelerometer data analysis [
The study was conducted in accordance with the necessary guidelines and approved by the ethical committee of the Canton of Zurich (Nr PB_2016_01843). A member of the study team explained the study to the participants and their parents and provided them with written participant information in age-adapted versions. Sufficient time was provided to reach a decision. Formal consent was obtained before any measurements were conducted. All the participants provided oral informed consent, and written informed consent was obtained from the adults, adolescents aged ≥14 years, and minors’ parents. They were further informed that they may withdraw from the study at any time and that the withdrawal of consent will not affect the participant’s subsequent medical treatment at the Swiss Children’s Rehab.
On the basis of a previous publication [
Measurement setup of the assessgame “Catch the Stars.” (A and B) Screenshots from the assessgame showing the owl avatar following and collecting the stars on the target trajectory. (C) Standardized testing position and placement of 6 accelerometer pairs (master-slave) that were fixed proximally and distally to each tested joint.
The participants performed 3 try-out trials (hip, knee, and ankle joints once on a self-selected side) to get familiarized with the game. During the actual measurement, each possible joint was selected once as the target joint (ie, resulting in a total of 6 trials) in a randomized order to control for possible learning effects. After completing the calibration, each game round started with an accommodation phase lasting 25 seconds. During this time, the children could familiarize themselves with steering the avatar by collecting a few stars at different positions on the screen. Immediately at the end of the accommodation phase, which was visualized by a starting line, the star-studded path, that is, the target trajectory, began (
To quantify how selectively the game was played, we calculated offline for each target joint an accuracy score and an involuntary movement score that included all simultaneously occurring movements in contralateral or adjacent joints or the trunk (refer to the study by Keller et al [
Then, we calculated the accuracy score and the involuntary movement score, which we standardized to the reference values of 31 neurologically intact adults representing movement mastery. The accuracy score represents the standardized error value (standardized to the SD of neurologically intact adults) between the actual trajectory of the avatar and the target path. It displayed how well the participant could move the target joint to follow the target path accurately. The involuntary movement score describes for all nontarget joints the average difference of the joint movement (angular joint speed) of the participant from the adult mean and is expressed in adult SD units. Larger values suggest worse selective control for both the outcomes.
The first comparator assessment was the SCALE, a valid and established clinical assessment of lower limb SVMC [
As a second comparator measure, we used the GMFCS, which classifies the functional abilities and limitations in the gross motor function of children with CP, emphasizing on sitting, transfers, and mobility [
The third comparator measure was the physiotherapist’s expert opinion. She rated the occurrence of involuntary movements during the assessgame by evaluating the video recordings afterward. Possible types of involuntary movements were mirror movements, trunk movements, or movements in any other joint. For the analysis, we assigned 1 point for each type of involuntary movement that occurred at least once; for example, if all 3 types of involuntary movements were observed, 3 points were given. A selectively performed movement was assigned 0 points. This therapist rating of involuntary movements occurring during the assessgame (ie, not during the SCALE) allows a simple validation of the algorithm extracting involuntary movements from the accelerometer data.
The standardized measurement procedure was carried out by 2 people out of a team of 3 testers (1 experienced neuropediatric physiotherapist and 2 human movement scientists) within 1 hour per session. The entire measurement was performed in a sitting position on a custom-made wooden seat to standardize the body position. For testing ankle movements, the active lower leg was placed on a support with the knee extended at 30°. First, the SCALE assessment was performed. Then, the participants were equipped with the accelerometers, the try-out trials were performed, and the actual measurements took place.
To evaluate the test-retest reliability of the assessgame, the measurement was repeated by the same team under similar conditions (time of day and room). Inpatients who received intensive multimodal rehabilitation were measured again within 1 week to ensure that they remained stable. Outpatients who received no intensive therapy were measured again within 3 weeks.
For all outcomes, we calculated the means of the joints on the more and less affected side as well as an overall mean. If a joint could not be tested with the assessgame (eg, too small ROM), we also excluded the corresponding SCALE score from the analysis. Shapiro-Wilk tests and visual inspection of the data showed that most scores were not normally distributed. Therefore, we applied robust methods to test our a priori formulated hypotheses.
Discriminative validity was determined by a robust, bootstrapped analysis of covariance [
Concurrent validity was evaluated by correlating the 2 assessgame outcomes (accuracy and involuntary movement scores) with (1) the SCALE score and (2) the GMFCS level. Furthermore, the game’s involuntary movement score was correlated with the therapist’s rating. For summary scores (total and leg means), we calculated Spearman rank correlation coefficients, whereas we used Kendall Tau-b rank correlation coefficients for individual joints, where we expected a high number of ties in the data. The magnitudes of the correlation coefficients were interpreted as negligible (0.00≤|r|≤0.29), low (0.30≤|r|≤0.49), moderate (0.50≤|r|≤0.69), high (0.70≤|r|≤0.89), or very high (|r|≥0.90) [
Relative test-retest reliability was investigated using a 2-way random effects model based on absolute agreement (ICC 2,1 according to Shrout and Fleiss nomenclature [
All statistical analyses were performed with R statistical package (version 3.4.4; R Foundation for Statistical Computing) [
A total of 24 children with CP provided informed consent. As 4 children were not able to complete the assessgame because of a lack of cognitive understanding of the game (3/4, 75%) or visual impairment (1/4, 25%), we included the data of 20 children with spastic and mixed types of CP (bilateral: 17/20, 85%; unilateral: 3/20, 15%).
Their age ranged from 7 years and 11 months to 17 years and 5 months with a mean age of 12 years and 5 months (SD 3 years and 4 months). Of the 20 participants, 7 (35%) were female. In total, 35% (7/20) of children had GMFCS level I, 15% (3/20) had GMFCS level II, 25% (5/20) had GMFCS level III, and 25% (5/20) had GMFCS level IV. Descriptive statistics of all SVMC measures are presented in
A descriptive summary of the peer control group (neurologically intact children: n=31; mean age 11 years and 1 month, SD 3 years and 6 months; n=16, 52% females) and adult reference group (n=31; mean age 33 years and 9 months, SD 7 years and 5 months; n=15, 48% females) is shown in
Descriptive statistics of the outcome measures.
|
More affected lega | Less affected leg | Total | ||||
|
Median (IQRb) | Range | Median (IQRb) | Range | Median (IQRb) | Range | |
|
|||||||
|
Accuracy | 2.42 (1.91-3.20) | 0.82-5.52 | 2.20 (1.63-3.88) | 0.81-5.84 | 2.55 (1.65-3.38) | 0.81-5.03 |
|
Involuntary movements | 1.65 (1.34-2.65) | 0.82-3.58 | 1.74 (1.23-2.44) | 0.75-4.13 | 1.73 (1.27-2.79) | 0.78-3.16 |
SCALEc | 1.0 (1.0-1.66) | 0.0-2.0 | 1.33 (1.0-1.75) | 0.0-2.0 | 1.2 (1.0-1.66) | 0.0-2.0 | |
Therapist’s opinion | 0.83 (0.62-2.00) | 0.00-3.00 | 1.17 (0.46-1.88) | 0.00-3.00 | 1.00 (0.50-1.88) | 0.00-3.00 |
aThe summary scores (for each leg and total) represent the average values of the individual joints, n=20.
b1st-3rd quartile.
cSCALE: Selective Control Assessment of the Lower Extremity.
Concerning our hypothesis on discriminative validity, a robust analysis of covariance compared the assessgame total scores between the children with CP and neurologically intact children at the discrete ages of 9.5, 12.5, and 15.5 years. The bootstrapped CIs for the mean difference did not include 0 in any of the comparisons, indicating that the children with CP had significantly worse SVMC (ie, higher scores) than their neurologically intact peers (
The assessgame scores and age. Scatterplots of the assessgame outcomes by age with smoothing lines are shown for children with cerebral palsy and neurologically intact children separately. Dashed lines depict at which age the patients and neurologically intact children were compared with a robust analysis of covariance. The differences between the groups and the bootstrapped 95% CIs are presented below.
The correlations between the assessgame outcomes and the SCALE and between the assessgame outcomes and GMFCS were negligible to low and nonsignificant (
Relationships between the assessgame outcomes and the comparator measures.
|
More affected leg | Less affected leg | Total | ||||||
|
ρ | ρ | ρ | ||||||
|
|||||||||
|
Game accuracy | −0.25 | .28 | −0.14 | .57 | −0.29 | .21 | ||
|
Game involuntary movements | −0.19 | .43 | −0.19 | .42 | −0.14 | .54 | ||
|
|||||||||
|
Game accuracy | 0.42 | .06 | 0.30 | .19 | 0.39 | .09 | ||
|
Game involuntary movements | 0.18 | .45 | 0.24 | .32 | 0.23 | .32 | ||
|
|||||||||
|
Game involuntary movements | 0.59 | .006 | 0.57 | .009 | 0.56 | .01 |
aSCALE: Selective Control Assessment of the Lower Extremity.
bGMFCS: Gross Motor Function Classification System.
Of the 20 participants, 3 (15%) children could not participate in a second measurement owing to organizational issues. Inpatients were reassessed 1 to 8 days (mean 5.3, SD 2.5 days) after the first appointment, and outpatients were reassessed 6 to 21 days (mean 14.9, SD 6.5 days) later.
With ICC values for the total scores ≥0.86 and the 95% CIs >0.75, the test-retest reliability was in a good range (
Test-retest reliability of the assessgame of selective voluntary motor control.
|
More affected leg | Less affected leg | Total | ||||
|
Game ASa | Game IMSb | Game AS | Game IMS | Game AS | Game IMS | |
Mean (SD) 1 | 2.60 (1.34) | 1.88 (0.84) | 2.49 (1.48) | 1.86 (0.84) | 2.55 (1.35) | 1.87 (0.73) | |
Mean (SD) 2 | 2.62 (1.47) | 2.09 (1.19) | 2.33 (1.35) | 1.62 (0.68) | 2.47 (1.35) | 1.84 (0.84) | |
.78 | .06 | .99 | .89 | .75 | .55 | ||
ICCd (2,1) (95% CI) | 0.94 (0.80-0.98) | 0.52 (0.06-0.85) | 0.87 (0.57-0.96) | 0.80 (0.17-0.93) | 0.97 (0.88-0.99) | 0.86 (0.76-0.93) | |
MDC95e | 0.97 | 1.97 | 1.40 | 0.98 | 0.63 | 0.82 | |
MDC95/grand meanf (%) | 37 | 99 | 58 | 56 | 25 | 44 |
aAS: accuracy score.
bIMS: involuntary movement score.
cUncorrected
dICC: intraclass correlation coefficient; the
eMDC95: minimal detectable change at 95% confidence level.
fThe grand mean was the average of the first and second means.
We investigated the discriminative and concurrent validity and test-retest reliability of a gamelike assessment for SVMC in children with CP. In summary, the assessgame could differentiate well between the neurologically intact children and children with CP. Although the assessgame’s involuntary movement sum scores correlated moderately with the therapist’s expert opinion about the involuntary movements that occurred during the game, these correlations were high for the ankle and knee joints of the more affected side. The assessgame’s accuracy and involuntary movement scores correlated worse with the SCALE score and the GMFCS level as hypothesized. Test-retest reliability was generally good to excellent, and most ICCs exceeded the minimum required threshold of 0.75, except for the occurrence of involuntary movements of the more affected leg. The acceptability of the absolute reliability was more challenging to interpret, but a high percentage of patients would be able to improve their total scores by the MDC without showing a ceiling effect. When interpreting the scores for the more and less affected sides separately, it seemed that most children with CP could improve in accuracy. At the same time, only some children might be able to reduce the occurrence of involuntary movements. The poorer reliability of the involuntary movement score of the more affected leg might indicate that such movements occur less regularly.
The assessgame could differentiate well between neurologically intact children and children with CP, with similar differences independent of age. Although we had expected smaller differences in the assessgame scores between young (ie, aged 6-7 years) patients and their neurologically intact peers, our data could not confirm this, as we were not able to recruit children with CP in this age range (
A comparison with other psychometric studies is only partly possible, as only a few other SVMC tools exist, and their psychometric properties have rarely been investigated [
The relationships with comparator measures were also weaker than those observed in other lower extremity studies. Although other studies using clinical lower extremity SVMC assessments such as the selective motor control scale or the SCALE found relationships with gross motor function [
Furthermore, during the measurements, we observed that the children were quite immersed in the assessgame. They focused on collecting as many stars as possible rather than on movement quality (ie, no involuntary movements). Therefore, the assessgame resembles a more playful situation where selective control is required but not the focus of the action. By contrast, during the SCALE, the children had immediate visual control and feedback over their movement, were continuously guided by the therapist, and could display their undivided attention solely on performing selective movements.
Although the relationships between the assessgame outcomes and SCALE were indeed weaker than we had anticipated, the moderate correlations between the assessgame outcomes and therapist’s rating of involuntary movements occurring during the game, which were moderate to high for the knee and high for the ankle joint specifically, indicate that the assessgame is valid in assessing SVMC. The relationships might have been even stronger if the therapist had also quantified the intensity and frequency of the occurring movements.
In our opinion, the assessgame seems to measure SVMC during a more difficult task in a different context (ie, an immersive gaming environment) and in greater detail compared with the current clinical SVMC assessments. It assesses movement control more accurately (ie, finer graded and accurately timed movements) and includes the magnitude and frequency of involuntary movements. These differences make the concurrent validity testing difficult.
As for the reliability results, the English and German versions of the SCALE and the selective motor control scale were shown to have nearly excellent relative reliability in children with CP [
The MDCs relative to the grand mean were mostly higher than those found for the SCALE. Balzer et al [
First, the sample size was relatively small; however, with 20 participants, we were sufficiently powered to detect correlations of ≥0.58 (pwr.r.test [
Second, the game requires good cognitive functions of participants to maintain concentration and an active ROM of at least 10°; this resulted in dropouts and missing data for some joints. Owing to dynamic or fixed ankle contractures in children with CP, their active ROM was often too small to play the game, and important information on ankle control was lost.
Third, we decided to study children with unilateral and bilateral spastic CP for this preliminary investigation of psychometric properties and usability. Although this sample was more homogeneous than the sample included in the psychometric study of the upper limb assessgame [
Fourth, reliability could likely be improved by testing each joint more than once and taking the average outcome. Several repetitions would also allow better control for learning effects, if they exist. However, repeating joint assessments seems unfeasible in terms of motivation and compliance if all joints are to be tested. The assessgame was developed to keep motivation and emotional engagement high during testing. Nevertheless, engagement differed between participants and might have influenced the outcomes independent of the participant’s selective control abilities.
Finally, we underestimated the differences in patient requirements for performing the clinical SCALE and the assessgame. Although both assessments build on the definition of reduced selective motor control, the assessgame differs in various aspects, as discussed earlier, which has negatively influenced parts of the validity analyses. Future psychometric studies (on similar assessments) should consider this and could further evaluate the validity of the assessgame by including comparator assessments of more refined control of movement or muscle activation, action planning or visuomotor coordination, and the frequency and amplitude of involuntary movements.
Unlike the SCALE, whose single outcome includes both timed gross motor control and the presence of involuntary movements, the assessgame separates movement accuracy from the occurrence of involuntary movements. This separation is of clinical importance, as it could direct therapists in personalizing the therapy program (eg, improving accurate motor control or inhibiting involuntary movements). However, in particular, the assessgame’s involuntary movement score should become more refined to inform therapeutic decisions in more detail (ie, differentiating between mirror movements and comovements). As a first attempt, we developed a possible clinical output for the assessgame providing information similarly to the descriptors of the SCALE (an example is shown in
The setup, test conduction, and analysis of the assessgame are rather time consuming, which limits the practicability of such an assessment in its current form. Technical adaptations could help optimize the setup such that fewer sensors are required. Furthermore, instead of testing all joints, the test could be focused on 1 or 2 specific target joints selected based on their relevance for the children and their families.
Future studies should investigate SVMC in a larger sample of healthy children to establish robust norm values for the assessgame scores. A first analysis of the current data from 31 healthy children already showed a strong correlation with age [
Another clinical and scientific application could be to adapt the concept of the assessgame for an intervention to train SVMC in children with CP. Meanwhile, we have developed an interactive computer game for improving SVMC [
Finally, although we evaluated this assessgame in children with CP, it could be applied to various patient groups with upper motor neuron lesions, both young and adult, as shown for the upper limbs [
This study provided preliminary evidence for the validity and good relative test-retest reliability of a new playful assessgame to measure SVMC of the lower extremities in children with CP. The assessgame differs from the existing assessments of SVMC, and its gaming character might have a complementary value in the measurement of SVMC. This may deepen our understanding of the complex mechanism of motor control. Future studies must show whether and which other aspects of motor control the assessgame includes for it to become an appropriate assessment for clinical or research use. As the relative reliability was good but the absolute reliability was rather low, further studies are needed to investigate the responsiveness of the assessment.
Descriptive statistics of the healthy reference groups, the relationships between the assessgame outcomes and comparator measures for individual joints, and test-retest reliability analyses for individual joints.
Example of a clinical output for the assessgame.
cerebral palsy
Gross Motor Function Classification System
intraclass correlation coefficient
minimal detectable change
range of motion
Selective Control Assessment of the Lower Extremity
selective voluntary motor control
The data sets generated and analyzed during this study are not publicly available owing to ethical concerns about the small number of patients but are available from the corresponding author upon reasonable request.
This research project was funded by the Swiss National Science Foundation (Grants 32003B_156646 and 32003B_179471), Physio Swiss, Foundation for Physiotherapy Science, and Mäxi-Foundation. None of the funders was involved in the study design, data collection and analysis, manuscript preparation, or publication decisions.
The authors are grateful to the therapists of the institutions Stiftung Vivendra and Schule für Körper- und Mehrfachbehinderte Zürich and the teachers of the Kantonsschule Frauenfeld for their assistance in recruiting participants. The authors would like to thank all the participants of this study and their parents.
AF carried out the data collection and analysis, performed the statistical analyses, and drafted the manuscript. JB conceived and designed the study, carried out the data collection and analysis, drafted the manuscript, and acquired funding. JWK conceived and designed the study, carried out the data collection and analysis, and performed the statistical analyses. HJAvH conceived and designed the study, acquired funding, and obtained ethics approval. All the authors read and approved the final manuscript.
None declared.