This paper is in the following e-collection/theme issue:

Emerging Technologies for Rehabilitation (241)

Published on in Vol 4, No 1 (2017): Jan-Jun

Counting Grasping Action Using Force Myography: An Exploratory Study With Healthy Individuals

Counting Grasping Action Using Force Myography: An Exploratory Study With Healthy Individuals

Counting Grasping Action Using Force Myography: An Exploratory Study With Healthy Individuals

Authors of this article:

Zhen Gang Xiao1 Author Orcid Image ;   Carlo Menon2 Author Orcid Image
Article Authors Cited by (19) Tweetations (4) Metrics

    Original Paper

    1Simon Fraser University, Burnaby, BC, Canada

    2Schools of Mechatronics Systems Engineering and Engineering Science, Simon Fraser University, Surrey, BC, Canada

    Corresponding Author:

    Carlo Menon, PhD

    Schools of Mechatronics Systems Engineering and Engineering Science

    Simon Fraser University

    SFU Surrey, 4374 Galleria

    250-13450 102 Avenue

    Surrey, BC, V3T 0A3

    Canada

    Phone: 1 778 782 9338

    Fax:1 778 782 7514

    Email: cmenon@sfu.ca


    Background: Functional arm movements generally require grasping an object. The possibility of detecting and counting the action of grasping is believed to be of importance for individual with motor function deficits of the arm, as it could be an indication of the number of the functional arm movements performed by the individuals during rehabilitation. In this exploratory work, the feasibility of using armbands recording radial displacements of forearm muscles and tendons (ie, force myography, FMG) to estimate hand grasping with healthy individuals was investigated. In contrast to previous studies, this exploratory study investigates the feasibility of (1) detecting grasping when the participants move their arms, which could introduce large artifacts to the point of potentially preventing the practical use of the proposed technology, and (2) counting grasping during arm-reaching tasks.

    Objective: The aim of this study was to determine the usefulness of FMG in the detection of functional arm movements. The use of FMG straps placed on the forearm is proposed for counting the number of grasping actions in the presence of arm movements.

    Methods: Ten healthy volunteers participated in this study to perform a pick-and-place exercise after providing informed consent. FMG signals were simultaneously collected using 2 FMG straps worn on their wrist and at the midposition of their forearm, respectively. Raw FMG signals and 3 additional FMG features (ie, root mean square, wavelength, and window symmetry) were extracted and fed into a linear discriminant analysis classifier to predict grasping states. The transition from nongrasping to grasping states was detected during the process of counting the number of grasping actions.

    Results: The median accuracy for detecting grasping events using FMG recorded from the wrist was 95%, and the corresponding interquartile range (IQR) was 5%. For forearm FMG classification, the median accuracy was 92%, and the corresponding IQR was 3%. The difference between the 2 median accuracies was statistically significant (P<.001) when using a paired 2-tailed sign test. The median percentage error for counting grasping events when FMG was recorded from the wrist was 1%, and the corresponding IQR was 2%. The median percentage error for FMG recorded from the forearm was 2%, and the corresponding IQR was also 2%. While the median percentage error for the wrist was lower than that of the forearm, the difference between the 2 was not statistically significant based on a paired 2-tailed sign test (P=.29).

    Conclusions: This study reports that grasping can reliably be counted using an unobtrusive and simple FMG strap even in the presence of arm movements. Such a result supports the foundation for future research evaluating the feasibility of monitoring hand grasping during unsupervised ADL, leading to further investigations with individuals with motor function deficits of the arm.

    JMIR Rehabil Assist Technol 2017;4(1):e5

    doi:10.2196/rehab.6901

    Keywords

    myography; classification; upper extremity; grasp; rehabilitation 


    Individuals with motor function deficits of their arm (eg, individuals with stroke and a hemiparetic arm) often involuntarily avoid using their weak arm during the activities of daily living (ADL)[1,2], leading to an inevitable and gradual degradation of their ability to move their arm. Published studies have shown that increasing the use of a person’s weak arm is believed to be an important factor for a successful recovery [3]. Technologies that provide objective feedback to the individual on the use of their arm could potentially encourage them to be more proactive in using their affected arm [4], and consequently gradually improve their arm motor functions.

    Some studies have used accelerometer-based devices to capture gross movements of the arm [3,5,6] and provide that information as activity feedback to the individual. However, such devices generally cannot discern between movements that are functional and believed to be relevant for the recovery [7,8] (ie, grasping a glass and drinking) from those that are not functional (ie, arm movements induced by movements of the body, such as turning, walking, moving during sleeping) [1,3]. Therefore, accelerometer-based devices could provide inaccurate and potentially counterproductive feedback [1].

    Devices with the ability to detect grasping motions, which is generally required during functional arm movements, could potentially provide a more suitable indication of functional use of the upper limb [2,9]. Studies showed that exercising the arm by grasping an object has the potential to greatly improve rehabilitation outcomes [10,11]. Hence, detecting the number of grasping actions performed by an individual during ADL could be used as feedback to facilitate rehabilitation. In addition to uses for rehabilitation, the ability to unobtrusively detect grasping motions could be used in applications such as monitoring the repetitive hand activity level of a worker for load transfer tasks [12], or could be used in identification of hand-held objects [13]. Hence, innovative solutions to detect grasping motions are, therefore, in need.

    Currently, commercial devices capable of detecting grasping motions do exist. They are mostly based on either a vision-based approach or using a wearable technology approach such as wearing data gloves. The vision-based systems, such as Microsoft’s Kinect [14], Leap Motion Controller [15], and Optotrak [16], have to be mounted externally to the user’s body and are generally used in well-controlled indoor environments such as a rehabilitation center or clinic. They cannot be used to monitor ADL, especially when the individual is in an outdoor setting. On the other hand, the use of a data glove, such as the CyberGlove [17], can be used for monitoring grasping motions outdoors. However, data gloves are generally not practical for use by individuals with a clutched hand, as in the case of individuals with a hemiparetic arm resulting from stroke. In fact, they require a considerable effort to be donned and doffed. Furthermore, they cannot be used in many ADL, such as washing dishes, taking a shower, etc, as they are not waterproof or are simply uncomfortable. Data gloves also reduce the tactile sensation of the hand and fingers, which poses a further barrier for being accepted by the users [18,19].

    In addition to the above-mentioned commercially available technologies, the academic community has investigated different approaches to classify grasping and other hand gestures. One of these approaches is based on surface electromyography (sEMG) recorded from the forearm [20]. While such an approach could potentially be used in a large variety of environments, including outdoors, its signal quality may degrade due to many environmental factors, such as sweating and electrical noise, which has been shown to drastically affect its performance [21]. Furthermore, medical-grade sEMG systems capable of capturing low noise sEMG signals generally cost thousands of US dollars (eg, Noraxon sEMG system), which makes them unsuitable for being implemented in practice.

    An alternative approach to detect grasping is force myography (FMG). FMG is a technique that uses sensors to capture displacements of muscles, skin, and tendons [22]. This technique was also referred to as topographic force mapping [21], residual kinetic imaging [23], or muscle pressure distribution mapping [24]. Although the FMG technique is relatively unexplored and less standardized compared with sEMG, it presents different potential advantages over the latter. Specifically, FMG signals do not degrade due to sweating or electrical noise [21]. As a consequence, the related hardware for the signal acquisition is also less sophisticated and expensive. While FMG devices are currently not commercially available, an experimental prototype of an FMG signal acquisition device costs less than US $50 [25].

    The use of FMG can be dated back as early as the 1960s, when FMG was proposed for controlling a single-degree prosthetic terminal device [26]. Since then, the use of FMG for controlling hand prosthesis has gained some interest in the research community [27-33]. At the same time, researchers also explored the use of FMG for individuals with intact limbs for various applications. For example, FMG signals taken from healthy individuals were studied for regressing isometric force applied by the fingers [22,25], as well as for recognizing different hand gestures and finger movements [24,31,34,35]. Also, a preliminary test performed by Yungher and Craelius showed that regressing the grasping force through the forearm FMG of individuals with poststroke condition was viable [36]. Recently, robotic orthosis with FMG sensing capability were proposed for potential stroke rehabilitation applications [37,38].

    FMG signals to detect hand movements are generally extracted from the middle of the forearm where large radial displacements can be recorded [22,24,25,31,34,39-42]. Recent studies have, however, explored the possibility of estimating hand movements by processing FMG signals recorded at the wrist. For instance, Morganti et al proposed a wrist strap consisting of four force-sensing resistors (FSRs) to detect wrist positions [43]. Dementyev and Paradiso subsequently developed a wrist strap that was capable of deciphering 6 static hand gestures [44]. The works of both Morganti and Dementryev showed the potential of embedding FMG sensors inside a watch strap, which could make the technology acceptable for users, especially those who may highly value the cosmetics of the device.

    The large majority of studies on FMG for the upper extremities presented in the literature considered exploratory tests in very controlled scenarios, where healthy volunteers were asked to move their hand or wrist while maintaining a fixed elbow position [21,22,24,25,29,31,35,43-46]. Despite being able to obtain high prediction accuracy, this approach does not truly reflect the capability of FMG for detecting hand action in a practical scenario, in which arm movements are generally present. To the best of the authors’ knowledge, the only studies that included arm movements with the use of FMG are the ones performed by Ogris et al [47] and Sadarangani and Menon [48]. Both of their studies used FMG in conjunction with inertial measurement unit (IMU) to decipher various ADL. The result of Ogris’s work showed that FMG improved classification accuracies of some ADL; however, the capability of using FMG to decipher hand action was not fully investigated. In the other study, Sadarangani and Menon’s work focused on the investigation of detecting hand actions, but only for 3 limited scenarios.

    As a fundamental step toward the development of a technology suitable to detecting grasping motions in the presence of arm movements as an indirect estimation of functional arm movements, this study investigates the ability of using FMG to count the number of grasping motions during a series of pick-and-place (PAP) actions. Two wireless FMG straps were prototyped and placed close to the wrist and on the forearm of 10 healthy individuals for this study.


    FMG Signal Extraction and Data Transmission

    Figure 1 shows the 2 FMG strap prototypes used in this study. The strap in the left of the figure is 28 cm long and it was designed to be donned on the forearm while the strap in the right is shorter (19 cm) and was designed to be donned on the wrist, like a watch. Each strap had 8 FSR sensors (FSR 402 from Interlink Electronics), which were evenly distributed on the straps’ inner surface (see Figure 1).

    A single FSR sensor has 2 terminals: one terminal is connected to a common analog input pin of a microcontroller (Atmega 328p from Atmel) with an internal pull-up resistor (37.5 kΩ) equipped, and the other terminal is connected to a digital control pin as shown in Figure 2.

    The analog input pin takes the reading of the signal and converts it into a 10-bit unsigned integer value. Since only 1 analog pin was used for sampling, the signal of each FSR was sampled sequentially. The order of sampling was determined by the digital control pin. When the selected FSR signal was ready to be sampled, the corresponding control pin was set to low, and the other control pins were set to be in high impedance states. At any single moment, only 1 control pin would be set to low and others would be changed to high impedance state in order to guarantee independent sampling. This configuration used a single analog pin with internal pull-up resistor in order to obtain the most simplified design under the constraints of the selected inexpensive microcontroller.

    The sampled data were transmitted wirelessly to a personal computer using a generic Bluetooth module (HC-05). A custom-made application with real-time signal display was developed in LabVIEW on a personal computer for querying and storing the sample data. When sampling began, the application sent a command to the microcontroller to retrieve a set of FSR data at every 100 milliseconds (10 Hz) as proposed by Amft et al [40].

    Experimental Protocol

    An experimental protocol was designed to capture both wrist and forearm FMG signals simultaneously during a PAP exercise. Before the experiment, the forearm FMG strap was donned on the belly of the right forearm of a volunteer with the help of the research assistance. The wrist FMG strap was instead donned on the distal end of the forearm (next to the ulna styloid process, see Figure 3). In order to reduce signal inconsistency due to the placement of the strap for different volunteers, the first sensor near the tail end of the forearm strap (see Figure 1) was always placed on the bulk of the flexor carpi ulnaris, and the wrist counterpart was always placed near the ulna styloid process. However, it should be noted that the rest of the sensors were not positioned on specific muscles or tendons (the FSR were evenly distributed in the strap). This approach was intentionally followed to avoid personalization of the strap and provide a generic strap that could be used by a layperson at home. Finally, another FSR sensor (FSR 400 Interlink Electronics) was taped to the pulp of the thumb to obtain the true label for investigational purposes.

    Once the straps were donned, the volunteer was asked to fully extend the fingers and then make a fist 3 times, while the research assistant monitored the raw signal through visual feedback from the display of the LabVIEW application. This hand action was shown to be able to generate a clear and visually distinguishable FMG pattern for healthy individuals [29,40,44], and therefore, the action was used to ensure the strap was able to register muscle-tendon movement activities. If this action did not generate a clear FMG pattern, the research assistant would readjust the tightness of the sensor. Also, the assistant would ensure the strap did not block blood circulation or cause discomfort to the volunteer through his or her oral feedback. On average, this calibration procedure took less than 3 minutes for each volunteer.

    In the experiment, the volunteers sat in front of a table as illustrated in Figure 4. They were then asked to pick up and place a cylindrical object from and to 6 locations following different sequences. The object used in the experiment was a 12-cm high hollow cylinder with a radius of 3 cm. It weighed only 73 g so that the participants did not need to apply a large force to lift the object. The 6 locations included 1 start location (Location 0) and 5 other target locations (Locations 1-5). Locations 1-5 were placed around Location 0, at a distance of 40 cm. Using Location 0 as the reference, each of the 5 locations were 30 degrees apart from the adjacent one. The elevations of the 5 target locations were 30 cm, 1 cm, 40 cm, 10 cm, and 20cm from the table, respectively. Each location had a circular area with a 5-cm radius, such that the upper limb joints of the volunteer must be highly coordinated in order to successfully place the object on the target locations.

    In order to capture FMG signals in the presence of various arm movements, 3 PAP sequences were designed for the participants to perform. These PAP sequences required the coordination of the shoulder, the elbow, wrist, and hand. Therefore, the FMG patterns that were associated with some of the elbow flexion/extension, forearm pronation/supination, wrist flexion/extension/abduction/adduction, hand opening/closing, and the overall arm motion would be captured. Some examples of the captured movement during the PAP sequences are shown in Figure 5.

    In the first sequence, the participant was asked to pick up the object from the start location and place it onto the target locations at a pace comfortable for them. Then the participant retrieved their hand to the start location without the object. Next, the participant picked up the object from the current location, and returned it to the start location. Finally, the participant released the object completely before starting the next PAP action. In total, each participant performed 10 PAP actions for the first action sequence as shown in the left picture of Figure 6. In the second sequence, the participants performed an additional 10 PAP actions following the order shown in the middle of Figure 6. In the third sequence, the participants repeated the path of the second sequence but in a reversed order, as shown in the right of Figure 6. Each participant was asked to repeat the 3 sequences (30 distinct PAP actions in total) 5 times. With 3 sequences and 5 repetitions, a total of 150 PAP actions were recorded.

    ‎
    Figure 1. Wireless FSR straps: (a) wireless FSR strap for the forearm and (b) wireless FSR strap for the wrist.
    View this figure
    ‎
    Figure 2. Schematic for FMG signal extraction and data transmission. The microcontroller sampled the signal from 8 FRS sensors, and the data were sent wirelessly to the computer through the Bluetooth transmitter module.
    View this figure
    ‎
    Figure 3. FSR straps placement: (a) forearm supinated view and (b) forearm pronated view.
    View this figure
    ‎
    Figure 4. Experimental setup. The start location is shown in gray, the five target locations are shown in green, and the object for grasping is shown in yellow.
    View this figure
    ‎
    Figure 5. Examples of upper limb position during the PAP sequence: (a) grasping the object from start position; (b) transporting the object from start position to target location 4; and (c) transporting the object from target location 2 to 4.
    View this figure
    ‎
    Figure 6. PAP action sequence. The red circles indicate the target positions, the arrows indicate the direction of the PAP actions, and the numerical labels indicate the orders of the steps in each sequence: (a) first PAP action sequence; (b) second PAP action sequence; and (c) third PAP action sequence.
    View this figure
    ‎
    Figure 7. Data processing sequence.
    View this figure

    Participants

    Ten healthy volunteers aged between 21 and 42 years participated in the experiment. Each participant signed an informed consent form (approved by the Office of Research Ethics, Simon Fraser University) before entering the study. Their wrist and forearm belly circumferences were recorded for performance analysis and are shown in Table 1.The average circumference of the wrist and forearm belly are 16.81 cm (SD 1.11) and 26.2 cm (SD 3.15), respectively.

    Table 1. Participant statistics.




    ID    		Wrist circumference (cm)Forearm belly circumference (cm)
    116.528
    21927
    317.527
    41627
    517.628.5
    615.527
    71617
    81826
    916.526.5
    1015.528
    Average16.8126.2
    SD1.113.15

    Data Processing

    The data collected from each participant consisted of both wrist and forearm FMG positions. The 2 streams of data were processed through identical but independent treatments. The collected data were first divided into training and testing sets. The training set data consisted of the first 30 PAP actions, and the testing set consisted of the rest of the 120 PAP actions data sets. The training set was used for extracting relevant statistical information about the signals and for generating a classifier model. The testing set was used for examining the generalized performance of the classifier for detecting grasping. The overall data processing sequence is shown in Figure 7.

    As shown in Figure 7, the raw FMG data of each channel was first centered by subtracting its mean and then normalized using its standard deviation. Both the mean and standard deviation parameters were obtained from the training set.

    Next, feature extraction was performed. The raw FMG data were considered as primary feature of the signal. Three additional signal features, namely the root mean square (RMS), waveform length (WL), and window symmetry (WS), were extracted from the raw data with a 300 ms window and a step size of 100 ms.

    RMS is the averaged signal magnitude of each window and its equation is shown in Figure 8. In the equation, xi is the value of the ith sample in the processing window and N is the window size, which in our case was 3.WL is the sum of the change of the input samples within the processing window, which provides speed-related information to the classifier. The formula for computing WL is shown in the middle of Figure 8.

    WS is the difference between the average of the first N data points and the one of the last N data points, which can provide directional information of the change of the input samples. The formula for computing WS is shown in the bottom of Figure 8. A total of 4 features were extracted from each channel including the normalized raw input signal magnitude. Since there were 8 input channels for each of the wrist and forearm FMG straps, a total of 32 features were extracted, respectively. Each of the extracted features were once again centered and normalized based on their mean and standard deviation obtained from the training set before being classified using the supervised classification scheme.

    Under the supervised classification scheme, the classifier needs to be trained using true label obtained from external source. In our case, the labeling signal was the one recorded by the FSR sensor placed on the thumb. This signal measured the amount of contact force between the object and the thumb. If the contact force was less than 2% of the maximum, then the corresponding FMG data was labeled as nongrasping (class 1); otherwise, the FMG data was labeled as grasping (class 2).

    Among different supervised classifiers, linear discriminant analysis (LDA) classifier using Fisher discriminant criteria is one of the most widely used for analysis. LDA fits a multivariate normal density to each class with a pooled estimate of covariance. It is capable of revealing linear separability of the signal features. Additionally, LDA is computationally efficient and suitable to be implemented in a microcontroller [49]. Therefore, it was selected for use in this study.

    The output of the LDA classifier was the predicted state of the hand. In order to count the number of grasping actions, the transition from nongrasping to grasping state needed to be identified. This transition could be detected by subtracting the current state output with the previous one. A positive result indicated a grasping action has occurred. However, the accuracy of such a counting method could be sensitive to any small glitches (misclassifications over a short period, eg, <1s) in the classification data output stream. Hence, an average filter was applied to smooth out the output stream of the classifier. The window size of the filter could affect the overall performance in terms of the counting accuracy and the delay. Therefore, the effect of different window sizes on the counting performance was examined (see Results section).

    ‎
    Figure 8. Equations for FMG feature extraction: (a) Root mean square, (b) Waveform length, and (c) Window symmetry.
    View this figure
    ‎
    Figure 9. Boxplot for FMG classification accuracies of the 10 participants. The bottom and top of each box are the first and third quartiles of the data set, respectively. The band inside the box is the median. The ends of the dashed lines (whiskers) are the minimum and maximum of the data. The red and blue boxes indicate classification accuracies related to FMG collected from the wrist and forearm, respectively. (a) Accuracies computed using training set data. (b) Accuracies computed using testing set data.
    View this figure
    ‎
    Figure 10. Classification result comparison: (a) accuracy comparison using wrist FMG and (b) accuracy comparison using forearm FMG.
    View this figure
    ‎
    Figure 11. Regression plots for force-myography (FMG) classification accuracies of the 10 participants: (a) wrist FMG classification accuracies versus wrist circumference; (b) forearm FMG classification accuracies versus wrist circumference; (c) wrist FMG classification accuracies versus time per action; and (d) forearm FMG classification accuracies versus time per action.
    View this figure
    ‎
    Figure 12. Boxplots for grasp counts versus number of filtered samples: (a) result generated using wrist FMG and (b) result generated using forearm FMG.
    View this figure
    ‎
    Figure 13. Boxplots for percentage error of grasp counts versus number of filtered samples: (a) result generated using wrist FMG and (b) result generated using forearm force-myography.
    View this figure

    Performance Evaluation

    The overall performance of the proposed method was evaluated based on 2 metrics: the classification accuracies and percentage errors of the grasp count obtained from the test set.

    The classification accuracy was calculated based on the sum of the correctly classified sample over the total number of samples. The difference of classification accuracies between wrist and forearm FMG using LDA was evaluated. Also, the performance of LDA was compared with other 2 popular classifiers, namely the Radial Basis Function kernel Support Vector Machine (RBF-SVM) and the 2-layer Artificial Neural Network (ANN). In addition, the correlations between the accuracies and action speeds, as well as the size of the wrist or forearm were assessed using a regression method.

    The percentage error of the grasp count was based on the result of the absolute difference between the predicted and the expected counts over the expected one. The expected count for the test set was 120 in this study. The difference between the errors of the wrist and forearm FMG approaches was also assessed.

    Paired sign test and Kruskal-Wallis test were used for examine the statistical significance of the obtained results. All the statistical analysis was performed using the same significance level (alpha) of .05.


    LDA Classification Results

    The LDA classification accuracies of the 10 participants for nongrasping (class 1) and grasping (class 2) an object are shown in Figure 9. The combined accuracies of the 2 classes are shown in the first pair of results on the left of this figure (see “Overall accuracies”). Due to the fact that the results were not normally distributed, the median accuracy was used as the indicator for classification performance.

    For the FMG recording from the wrist, the median training accuracy (see the top of Figure 9) was 97% and the corresponding interquartile range (IQR) was 2%. These high training accuracies suggest that FMG patterns recorded from the wrist are suitable to detect grasping and nongrasping during PAP actions. The median accuracy for the testing data set (see the lower plot of Figure 9) was 95% and the corresponding IQR was 5%. The high accuracies for both the training and testing data suggested that the training data was a good representation of the testing data set; no under- or overfitted phenomena was observed.

    Similar results were obtained for FMG data recorded from the forearm: the median training accuracy was 95% and the corresponding IQR was 4%. The median testing accuracy was 91% and the corresponding IQR was 3%.

    The difference between the medians of the wrist and forearm FMG testing accuracies was 4%. With a P value less than .001, the paired right-tailed sign test showed that the prediction using wrist FMG had a statistically significantly higher median accuracy than the one using forearm FMG.

    The second pair of results in Figure 9 (see “Accuracies for nongrasping”) shows the prediction accuracies when the participants did not grasp the object. For FMG recorded from the wrist, the median testing accuracy was 96% and the corresponding IQR was 2% (see the lower plot of Figure 9). For FMG recorded from the forearm, the median testing accuracy was 94% and the corresponding IQR was 8%. With P value equals to .94, the paired right-tailed sign test did not show that the prediction accuracy of using wrist FMG was statistically different from the one using the forearm FMG for the nongrasping state.

    The third pair of results in Figure 9 (see “Accuracy for grasping”) shows the prediction accuracies while the participants grasped the object. For FMG recorded from the wrist, the median testing accuracy was 95% and the corresponding IQR was 4%. For FMG recorded from the forearm, the median testing accuracy was 85% and the corresponding IQR was 4%. With a P value less than .001, the paired right-tailed sign test showed that FMG recorded from the wrist yielded a better prediction accuracy for grasping than FMG recorded from the forearm.

    Classification Result Comparison Between LDA and Other Classifiers

    The performance of using LDA was compared with the ones of RBF-SVM and ANN. Standard model generation procedures for SVM and ANN, which are described in [50] and [51], were followed. For training the SVM model, a ten-fold cross-validation procedure was used to obtain best RBF parameters. For training the ANN model, a 2-layer network with 100 hidden nodes was trained based on a back-propagation algorithm. The obtained testing accuracies from all 3 classifiers are shown in Figure 10. For both wrist and forearm FMG classifications, no statistically significant difference among the results could be established using the Kruskal-Wallis test. The P values obtained from the test were .5 for wrist FMG classification and .9 for forearm FMG classification.

    Classification Results and Participants’ Physical and Performance Factors

    In order to examine if the size of the participants’ limbs and action speeds influence the classification accuracies, 4 regression plots were generated and shown in Figure 11.

    The first row of Figure 11 shows the regression plots of accuracies versus the wrist and the forearm circumference, respectively; the second row shows the regression plots of accuracies of the wrist and the forearm FMG classification versus the average time for the participant to complete a PAP action, respectively. With all P values larger than the specified significance level (alpha=.05), no statistically significant correlation could be established between accuracies and the 2 factors.

    Grasping Count Result

    In order to assess the effect of the filtering window, the grasp counts of the participants were recomputed using different window sizes. Figure 12 shows the corresponding grasping count using box plot for the window sizes of 1-20 samples. The expected counts are shown as a solid line, which was 120 in this experiment.

    Without averaging (sample size equals 1), the numbers of grasping were overestimated by a large margin for all the participants’ data. As the size of the averaging window increased, the counts were closer to the expected value in general. However, when the size continued to increase, the count became increasingly underestimated. On the basis of the result shown in Figure 12, medians of the counting error smaller than 5% were obtained when the number of filtered samples ranged between 4 and 11. The counting errors within such a range of filtered samples are shown in Figure 13.

    As shown in Figure 13, the smallest maximum percentage errors were obtained by using 7 filtered samples for both wrist and forearm FMG counts (5% maximum percentage errors for both cases). Under such conditions, the median percentage error for wrist FMG was 1%, and the corresponding IQR was 2%. For the forearm FMG, the median percentage error was 2%, and the corresponding IQR was also 2%. A P value of .29 was obtained by using the paired 2-side sign test. Despite the wrist FMG having a smaller median, the statistical significant difference between the 2 FMG counts could not be established.


    Primary Findings

    The LDA classifier was selected as the main classifier for this study, and its performance was as good as the more computational-intensive SVM and ANN classifiers. The LDA classification result (see Figure 9) shows that the wrist FMG band produced significantly higher classification accuracies than the forearm FMG band for detecting a grasping state, but no statistical difference was found for detecting a nongrasping state. This result could be associated to the fact that the grasping action occurred closer to the wrist than the forearm. During the object manipulation, movement of the thumb could be better registered by the wrist strap as the tendon and the nearby skin movements contributed to a more distinct FMG pattern. In addition, the FMG from the forearm also captured the pattern related to elbow movement [34], which was a confounding factor for grasp classification. Nevertheless, both FMG methods were capable of producing high classification accuracies (>85%) for all participants. These results confirmed that FMG was an effective method to detect hand grasping even with the presence of complex arm movements. The scope of this study focuses on the capability of FMG only, however, other wearable sensors such as an IMU should be considered along with FMG to further improve accuracy.

    The regression method was used to test if the size of the participants’ limbs and action speeds influence the classification accuracies. The results of the test showed there were no statistically significant correlation between the accuracies and the 2 factors. These results suggested the performance of the FMG method was independent of the limb size and action speed in this study. However, since the participants in this study were healthy individuals, the variations of their physical status and action performance were expected to be insignificant. For future investigation, grasping at various speeds by individuals with different limb sizes should be considered.

    The grasping counts were extracted from the filtered output of the LDA classifiers. As shown in Figure 12, the size of the filtering window largely influenced the accuracies of the counts. Suitable window sizes (medians of the counting error smaller than 5%) were identified to be from 4 to 11 samples, and such window sizes could add 300 ms to 1000 ms of delays to the system. Combined with the delay introduced by the feature extraction process (300 ms), a grasping could be registered by the system after at least 600 ms from the start of the action. While such a delay might be problematic for some real-time human machine interfaces [52-54], it is not considered to be of concern for the targeted activity monitor application, as the user’s instant action does not depend on the feedback [55-57]. Under the optimal settings, which was based on the smallest maximum percentage error, a median percentage error of 1% with IQR of 2% for the grasping count using the wrist FMG was obtained. Compared with the results obtained using forearm FMG, no statistically significant difference was established using the paired sign test. These results show great potential for both FMG approaches.

    This study investigated the capability of using FMG to predict and count grasping actions with healthy individuals. Therefore, the knowledge obtained from this study can be directly applied to the applications in which healthy individuals are the wearers of the FMG bands. An example of such an application could be for monitoring the repetitive hand activity level of a worker during load transfer tasks. However, for rehabilitation applications, as the targeted population will be the people with a weak arm, such as individuals recovering from stroke, further studies are needed to examine the transferability of the result of this study. For example, in poststroke rehabilitation, the targeted wearers are often seniors with limited mobility and range of motion. Their muscles are normally much weaker than the ones of healthy individuals, and may even have significantly deteriorated if the individuals are chronic stroke patients. These characteristics posed questions on whether the proposed FMG method could be used with such a population. Currently, there is lack of in-depth studies that examine the FMG pattern characteristic of stroke patients or people with a weak arm. However, the pilot investigation of Yungher and Craelius showed that the grasping force could be regressed from the forearm of stroke patients (n=4) with mild to moderate spasticity [36]. This study indicated some useful FMG information could still be extracted as long as the patients had some range of motion on the limb. For object manipulation task, this type of patient tends to produce larger grip force, but with less control when compared with the healthy counterpart [58,59]. In such a scenario, distinct FMG patterns associated with some movements are still expected to be captured; however, the consistency of the patterns is likely to be less. The inconsistency due to muscle fatigue may also become more prominent, which might require modification of algorithms to adjust the training parameter of the classifier (eg, normalization parameters of the FMG signals) in order to compensate for the change. In addition, the FMG pattern can be very different among patients due to the different degree of impairment. Because of these conditions, the classifier model may need to be very user- and task-specific in order to tailor for the needs and obtain high prediction accuracy. In order to examine the transferability of the proposed FMG approach for rehabilitation, testing on the stroke population or individuals with weak arm should be the next logical step.

    Conclusions

    The possibility of detecting and counting grasping in the presence of arm movements (PAP exercise) was explored using wrist and forearm FMG strap prototypes. The 2 main performance parameters that were considered were classification accuracy for detecting grasping and percentage error for counting grasping. A high median grasping prediction accuracy was obtained from 10 subjects (95% and 91% for FMG recorded from the wrist and forearm, respectively). A low median grasping count error was also found (1% and 2% for wrist and forearm, respectively). These results provide evidence that FMG-based straps could be used to monitor grasping activities during functional arm movements in a controlled environment. This work poses the foundation for future studies investigating the applicability of using FMG to detect grasping in activities of the daily living first with healthy participants and then with individuals with a weak arm (eg, seniors, individuals with a hemiparetic arm resulting from stroke).

    Acknowledgments

    The authors thank the members of the MENRVA lab, Dr Janice Eng, and Ms Lisa Fraser for their advices and feedback on the use of the proposed technology. This work was supported by the Natural Sciences and Engineering Research Council of Canada, the Canadian Institutes of Health Research, and the Michael Smith Foundation for Health Research.

    Authors' Contributions

    ZGX prototyped the FMG straps, implemented data collection software, performed experiments, analyzed the experimental results, and participated in the manuscript preparation. CM, Principal Investigator of this research, contributed to the design of the study, supervised the work, participated in the interpretation of the results and contributed in writing the manuscript. All authors read and approved the final manuscript.

    Conflicts of Interest

    The Principal Investigator, Carlo Menon, and members of his research team have a vested interest in commercializing the technology tested in this study, if it is proven to be successful and may benefit financially from its potential commercialization. The data are readily available upon request.

    1. Rand D, Eng JJ. Disparity between functional recovery and daily use of the upper and lower extremities during subacute stroke rehabilitation. Neurorehabil Neural Repair 2012 Jan;26(1):76-84 [FREE Full text] [CrossRef] [Medline]
    2. Lin K, Wu C, Wei T, Lee C, Liu J. Effects of modified constraint-induced movement therapy on reach-to-grasp movements and functional performance after chronic stroke: a randomized controlled study. Clin Rehabil 2007 Dec;21(12):1075-1086. [CrossRef] [Medline]
    3. Connell LA, McMahon NE, Simpson LA, Watkins CL, Eng JJ. Investigating measures of intensity during a structured upper limb exercise program in stroke rehabilitation: an exploratory study. Arch Phys Med Rehabil 2014 Dec;95(12):2410-2419 [FREE Full text] [CrossRef] [Medline]
    4. Timmermans AA, Seelen HA, Willmann RD, Kingma H. Technology-assisted training of arm-hand skills in stroke: concepts on reacquisition of motor control and therapist guidelines for rehabilitation technology design. J Neuroeng Rehabil 2009 Jan 20;6:1 [FREE Full text] [CrossRef] [Medline]
    5. Bailey RR, Klaesner JW, Lang CE. An accelerometry-based methodology for assessment of real-world bilateral upper extremity activity. PLoS One 2014 Jul;9(7):e103135 [FREE Full text] [CrossRef] [Medline]
    6. Bailey RR, Klaesner JW, Lang CE. Quantifying real-world upper-limb activity in nondisabled adults and adults with chronic stroke. Neurorehabil Neural Repair 2015 Apr;29(10):969-978. [CrossRef] [Medline]
    7. Bayona NA, Bitensky J, Salter K, Teasell R. The role of task-specific training in rehabilitation therapies. Top Stroke Rehabil 2005 Feb;12(3):58-65. [CrossRef] [Medline]
    8. Arya KN, Verma R, Garg RK, Sharma VP, Agarwal M, Aggarwal GG. Meaningful task-specific training (MTST) for stroke rehabilitation: a randomized controlled trial. Top Stroke Rehabil 2012 Jan;19(3):193-211. [CrossRef] [Medline]
    9. Castiello U. The neuroscience of grasping. Nat Rev Neurosci 2005 Sep;6(9):726-736. [CrossRef] [Medline]
    10. Harris JE, Eng JJ, Miller WC, Dawson AS. A self-administered Graded Repetitive Arm Supplementary Program (GRASP) improves arm function during inpatient stroke rehabilitation: a multi-site randomized controlled trial. Stroke 2009 Jun;40(6):2123-2128 [FREE Full text] [CrossRef] [Medline]
    11. Turton AJ, Cunningham P, Heron E, van WF, Sackley C, Rogers C, et al. Home-based reach-to-grasp training for people after stroke: study protocol for a feasibility randomized controlled trial. Trials 2013 Apr 25;14:109 [FREE Full text] [CrossRef] [Medline]
    12. Chen C, Hu YH, Yen TY, Radwin RG. Automated video exposure assessment of repetitive hand activity level for a load transfer task. Hum Factors 2013 Apr;55(2):298-308 [FREE Full text] [CrossRef] [Medline]
    13. Theiss M, Scholl P, Van Laerhoven K. Predicting grasps with a wearable inertial and EMG sensing unit for low-power detection of in-hand objects. 2016 Presented at: The 7th Augmented Human International Conference 2016; February 25-27, 2016; Geneva. [CrossRef]
    14. Cordella F, Di Corato F, Zollo L, Siciliano B, van der Smagt P. Patient performance evaluation using Kinect and Monte Carlo-based finger tracking. 2012 Presented at: 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob); June 24-27, 2012; Rome. [CrossRef]
    15. Tung JY, Lulic T, Gonzalez DA, Tran J, Dickerson CR, Roy EA. Evaluation of a portable markerless finger position capture device: accuracy of the Leap Motion controller in healthy adults. Physiol Meas 2015 May;36(5):1025-1035. [CrossRef] [Medline]
    16. Grujic T, Bonkovic M. Measurement and analysis of human hand kinematics. International Journal of Medical, Health, Biomedical and Pharmaceutical Engineering 2015;9:97-102 [FREE Full text]
    17. Adamovich SV, Merians AS, Boian R, Tremaine M, Burdea GS, Recce M, et al. A virtual reality based exercise system for hand rehabilitation post-stroke: transfer to function. 2004 Presented at: The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2004; San Francisco p. 4936-4939. [CrossRef]
    18. Dipietro L, Sabatini A, Dario P. A survey of glove-based systems and their applications. IEEE Trans Syst, Man, Cybern C 2008 Jul;38(4):461-482. [CrossRef]
    19. Zhou H, Hu H. Human motion tracking for rehabilitation—a survey. Biomed Signal Process Control 2008 Jan;3(1):1-18. [CrossRef]
    20. Liarokapis MV, Artemiadis PK, Kyriakopoulos KJ, Manolakos ES. A learning scheme for reach to grasp movements: on EMG-based interfaces using task specific motion decoding models. IEEE J Biomed Health Inform 2013 Sep;17(5):915-921. [CrossRef] [Medline]
    21. Castellini C, Artemiadis P, Wininger M, Ajoudani A, Alimusaj M, Bicchi A, et al. Proceedings of the first workshop on peripheral machine interfaces: going beyond traditional surface electromyography. Front Neurorobot 2014;8:22 [FREE Full text] [CrossRef] [Medline]
    22. Wininger M, Kim NH, Craelius W. Pressure signature of forearm as predictor of grip force. J Rehabil Res Dev 2008;45(6):883-892 [FREE Full text] [Medline]
    23. Phillips SL, Craelius W. Residual kinetic imaging: a versatile interface for prosthetic control. Robotica 1999;23(3):277-282. [CrossRef]
    24. Li N, Yang D, Jiang L, Liu H, Cai H. Combined use of FSR sensor array and SVM classifier for finger motion recognition based on pressure distribution map. J Biomech Eng 2012 Mar;9(1):39-47. [CrossRef]
    25. Castellini C, Ravindra V. A wearable low-cost device based upon force-sensing resistors to detect single-finger forces. 2014 Presented at: 5th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics; August 12-15, 2014; São Paulo. [CrossRef]
    26. Lucaccini LF, Kaiser PK, Lyman J. The French electric hand: some observations and conclusions. Bull Prosthet Res 1966;10(6):31-51.
    27. Rasouli M, Ghosh R, Lee WW, Thakor NV, Kukreja S. Stable force-myographic control of a prosthetic hand using incremental learning. In: Conf Proc IEEE Eng Med Biol Soc. 2015 Aug Presented at: 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2015; Milano p. 4828-4831. [CrossRef]
    28. Sanford J, Patterson R, Popa D. Surface EMG and intra-socket force measurement to control a prosthetic device. 2015 Presented at: SPIE 9494, Next-Generation Robotics II; and Machine Intelligence and Bio-inspired Computation: Theory and Applications IX; June 25, 2015; Baltimore. [CrossRef]
    29. Radmand A, Scheme E, Englehart K. High resolution muscle pressure mapping for upper limb prosthetic control. 2014 Presented at: Proceeding of MEC - Myoelectric Control Symposium; August 19-22,2014; Fredericton p. 189-193.
    30. Carbonaro N, Anania G, Bacchereti M, Donati G, Ferretti L, Pellicci G, et al. An innovative multisensor controlled prosthetic hand. 2014 Presented at: IFMBE Proceedings; October 29-31, 2014; Paraná p. 93-96. [CrossRef]
    31. Wang X, Zhao J, Yang D, Li N, Sun C, Liu H. Biomechatronic approach to a multi-fingered hand prosthesis. 2010 Presented at: 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics; September 26-29, 2010; Tokyo p. 209-214. [CrossRef]
    32. Cho E, Chen R, Merhi L, Xiao Z, Pousett B, Menon C. Force myography to control robotic upper extremity prostheses: a feasibility study. Front Bioeng Biotechnol 2016 Mar;4:18 [FREE Full text] [CrossRef] [Medline]
    33. Radmand A, Scheme E, Englehart K. High-density force myography: a possible alternative for upper-limb prosthetic control. J Rehabil Res Dev 2016;53(4):443-456 [FREE Full text] [CrossRef] [Medline]
    34. Xiao ZG, Menon C. Towards the development of a wearable feedback system for monitoring the activities of the upper-extremities. J Neuroeng Rehabil 2014 Jan 08;11:2 [FREE Full text] [CrossRef] [Medline]
    35. Kadkhodayan A, Jiang X, Menon C. Continuous prediction of finger movements using force myography. J Med Biol Eng 2016 Jul 29;36(4):594-604. [CrossRef]
    36. Yungher D, Craelius W. Improving fine motor function after brain injury using gesture recognition biofeedback. Disabil Rehabil Assist Technol 2012 Nov;7(6):464-468. [CrossRef] [Medline]
    37. Yap HK, Mao A, Goh JC, Yeow CH. Design of a wearable FMG sensing system for user intent detection during hand rehabilitation with a soft robotic glove. 2016 Presented at: 6th IEEE International Conference on Biomedical Robotics and Biomechatronics; June 26-29, 2016; Singapore. [CrossRef]
    38. Xiao ZG, Elnady AM, Menon C. Control an exoskeleton for forearm rotation using FMG. 2014 Presented at: 5th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics; August 12-15, 2014; São Paulo. [CrossRef]
    39. Lukowicz P, Hanser F, Szubski C, Schobersberger W. Detecting and interpreting muscle activity with wearable force sensors. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Berlin: Springer; 2006:101-116.
    40. Amft O, Troster G, Lukowicz P, Schuster C. Sensing muscle activities with body-worn sensors. 2006 Presented at: International Workshop on Wearable and Implantable Body Sensor Networks (BSN’06); April 3-5, 2006; Cambridge p. 138-141. [CrossRef]
    41. Yungher DA, Wininger MT, Barr JB, Craelius W, Threlkeld AJ. Surface muscle pressure as a measure of active and passive behavior of muscles during gait. Med Eng Phys 2011 May;33(4):464-471. [CrossRef] [Medline]
    42. Bin Ambar R, Hazwaj BM, Abdul MB, Bin Ahmad MS, Bin Abdul Jamil MM. Multi-sensor arm rehabilitation monitoring device. 2012 Presented at: International Conference on Biomedical Engineering; May 28-30, 2012; Macau p. 424-429. [CrossRef]
    43. Morganti E, Angelini L, Adami A, Lalanne D, Lorenzelli L, Mugellini E. A smart watch with embedded sensors to recognize objects, grasps and forearm gestures. Procedia Engineering 2012;41:1169-1175. [CrossRef]
    44. Dementyev A, Paradiso J. WristFlex. 2014 Presented at: Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST ’14; October 5-8, 2014; New York p. 161-166. [CrossRef]
    45. Rekimoto J. GestureWrist and GesturePad: unobtrusive wearable interaction devices. 2001 Presented at: Proceedings Fifth International Symposium on Wearable Computers; October 8-9, 2001; Zurich p. 21-27. [CrossRef]
    46. Ravindra V, Castellini C. A comparative analysis of three non-invasive human-machine interfaces for the disabled. Front Neurorobot 2014 Oct;8:24 [FREE Full text] [CrossRef] [Medline]
    47. Ogris G, Kreil M, Lukowicz P. Using FSR based muscule activity monitoring to recognize manipulative arm gestures. 2007 Presented at: 11th IEEE International Symposium on Wearable Computers; October 11-13, 2007; Boston p. 1-4. [CrossRef]
    48. Sadarangani G, Menon C. A wearable sensor system for rehabilitation apllications. 2015 Presented at: 2015 IEEE International Conference on Rehabilitation Robotics; August 11-14, 2015; Singapore p. 672-677. [CrossRef]
    49. Garcia-Breijo E, Atkinson J, Gil-Sanchez L, Masot R, Ibañez J, Garrigues J, et al. A comparison study of pattern recognition algorithms implemented on a microcontroller for use in an electronic tongue for monitoring drinking waters. Sensors and Actuators A: Physical 2011 Dec;172(2):570-582. [CrossRef]
    50. Chih-Wei H, Chih-Chung C, Chih-Jen L. A practical guide to support vector classification. BJU International 2008 Jun;101(1):1396-1400.
    51. Swingler K. Applying neural networks: a practical guide. San Francisco: Morgan Kaufman Publishers, Inc; 1996.
    52. Shimada S, Fukuda K, Hiraki K. Rubber hand illusion under delayed visual feedback. PLoS One 2009 Jul 09;4(7):e6185 [FREE Full text] [CrossRef] [Medline]
    53. Antfolk C, D'Alonzo M, Rosén B, Lundborg G, Sebelius F, Cipriani C. Sensory feedback in upper limb prosthetics. Expert Rev Med Devices 2013 Jan;10(1):45-54. [CrossRef] [Medline]
    54. Ritter W, Kempter G, Werner T. User-acceptance of latency in touch interactions. In: UAHCI 2015: Universal Access in Human-Computer Interaction. Access to Interaction. Cham, Switzerland: Springer; 2015.
    55. Rothney MP, Schaefer EV, Neumann MM, Choi L, Chen KY. Validity of physical activity intensity predictions by ActiGraph, Actical, and RT3 accelerometers. Obesity (Silver Spring) 2008 Aug;16(8):1946-1952 [FREE Full text] [CrossRef] [Medline]
    56. Murphy SL. Review of physical activity measurement using accelerometers in older adults: considerations for research design and conduct. Prev Med 2009 Feb;48(2):108-114. [CrossRef] [Medline]
    57. Crouter SE, Dellavalle DM, Horton M, Haas JD, Frongillo EA, Bassett DR. Validity of the actical for estimating free-living physical activity. Eur J Appl Physiol 2011 Jul;111(7):1381-1389 [FREE Full text] [CrossRef] [Medline]
    58. Nowak DA, Hermsdörfer J, Topka H. Deficits of predictive grip force control during object manipulation in acute stroke. J Neurol 2003 Jul;250(7):850-860. [CrossRef] [Medline]
    59. Naik SK, Patten C, Lodha N, Coombes SA, Cauraugh JH. Force control deficits in chronic stroke: grip formation and release phases. Exp Brain Res 2011 May;211(1):1-15. [CrossRef] [Medline]

    ‎
    ADL: activity of the daily living
    ANN: artificial neural network
    FMG: force myography
    FSRs: force-sensing resistors
    IMU: inertial measurement unit
    IQR: interquartile range
    LDA: linear discriminant analysis
    RBF-SVM: Radial Basis Function kernel Support Vector Machine
    RMS: root mean square
    sEMG: surface electromyography
    WL: waveform length
    WS: window symmetry

    Edited by G Eysenbach; submitted 27.10.16; peer-reviewed by A Tognetti, HK Yap, K Van Laerhoven; comments to author 08.12.16; revised version received 09.02.17; accepted 10.02.17; published 16.05.17

    Copyright

    ©Zhen Gang Xiao, Carlo Menon. Originally published in JMIR Rehabilitation and Assistive Technology (http://rehab.jmir.org), 16.05.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Rehabilitation and Assistive Technology, is properly cited. The complete bibliographic information, a link to the original publication on http://rehab.jmir.org/, as well as this copyright and license information must be included.