This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Rehabilitation and Assistive Technology, is properly cited. The complete bibliographic information, a link to the original publication on http://rehab.jmir.org/, as well as this copyright and license information must be included.
Functional arm movements generally require grasping an object. The possibility of detecting and counting the action of grasping is believed to be of importance for individual with motor function deficits of the arm, as it could be an indication of the number of the functional arm movements performed by the individuals during rehabilitation. In this exploratory work, the feasibility of using armbands recording radial displacements of forearm muscles and tendons (ie, force myography, FMG) to estimate hand grasping with healthy individuals was investigated. In contrast to previous studies, this exploratory study investigates the feasibility of (1) detecting grasping when the participants move their arms, which could introduce large artifacts to the point of potentially preventing the practical use of the proposed technology, and (2) counting grasping during arm-reaching tasks.
The aim of this study was to determine the usefulness of FMG in the detection of functional arm movements. The use of FMG straps placed on the forearm is proposed for counting the number of grasping actions in the presence of arm movements.
Ten healthy volunteers participated in this study to perform a pick-and-place exercise after providing informed consent. FMG signals were simultaneously collected using 2 FMG straps worn on their wrist and at the midposition of their forearm, respectively. Raw FMG signals and 3 additional FMG features (ie, root mean square, wavelength, and window symmetry) were extracted and fed into a linear discriminant analysis classifier to predict grasping states. The transition from nongrasping to grasping states was detected during the process of counting the number of grasping actions.
The median accuracy for detecting grasping events using FMG recorded from the wrist was 95%, and the corresponding interquartile range (IQR) was 5%. For forearm FMG classification, the median accuracy was 92%, and the corresponding IQR was 3%. The difference between the 2 median accuracies was statistically significant (
This study reports that grasping can reliably be counted using an unobtrusive and simple FMG strap even in the presence of arm movements. Such a result supports the foundation for future research evaluating the feasibility of monitoring hand grasping during unsupervised ADL, leading to further investigations with individuals with motor function deficits of the arm.
Individuals with motor function deficits of their arm (eg, individuals with stroke and a hemiparetic arm) often involuntarily avoid using their weak arm during the activities of daily living (ADL)[
Some studies have used accelerometer-based devices to capture gross movements of the arm [
Devices with the ability to detect grasping motions, which is generally required during functional arm movements, could potentially provide a more suitable indication of functional use of the upper limb [
Currently, commercial devices capable of detecting grasping motions do exist. They are mostly based on either a vision-based approach or using a wearable technology approach such as wearing data gloves. The vision-based systems, such as Microsoft’s Kinect [
In addition to the above-mentioned commercially available technologies, the academic community has investigated different approaches to classify grasping and other hand gestures. One of these approaches is based on surface electromyography (sEMG) recorded from the forearm [
An alternative approach to detect grasping is force myography (FMG). FMG is a technique that uses sensors to capture displacements of muscles, skin, and tendons [
The use of FMG can be dated back as early as the 1960s, when FMG was proposed for controlling a single-degree prosthetic terminal device [
FMG signals to detect hand movements are generally extracted from the middle of the forearm where large radial displacements can be recorded [
The large majority of studies on FMG for the upper extremities presented in the literature considered exploratory tests in very controlled scenarios, where healthy volunteers were asked to move their hand or wrist while maintaining a fixed elbow position [
As a fundamental step toward the development of a technology suitable to detecting grasping motions in the presence of arm movements as an indirect estimation of functional arm movements, this study investigates the ability of using FMG to count the number of grasping motions during a series of pick-and-place (PAP) actions. Two wireless FMG straps were prototyped and placed close to the wrist and on the forearm of 10 healthy individuals for this study.
A single FSR sensor has 2 terminals: one terminal is connected to a common analog input pin of a microcontroller (Atmega 328p from Atmel) with an internal pull-up resistor (37.5 kΩ) equipped, and the other terminal is connected to a digital control pin as shown in
The analog input pin takes the reading of the signal and converts it into a 10-bit unsigned integer value. Since only 1 analog pin was used for sampling, the signal of each FSR was sampled sequentially. The order of sampling was determined by the digital control pin. When the selected FSR signal was ready to be sampled, the corresponding control pin was set to low, and the other control pins were set to be in high impedance states. At any single moment, only 1 control pin would be set to low and others would be changed to high impedance state in order to guarantee independent sampling. This configuration used a single analog pin with internal pull-up resistor in order to obtain the most simplified design under the constraints of the selected inexpensive microcontroller.
The sampled data were transmitted wirelessly to a personal computer using a generic Bluetooth module (HC-05). A custom-made application with real-time signal display was developed in LabVIEW on a personal computer for querying and storing the sample data. When sampling began, the application sent a command to the microcontroller to retrieve a set of FSR data at every 100 milliseconds (10 Hz) as proposed by Amft et al [
An experimental protocol was designed to capture both wrist and forearm FMG signals simultaneously during a PAP exercise. Before the experiment, the forearm FMG strap was donned on the belly of the right forearm of a volunteer with the help of the research assistance. The wrist FMG strap was instead donned on the distal end of the forearm (next to the ulna styloid process, see
Once the straps were donned, the volunteer was asked to fully extend the fingers and then make a fist 3 times, while the research assistant monitored the raw signal through visual feedback from the display of the LabVIEW application. This hand action was shown to be able to generate a clear and visually distinguishable FMG pattern for healthy individuals [
In the experiment, the volunteers sat in front of a table as illustrated in
In order to capture FMG signals in the presence of various arm movements, 3 PAP sequences were designed for the participants to perform. These PAP sequences required the coordination of the shoulder, the elbow, wrist, and hand. Therefore, the FMG patterns that were associated with some of the elbow flexion/extension, forearm pronation/supination, wrist flexion/extension/abduction/adduction, hand opening/closing, and the overall arm motion would be captured. Some examples of the captured movement during the PAP sequences are shown in
In the first sequence, the participant was asked to pick up the object from the start location and place it onto the target locations at a pace comfortable for them. Then the participant retrieved their hand to the start location without the object. Next, the participant picked up the object from the current location, and returned it to the start location. Finally, the participant released the object completely before starting the next PAP action. In total, each participant performed 10 PAP actions for the first action sequence as shown in the left picture of
Wireless FSR straps: (a) wireless FSR strap for the forearm and (b) wireless FSR strap for the wrist.
Schematic for FMG signal extraction and data transmission. The microcontroller sampled the signal from 8 FRS sensors, and the data were sent wirelessly to the computer through the Bluetooth transmitter module.
FSR straps placement: (a) forearm supinated view and (b) forearm pronated view.
Experimental setup. The start location is shown in gray, the five target locations are shown in green, and the object for grasping is shown in yellow.
Examples of upper limb position during the PAP sequence: (a) grasping the object from start position; (b) transporting the object from start position to target location 4; and (c) transporting the object from target location 2 to 4.
PAP action sequence. The red circles indicate the target positions, the arrows indicate the direction of the PAP actions, and the numerical labels indicate the orders of the steps in each sequence: (a) first PAP action sequence; (b) second PAP action sequence; and (c) third PAP action sequence.
Data processing sequence.
Ten healthy volunteers aged between 21 and 42 years participated in the experiment. Each participant signed an informed consent form (approved by the Office of Research Ethics, Simon Fraser University) before entering the study. Their wrist and forearm belly circumferences were recorded for performance analysis and are shown in
Participant statistics.
Wrist circumference (cm) | Forearm belly circumference (cm) | |
1 | 16.5 | 28 |
2 | 19 | 27 |
3 | 17.5 | 27 |
4 | 16 | 27 |
5 | 17.6 | 28.5 |
6 | 15.5 | 27 |
7 | 16 | 17 |
8 | 18 | 26 |
9 | 16.5 | 26.5 |
10 | 15.5 | 28 |
Average | 16.81 | 26.2 |
SD | 1.11 | 3.15 |
The data collected from each participant consisted of both wrist and forearm FMG positions. The 2 streams of data were processed through identical but independent treatments. The collected data were first divided into training and testing sets. The training set data consisted of the first 30 PAP actions, and the testing set consisted of the rest of the 120 PAP actions data sets. The training set was used for extracting relevant statistical information about the signals and for generating a classifier model. The testing set was used for examining the generalized performance of the classifier for detecting grasping. The overall data processing sequence is shown in
As shown in
Next, feature extraction was performed. The raw FMG data were considered as primary feature of the signal. Three additional signal features, namely the root mean square (RMS), waveform length (WL), and window symmetry (WS), were extracted from the raw data with a 300 ms window and a step size of 100 ms.
RMS is the averaged signal magnitude of each window and its equation is shown in
WS is the difference between the average of the first
Under the supervised classification scheme, the classifier needs to be trained using true label obtained from external source. In our case, the labeling signal was the one recorded by the FSR sensor placed on the thumb. This signal measured the amount of contact force between the object and the thumb. If the contact force was less than 2% of the maximum, then the corresponding FMG data was labeled as nongrasping (class 1); otherwise, the FMG data was labeled as grasping (class 2).
Among different supervised classifiers, linear discriminant analysis (LDA) classifier using Fisher discriminant criteria is one of the most widely used for analysis. LDA fits a multivariate normal density to each class with a pooled estimate of covariance. It is capable of revealing linear separability of the signal features. Additionally, LDA is computationally efficient and suitable to be implemented in a microcontroller [
The output of the LDA classifier was the predicted state of the hand. In order to count the number of grasping actions, the transition from nongrasping to grasping state needed to be identified. This transition could be detected by subtracting the current state output with the previous one. A positive result indicated a grasping action has occurred. However, the accuracy of such a counting method could be sensitive to any small glitches (misclassifications over a short period, eg, <1s) in the classification data output stream. Hence, an average filter was applied to smooth out the output stream of the classifier. The window size of the filter could affect the overall performance in terms of the counting accuracy and the delay. Therefore, the effect of different window sizes on the counting performance was examined (see Results section).
Equations for FMG feature extraction: (a) Root mean square, (b) Waveform length, and (c) Window symmetry.
Boxplot for FMG classification accuracies of the 10 participants. The bottom and top of each box are the first and third quartiles of the data set, respectively. The band inside the box is the median. The ends of the dashed lines (whiskers) are the minimum and maximum of the data. The red and blue boxes indicate classification accuracies related to FMG collected from the wrist and forearm, respectively. (a) Accuracies computed using training set data. (b) Accuracies computed using testing set data.
Classification result comparison: (a) accuracy comparison using wrist FMG and (b) accuracy comparison using forearm FMG.
Regression plots for force-myography (FMG) classification accuracies of the 10 participants: (a) wrist FMG classification accuracies versus wrist circumference; (b) forearm FMG classification accuracies versus wrist circumference; (c) wrist FMG classification accuracies versus time per action; and (d) forearm FMG classification accuracies versus time per action.
Boxplots for grasp counts versus number of filtered samples: (a) result generated using wrist FMG and (b) result generated using forearm FMG.
Boxplots for percentage error of grasp counts versus number of filtered samples: (a) result generated using wrist FMG and (b) result generated using forearm force-myography.
The overall performance of the proposed method was evaluated based on 2 metrics: the classification accuracies and percentage errors of the grasp count obtained from the test set.
The classification accuracy was calculated based on the sum of the correctly classified sample over the total number of samples. The difference of classification accuracies between wrist and forearm FMG using LDA was evaluated. Also, the performance of LDA was compared with other 2 popular classifiers, namely the Radial Basis Function kernel Support Vector Machine (RBF-SVM) and the 2-layer Artificial Neural Network (ANN). In addition, the correlations between the accuracies and action speeds, as well as the size of the wrist or forearm were assessed using a regression method.
The percentage error of the grasp count was based on the result of the absolute difference between the predicted and the expected counts over the expected one. The expected count for the test set was 120 in this study. The difference between the errors of the wrist and forearm FMG approaches was also assessed.
Paired sign test and Kruskal-Wallis test were used for examine the statistical significance of the obtained results. All the statistical analysis was performed using the same significance level (alpha) of .05.
The LDA classification accuracies of the 10 participants for nongrasping (class 1) and grasping (class 2) an object are shown in
For the FMG recording from the wrist, the median training accuracy (see the top of
Similar results were obtained for FMG data recorded from the forearm: the median training accuracy was 95% and the corresponding IQR was 4%. The median testing accuracy was 91% and the corresponding IQR was 3%.
The difference between the medians of the wrist and forearm FMG testing accuracies was 4%. With a
The second pair of results in
The third pair of results in
The performance of using LDA was compared with the ones of RBF-SVM and ANN. Standard model generation procedures for SVM and ANN, which are described in [
In order to examine if the size of the participants’ limbs and action speeds influence the classification accuracies, 4 regression plots were generated and shown in
The first row of
In order to assess the effect of the filtering window, the grasp counts of the participants were recomputed using different window sizes.
Without averaging (sample size equals 1), the numbers of grasping were overestimated by a large margin for all the participants’ data. As the size of the averaging window increased, the counts were closer to the expected value in general. However, when the size continued to increase, the count became increasingly underestimated. On the basis of the result shown in
As shown in
The LDA classifier was selected as the main classifier for this study, and its performance was as good as the more computational-intensive SVM and ANN classifiers. The LDA classification result (see
The regression method was used to test if the size of the participants’ limbs and action speeds influence the classification accuracies. The results of the test showed there were no statistically significant correlation between the accuracies and the 2 factors. These results suggested the performance of the FMG method was independent of the limb size and action speed in this study. However, since the participants in this study were healthy individuals, the variations of their physical status and action performance were expected to be insignificant. For future investigation, grasping at various speeds by individuals with different limb sizes should be considered.
The grasping counts were extracted from the filtered output of the LDA classifiers. As shown in
This study investigated the capability of using FMG to predict and count grasping actions with healthy individuals. Therefore, the knowledge obtained from this study can be directly applied to the applications in which healthy individuals are the wearers of the FMG bands. An example of such an application could be for monitoring the repetitive hand activity level of a worker during load transfer tasks. However, for rehabilitation applications, as the targeted population will be the people with a weak arm, such as individuals recovering from stroke, further studies are needed to examine the transferability of the result of this study. For example, in poststroke rehabilitation, the targeted wearers are often seniors with limited mobility and range of motion. Their muscles are normally much weaker than the ones of healthy individuals, and may even have significantly deteriorated if the individuals are chronic stroke patients. These characteristics posed questions on whether the proposed FMG method could be used with such a population. Currently, there is lack of in-depth studies that examine the FMG pattern characteristic of stroke patients or people with a weak arm. However, the pilot investigation of Yungher and Craelius showed that the grasping force could be regressed from the forearm of stroke patients (n=4) with mild to moderate spasticity [
The possibility of detecting and counting grasping in the presence of arm movements (PAP exercise) was explored using wrist and forearm FMG strap prototypes. The 2 main performance parameters that were considered were classification accuracy for detecting grasping and percentage error for counting grasping. A high median grasping prediction accuracy was obtained from 10 subjects (95% and 91% for FMG recorded from the wrist and forearm, respectively). A low median grasping count error was also found (1% and 2% for wrist and forearm, respectively). These results provide evidence that FMG-based straps could be used to monitor grasping activities during functional arm movements in a controlled environment. This work poses the foundation for future studies investigating the applicability of using FMG to detect grasping in activities of the daily living first with healthy participants and then with individuals with a weak arm (eg, seniors, individuals with a hemiparetic arm resulting from stroke).
activity of the daily living
artificial neural network
force myography
force-sensing resistors
inertial measurement unit
interquartile range
linear discriminant analysis
Radial Basis Function kernel Support Vector Machine
root mean square
surface electromyography
waveform length
window symmetry
The authors thank the members of the MENRVA lab, Dr Janice Eng, and Ms Lisa Fraser for their advices and feedback on the use of the proposed technology. This work was supported by the Natural Sciences and Engineering Research Council of Canada, the Canadian Institutes of Health Research, and the Michael Smith Foundation for Health Research.
ZGX prototyped the FMG straps, implemented data collection software, performed experiments, analyzed the experimental results, and participated in the manuscript preparation. CM, Principal Investigator of this research, contributed to the design of the study, supervised the work, participated in the interpretation of the results and contributed in writing the manuscript. All authors read and approved the final manuscript.
The Principal Investigator, Carlo Menon, and members of his research team have a vested interest in commercializing the technology tested in this study, if it is proven to be successful and may benefit financially from its potential commercialization. The data are readily available upon request.