All data collection and analysis were conducted by a single researcher, with input from a second researcher during the initial coding phase of the data. Data processing and analysis were carried out using MAXQDA 2022 (VERBI GmbH).

For qualitative data, including those from observations and focus groups, an evaluative qualitative content analysis was used [32,33]. This method assessed, classified, and evaluated content, akin to a content-structuring qualitative content analysis. However, in an evaluative content analysis, additional categories are generated to allow researchers to rate the material on the selected dimensions [33-35]. In this research, these assessment dimensions were defined as being less than, equal to, or better than the original BBT measurement instrument. The predefined coding categories in the content analysis process were grounded in 5 key dimensions of clinical utility. Subcategories and guiding questions were subsequently developed for each dimension, serving as the foundation for the observation studies and focus group (see Table 1).

The aim of these observations was to evaluate the clinical utility of the dBBT. Observations in general can provide important real-time data on behavior [36]. The object of observation was the use of the dBBT to measure hand dexterity by a therapist, with another participant as the person to be tested. Each occupational therapist completed the hand dexterity assessment with the dBBT as a test administrator once, whereas another participant took the role of the tested person. The exact procedure was as follows: following the standardized procedure as defined for the original BBT [4], the test administrator read the test instructions to the person to be tested and performed the hand dexterity measurement once on the writing hand of the tested person. The whole exercise session was observed by the researcher, using the previously developed observation guide.

As the observation sequence lasts only a few minutes (including the start-up of the dBBT by the therapist, instruction of the participant by the therapist, practice run following the test protocol, and the actual dexterity test), there was limited time for extensive note-taking. Therefore, the researcher opted for a quantitative assessment of the observations. The following aspects of the five dimensions of clinical utility were assessed, which were directly observable and comparable to the original BBT using a three-part scale of less than, equal to, or better than: (1) time for preparation by the therapist, (2) time for patient instruction by the therapist, and (3) time for the person to be tested to understand the test task. Further assessment points covered possible application problems: (4) problems during preparation (which ones, severity, and consequences), (5) problems during implementation (which ones, severity, and consequences), and (6) open questions from the therapist (see Multimedia Appendix 1).

Each session was video recorded using a Dell Latitude 5480 Laptop, and data were collected using the aforementioned observation protocol.

The data were transcribed verbatim, anonymized, and coded using predefined categories (see Table 1), and additional categories were developed based on the data (for potential improvements to the dBBT). Subsequently, the data were analyzed using verbal-interpretative methods based on the categories, and key statements were presented accordingly. All these steps were carried out by the same person, the researcher (EP).

The usability of dBBT was then assessed quantitatively with the SUS (see Multimedia Appendix 2). The participants (N=4) completed the SUS directly after using the dBBT.

The SUS is one of the most frequently used questionnaires for evaluating the usability of eHealth applications [37]. Even with a very small sample, the SUS provides valid results about whether an application has problems in the area of usability [38].

The process for computing a SUS score was following: (1) subtract 1 from the Likert ratings for odd-numbered items, (2) subtract the Likert ratings from 5 for even-numbered items (each item score will range from 0 to 4), and (3) sum the numbers and multiply the total by 2.5 [24]. This resulted in SUS scores ranging from 0 to 100. The following SUS score ratings were used in this study: scores ≥52 represented “OK” usability, ≥73 represented “good” usability, and ≥85 represented “excellent” usability [39].

The mean SUS score from 3500 surveys within 273 studies was 72 [39]. It is recommended to the report mean, SD, median, CI, and P value (Shapiro-Wilk) in addition to the SUS score [24].

Following the completion of the usability questionnaire, a focus group session was conducted with all 4 occupational therapists. The aim was to evaluate the clinical utility of the dBBT compared to the original version and to gather suggestions for potential system improvements.

The focus group followed a predefined guideline (see Multimedia Appendix 3), developed in accordance with qualitative research standards [40] and reviewed by a second researcher. This guideline was structured around the dimensions of clinical utility (see Table 1), with the assessment criteria for the evaluative content analysis (less than, equal to, or better than the BBT) also included.

An audio recording was made during the focus group session. This audio file, an observation protocol created by the researcher following the focus group, and notes from the guideline were used for data analysis. For analysis, an evaluative qualitative content approach was chosen [33]. An initial coding frame was derived from the focus group guideline and refined as more data were analyzed. This involved identifying patterns, assigning codes, and establishing themes and subthemes from the coded data [41]. Ultimately, the data were interpreted verbally according to categories and presented alongside relevant statements.

We used a convergent mixed methods design, integrating the findings from both qualitative and quantitative data [42]. Following separate collection and analysis of quantitative and qualitative information, the 2 data sets were combined.

The purpose of merging the results was twofold: (1) to enhance the validation of clinical utility and usability and (2) to identify potential optimization strategies.

The following data were included in the merging process:

Subsequently, recommendations for future improvements were extracted from the compiled data and presented.

The acceptance of the newly developed prototype dBBT differed from the original BBT in several ways. First, simplicity was enhanced by eliminating the need for manual counting (with the original BBT, the therapist has to count the transported blocks by hand to obtain a final test result; on average, 80-100 blocks have to be counted by hand, which makes the evaluation more time-consuming than the BBT itself) and by automating time measurement instead of using a stopwatch: “for me the automatic time measurement easier than dealing with a stopwatch - because I just never use a stopwatch otherwise” (p.25).

Second, the trustworthiness of the results provided by the dBBT was emphasized, ensuring that the results are credible to clients: “and above all, that the result [note: for clients] is credible - if a ‘device’ measure that” (p.125).

Another important finding was the clinical and professional appearance of the dBBT, which was documented in several statements, such as “[the dBBT] transports a higher level of professionalism to the external environment” (p.128).

The evaluative analysis showed that the dBBT achieved higher acceptance compared to the BBT. As shown in Table 2, a total of 89% (33/37) of statements by the occupational therapists showed a higher acceptance of the dBBT than the original BBT. Particularly, all occupational therapists judged the “professionalism” (defined as whether the test looks professional [13]) of the dBBT as higher than that of the original BBT.

The clarity of the components was rated as equal to the BBT, but the transportability of the dBBT was rated as lesser than that of the BBT. This is because the original BBT can be folded to half its size, whereas the dBBT does not offer this feature: “possibly it is bulkier the dBBT” (p.92).

Regarding the clarity of required components (“Is it easy to handle in terms of the number of components required?”), the dBBT was rated as equal to the BBT: “so there is no difference to the BBT - except that you don’t have to assemble the dBBT - then the dBBT is even rather clearer.” (p.116).

The dBBT is slightly heavier than the BBT. However, the therapists came to the conclusion that the higher weight of the dBBT is irrelevant because “normally the BBT will not be transported either - it will be in the therapy room” (p.112).

According to the evaluative analysis results for portability, 67% (6/9) of statements reported that the portability of the dBBT was equal to that of the BBT. The remaining statements (3/9, 33%) suggested that the dBBT had less transportability than the BBT (Table 2).

In most of the statements in the energy and effort category and its subcategories (physical exertion, ease of test execution, and ease of learning), no difference was found between the dBBT and BBT: “so certainly none of the three aspects [note: of energy and effort] shows somehow more effort or energy than with the BBT” (p.108) and “I would see it the same way” (p.109).

The energy required to perform the measurement with the dBBT and to learn how to perform dexterity measurement with the dBBT was also rated as equal to that for the BBT: “the physical effort for the test administrator is the same as for a measurement with the BBT” (p.105) and “the effort required for implementation is the same, learning is just as easy as with the BBT” (p.110).

In the evaluative analysis results for energy and effort in Table 2, a total of 86% (6/7) of the statements reported equal energy and effort in using the dBBT in comparison to the BBT.

In the subcategory of time for evaluation, the dBBT was rated as better than the BBT by all participants. The therapists appreciated the simplification resulting from the omission of counting the blocks, especially when evaluating multiple patients consecutively: “then I would also prefer the digital BBT - because it would be tedious to count and check it all the time” (p.52) and “slightly less time for the evaluation of the test with the dBBT than with the BBT” (p.96).

In the subcategories for learning test execution, preparation, and implementation, the dBBT was rated as equal to the BBT (Table 2): “I only see a little less time for the evaluation - everything else is the same” (p.86) and “the time to learn how to perform the test cannot be different” (p.98).

The preparation and execution of the hand function measurement with the dBBT were evaluated as equal to the BBT: “you have to plug in the dBBT and try it out, I guess – but at the BBT I have to check whether the stopwatch is working” (p. 89-90) and “The preparation is also no different than with the BBT - put it there…” (p.88).

In the evaluative analysis results for time in Table 2, a total of 83% (20/24) of statements in the time category rated the dBBT as equal to the BBT, whereas 17% (4/24) rated the dBBT as better than the BBT.

The BBT is available for purchase at prices ranging from US $240 to US $450. The new dBBT was estimated to cost a fraction of this amount. The manufacturing costs are estimated to be less than US $65. This information was given to the test participants before the discussion of costs.

The ongoing costs for dBBT were estimated to be relatively lower (if one classified the power consumption as negligible): “less are the running costs with the dBBT - because I don’t need a battery for the stopwatch” (p.78).

Regarding necessary training, all therapists were receptive to the fact that the technical device requires minimal additional effort due to its straightforward operation. However, it was noted that an introduction to the functions of the dBBT was required initially: “I think it balances out - since you don’t have to count with the dBBT. that’s not necessary. but the three buttons and plugging the device in [to the power supply] are the ‘more’ - but once you know it, you can do it anyway” (p.17) and “you have to be told at least once what, for example, the black button is for and so on” (p.19).

At the same time, however, familiarity with using a stopwatch, which was required for the original BBT, had decreased: “the stopwatch I need to use in the original, I also need to practice” (p.22). Therefore, the expense of required training was rated as equal to the BBT.

The required qualification for executing a dexterity measurement with the dBBT was rated as equal to the BBT: “the qualification for the admin is the same, as the standardized test protocol is just as possible with the dBBT as with the BBT” (p.5).

In total, 50% (11/22) of statements in the cost category rated the dBBT as equal to the the original BBT and 50% (11/22) rated it as better than the BBT. The purchase cost of the dBBT was considered better than the BBT, whereas in all other subcategories, the dBBT was considered equal to the BBT.

The evaluative qualitative content analysis used selected dimensions (less than, equal to, and better than) to assess the clinical utility of the dBBT compared to the original BBT measurement instrument. In summary, 45% (45/99) of all statements reported an equivalent assessment of the dBBT. Furthermore, 48% (48/99) of all statements rated the clinical utility of the dBBT as better than that of the dBBT, whereas only 6% (6/99) of the statements rated the dBBT as less than the BBT (refer to Table 2).

Therefore, in this study, the dBBT consistently appeared to have at least comparable, and often superior, results in terms of clinical utility compared to the BBT.

The most commonly suggested improvement for the system pertained to the shape of the blocks. Participants expressed that making the edges less sharp would enhance the ease of handling the blocks: “The cubes are more difficult to grip [the edges are sharper than on the BBT]…“ (p.10) and “...Edges are sharper or very sharp in the dBBT, which means that they are arranged more closely in the box and it is harder to grip them” (p.13).

The second point focused on signaling the end of the test period. Currently, the dBBT uses 2 LEDs, placed on the partition, that change from green to red when the 60-second test period concludes. However, the participants preferred an audible signal, as it would provide a clearer notification for both the person being tested and the therapist: “…the stopwatch beeps so nicely [note: on the original BBT] - then the patient knows that the measurement period is over” (p.80) and “…that would also be good if this prototype can do that…” (p.81).

Additionally, it was observed in 3 (75%) out of 4 instances, the visual signal for the test ending was not perceived by either the test administrator or the person being tested.

The third point became evident from observations alone. During the test, 2 (50%) out of the 4 occupational therapists were distracted by the display, which constantly showed the elapsed time and the number of blocks currently being counted. The standardized test procedure requires occupational therapists to ensure that the person being tested (1) crosses the partition with their fingers and (2) transports only one block at a time. However, constantly checking the changing display diverted the therapists’ attention from this task. None of the participants in the focus group mentioned this issue. It is possible that this observed behavior is a result of using a new device, and whether this problem persists with continued use of the dBBT cannot be conclusively answered by this study alone.

No other subthemes regarding future changes were found.