Given the global phenomenon of aging populations, strategies to reduce the risk of physical and cognitive decline and the associated consequences on the well-being of older adults and their ability to cope with everyday life are urgently needed [1]. One resource in this context might be the use of the so-called social robots. According to Anzalone and colleagues [2], social robots can be understood as “machines that humans should perceive as realistic, effective partners, able to communicate and cooperate with them as naturally as possible interestingly enough.” The acceptance of social robots and their potential to promote the well-being of older adults have been explored and demonstrated in several studies [3-8]. Most of the robots studied are animallike, with PARO [9], a seal-shaped robot, being a prominent example [2-5,10,11]. However, animallike companion robots are not multifunctional and their interactions are not sufficient for those who require care and support. A study comparing animallike and humanlike social robots in group settings provided the first evidence that humanlike robots have greater effects on cognitive training than animallike robots [12], which brings humanlike social robots into the focus of research for group activities for older adults. This so-called third generation of social robots, including Nao, Pepper, QT, Sophia, Jack, LOVOT, or Tessa [12-18], continue to evolve, as new software is developed and released into the market [13]. Their humanlike forms [19] and integrated voice capability allow for interactions through facial expression, gestures, and voice. Thus, these robots can support cognitive and physically stimulating exercises, which in combination, achieve the best results in maintaining cognitive abilities in older adults [1].

Few studies [7,12,13,20-25] have investigated whether older adults actively engage in and experience positive moods during these activities. Since mood and engagement are crucial for the effectiveness of such group activities with a humanlike social robot, this study aims to explore these 2 constructs empirically. In doing so, we chose the method of systematic behavioral observation, because self-report data of older adults in nursing homes are often difficult to obtain and might interfere with their experience of the activity itself [9]. As no suitable observational coding scheme could be identified in the literature, a second aim of this study was the development and piloting of an observational coding scheme. In summary, this pilot study addresses the following questions: (1) can the engagement and mood of older adults in a robot-assisted group activity be assessed through systematic behavioral observation? and (2) do older adults actively engage in a robot-assisted group activity and what mood (ie, positive or negative affect) can be observed in the group during such a robot-assisted group activity?

A review identified group activities for older adults assisted by social robots in 5 domains: affective therapy, cognitive training, social facilitation, companionship, and physiological therapy [7]. Three studies [12,13,20] showed a great potential of humanlike robots in group activities, with broader functionalities for physical activities. The first indications that older adults liked to participate in robot-assisted group activities for physical activities are shown in [21,22]. The robot NAO was found suitable to be used in group settings for moving, memory training, entertainment, music, dancing, and games [23]. One study showed that older adults in a nursing home prefer walking with a robot rather than walking alone [24]. Another study showed that older adults actively participated in robot-assisted cognitive therapy and physiotherapy sessions, and a trend toward improved neuropsychiatric symptoms, reduced apathy, and higher quality of life was observed [25]. Although these studies [12,13,20-25] provide first insights into the acceptance of humanlike robots assisting in group activities of older adults, we identified only 1 study that systematically developed and used an observation system for examining the engagement of groups of older adults during activity sessions assisted by a humanlike social robot [13]. Even though this observation system indicates that systematic observation is a fruitful methodological approach in this research context, it does not fully capture the psychological constructs of engagement and mood that are at the center of our pilot study and that are usually captured using self-report surveys.

We considered this as a pilot study because we developed and tested the applicability of a systematic observation system for rating participants’ engagement and mood during robot-assisted group activity sessions for older adults in nursing homes.

A pool of about 200 nursing homes in the German-speaking part of Switzerland were contacted by telephone and invited to participate in this study. Four nursing homes expressed their interest, and they were selected to participate in this observational field study. All participating nursing homes provide various services for leisure activities and physical and cognitive stimulation. They offer accommodation and care to 50-160 residents and provide specialized dementia care. As part of this study, the management of each nursing home agreed to co-organize a robot-assisted group activity together with the research team and made nursing staff available to accompany residents to the session. Residents of the participating nursing homes were informed about the robot-assisted group activity and the study procedure, and they were invited to participate on a voluntary basis.

The robot used for the robot-assisted group activity was the NAO robot from SoftBanks Robotics [35]. We used the software of Avatarion [36] developed by Smart Companion [37]. The software was developed in collaboration with experts for leisure activities and physical and cognitive stimulation for older adults, specifically for robot-assisted leisure activities during their care. In this study, we used 3 software modules that support common elements of group activities for older adults: singing, storytelling, and gymnastics.

Figures 1-2 illustrate the robot-assisted physical exercise sessions in 2 different nursing homes. Photos were taken during 2 robot-assisted group activity sessions in 2 different nursing homes. The pictures show the NAO robot demonstrating movements with its hands and residents participating in this physical exercise by imitating the robot’s movements.

According to Swiss law this study did not require formal ethics approval and was thus exempt from formal ethics review. For more information please see the corresponding section of the Swiss Human Research Act. The participating nursing homes consented to this study and informed the residents in advance about the robot-assisted activity sessions. Participation in the robot-assisted activity sessions was voluntary. Consent was obtained for using the anonymized photographs in this paper.

Robot-assisted group activity sessions were offered in the participating nursing homes in July and August 2019. Chairs and free spaces for wheelchair users were arranged in a way that allowed the participants to see the NAO6 robot that was placed on a table. All robot-assisted group activity sessions took place 1 hour before lunch. Participating residents arrived independently. During the session, 2-5 health care professionals were available in the room for the general support of the residents. All group activity sessions in this study were conducted by research team members and lasted 1 hour, with the actual robot-assisted group activity taking about 30 minutes.

The activity session was structured in 3 parts. First, a representative of the research team welcomed the residents, explained the procedure of the session, reminded them that participation was voluntary, informed them about data protection issues, asked for their approval regarding video recording, and introduced the persons involved. Second, a technical expert from the Smart Companion team started the robot program. The participants first performed a gymnastics exercise, then sang a song together with the robot, and toward the end of the session, they listened to a story. The research team did not interact with the participating residents during these sessions. Third, an additional exercise was conducted by the robot to get the residents in the mood for lunch. This exercise was not recorded and was not part of this study, as it did not aim at their physical and cognitive stimulation. At the end of the activity session, the robot wished the residents bon appétit and said goodbye. Finally, the research team also said goodbye and thanked the residents for their participation.

To systematically analyze residents’ engagement and mood in group activities with the robot, sessions were recorded on video. Video recording has the advantage that, for example, behavior can be observed unobtrusively and participants do not have to be bothered afterwards, as they would be when using interviews. Further, for a high number of residents in nursing homes, other forms of data collection such as surveys present an inaccurate form of assessment, since retrieval, reporting, and ranking of relevant information may be compromised. Therefore, almost all assessment techniques for people with dementia rely on behavior observation [27]. Video recording was done in a way such that residents should not be disturbed, and the Hawthorne effect could be reduced [38]. Hence, short video clips of all 3 exercises were filmed as discreetly as possible. The video clips lasted between 30 seconds and 3 minutes and were distributed across the whole duration of the 3 exercises. The time of the start of the clip in the exercise was chosen randomly. For practical reasons, video clips were recorded with a smartphone camera. For ethical reasons, we collected no personal data such as the age of participants as well as the presence and severity of dementia symptoms. The videos only show the number of participants during each session.

For a structured observation of engagement and mood during a robot-assisted group activity, an observation system was developed. The observation system builds on existing observation systems for engagement and mood of individuals but was adapted for direct observation in a group setting. For example, in studies of children’s engagement during one-on-one interactions with robots [30,31], the variables used to measure engagement were direction of gaze, facial expressions, responses, or gestures. Another study related to children with autism spectrum disorders interacting with social robots [32] used measures of engagement based on nonverbal behavior focusing on social and antisocial behaviors. Another system used for older adults observed in a session with a social robot includes measures of engagement and mood targeted to a setting with small groups and a facilitator. Engagement was measured by someone leaning toward the collaborator, and mood was assessed by movements that were accompanied by a positive or negative affect [27]. Further, we analyzed the Observational Measurement of Engagement. This tool is based on a self-identity questionnaire and the 3 dimensions of observational measurements, namely, duration, attention, and attitude. This instrument did not meet all our needs, as we had a predefined duration of an interaction, and attitude was not the focus of our study. However, attention was in our interest and was included in our observation instrument. Another study measured affect and social interaction during a game [33]. Positive affect included smiling and clapping, and negative affect included sadness and anger [33]. Both studies show the relevance of gaze direction for capturing engagement and of observable behaviors for capturing positive and negative affect.

Extending the previous research, we aimed to assess participants’ engagement in robot-assisted group activities. We adapted an established rating system that has been used for the observation of students’ attention in class [39]. This observation system captures 3 aspects of engagement: (1) direction of gaze (looks toward the teaching center vs looks elsewhere), (2) posture and body expression (oriented toward the teaching center and alert vs averted or flaccid), and (3) activity (performs the activity necessary for the task vs does something else on the side). Since we analyzed groups of nursing home residents and were not interested in individual differences, we assessed engagement of the group as a whole. To do so, we created a 5-point rating scale reflecting the degree of engagement in the group. For example, in the original systematic behavior observation instrument [39], sequences were rated whether a child looks toward the teaching center. We have modified the formulation from “none” of the participants looks (score 1) to “all” participants look to the center of the robot-assisted activity (score 5), and this 5-level rating scale aimed to assess engagement from very low to very high (see Table 1). Therefore, we distributed the number of people who showed the behavior depending on group size on the 5-level scale.

To capture participants’ mood during the robot-assisted activities, we developed an observational rating scale based on the German version of the Positive and Negative Affect Schedule (PANAS) [40]. The PANAS is frequently used in studies in which human mood states are of interest. The questionnaire consists of 20 adjectives describing different feelings with 10 adjectives capturing positive affect and the other 10 capturing negative affect. The items of the original PANAS are shown in Textbox 1 [25]. This survey instrument was chosen because it contains a set of mood variables that describe mood with positive and negative affects with several adjectives that we assumed were observable by a rater. Based on findings by Reisenzein and colleagues [41] that emotions can be detected by observers using a variety of cues (eg, facial expressions, verbal expressions, physical expressions), we transformed the survey instrument PANAS into an observational rating scale for mood at the group level.

The original 5-level response scale contains the gradations “very slightly or not at all,” “a little,” “moderately,” “quite a bit,” and “extremely.” Again, because we were interested in the mood at the group level, we adapted the rating scale to reflect observable indicators of mood in the group, and a 5-point rating scale from “very low” to “very high” was used. For example, “very low” signified none of the participants were attentive in the robot-assisted group activity, and “very high” signified all participants were attentive in the robot-assisted group activity (see Table 2). To make an objective assessment of group mood during the robot-assisted group activity, sequences from the observation were rated in relation to each adjective from the PANAS. The description of the 5-point rating scale of mood according to the PANAS is shown in Table 2. We considered the observation at group level to be particularly relevant for capturing mood in the group so that situational factors that are an important component in the observation of mental states [41] could be included.

Video clips were rated independently by 2 trained observers (rater 1 and rater 2). Rater 1 was present during all robot-assisted group activities and rater 2 during two randomly selected sessions. Both raters were trained in the observation of nonverbal communication and body language for assessing the items for all 3 aspects of engagement (ie, direction of gaze, posture and body expression, activity) and for mood (ie, positive and negative affect). Clearly visible signs of dementia and severe physical limitations of the residents had to be considered, and the rating of engagement had to be adjusted to the residents’ possibilities of participation (eg, physical limitations). However, no individual was excluded from the analysis, as all ratings were performed at the group level. Each observer rated the group as a whole in every video clip by assessing whether none of the participants, some of the participants, most of the participants, or all of the participants exhibited a particular behavior indicating engagement (eg, direction of gaze, posture and body expression, activity) or positive or negative affect (eg, attentive, scared). To provide specific and context-sensitive anchors for the ratings, we counted the number of participants and distributed them proportionally across the 5-point scale. Thus, it depended on the actual group size what most and some participants meant. During the initial trial, it became apparent that all variables of engagement could easily be observed and rated by both raters. However, for rating the perceived mood in the group of study participants according to PANAS, additional coding rules had to be defined. The items “strong” and “guilty” were difficult to observe and hard to differentiate from 2 other items (eg, proud, ashamed) and thus not considered in the analysis. Each video clip was rated independently by the 2 raters to allow for reliability assessment.

Of the 34 video clips recorded during 5 robot-assisted group activities, 3 videos had to be excluded because not all participants were visible or the video was too short to be rated. Thus, we finally included 31 video clips. In the 4 participating nursing homes, 85 older residents participated in 5 robot-assisted group activity sessions. Participant characteristics are provided in Table 3.

Agreement between the 2 raters was high for engagement and positive and negative affect. Engagement had an ICC score of 0.83 (95% CI 0.65-0.92). Negative affect reached an ICC of 0.84 (95% CI 0.67-0.93), and positive affect had an ICC of 0.90 (95% CI 0.79-0.96). Individual items, specifically adjectives that belonged to negative affect, received a rather weak ICC. These include the items “ashamed” (ICC 0.37, 95% CI –0.32 to 0.70) and “afraid” (ICC 0.39, 95% CI –1.07 to 0.52).

As Table 2 demonstrates, the results show that the engagement of the participants in the robot-assisted group activity was high (mean 4.19, SD 0.47; median 4.0). The direction of gaze was measured as almost very high (mean 4.65, SD 0.49; median 5.0); posture and body expression (mean 4.03, SD 0.71; median 4.0) and activity (mean 3.90, SD 0.65; median 4.0) were also rated as high.

Overall, no negative affect could be observed (mean 1.13, SD 0.20; median 1.0). The mean value of positive affect was 3.22 (SD 0.55; median 3.2), which indicates the observer perceived a good mood during the sessions. Adjectives of the positive affect such as interested (mean 4.13, SD 0.56; median 4.0), alert (mean 4.39, SD 0.67; median 4.0), inspired (mean 3.87, SD 0.96; median 4.0), attentive (mean 4.19, SD 1.05; median 4.0), and active (mean 4.16, SD 0.64; median 4.0) received high ratings around the value 4, while enthusiastic (mean 2.42, SD 1.03; median 2.0), proud (mean 1.23, SD 0.43; median 1.0), and determined (mean 1.94, SD 0.77; median 2.0) were observed to be very low or low within the group of participants. Table 4 shows the detailed results.

Although we did not collect this information systematically, we observed that more residents participated in the robot-assisted activity sessions than expected by the nursing home staff and the research team. The different types of robot-assisted exercises (ie, singing, storytelling, gymnastics) promoted a variety of cognitive and physical stimulations as would a human instructor. Further, when watching the video recordings, we noted that the nursing home staff took time to assist and support participants individually during the robot-assisted activity session. Conversations took place between the residents and the nursing staff, and it seemed that the robot conducting all the instructions allowed more time for personal care.