We built a VR platform to assess the effect of different types of feedback modalities and their combination on users’ behavior and navigation performance. VR constitutes an ideal framework to test different conditions in highly realistic and dynamic scenarios [49-51]. We designed a multisensory, AT-integrated VR system comprising audio feedback implemented in VR and a haptic feedback device interfaced with the virtual environment. We conceived an obstacle avoidance task to assess the ability of the 2 feedback modalities (individually or together) to enhance the mobility of persons with VI.

The application was built and run on a Lenovo Legion 5 15IMH05H gaming laptop. To optimize the gaming stream and ensure the immersiveness of the application, we used a TP-Link Archer GX90 AX6600 Tri-Band Wi-Fi 6 Gaming Router. The Unity game engine (version 2019.4.9f1) was used to develop a VR application for the Meta Quest 2 headset and Touch controllers. Users navigated the virtual environment by physically walking in a first-person perspective. In VR, we designed 2 floors of a subway station whose size matched the dimensions of the physical environment where the experiment took place. The 2 environments included common obstacles and hazards that may be encountered while walking in a subway station, such as broken elevators, construction sites, working tools, garbage, scaffoldings, signage furnishing, and turnstiles (Figure 1). A food vendor, a street musician, and other pedestrians were included to increase engagement and dynamism of the overall environment (Figure 1). We also simulated an elevator ride from the first floor to the second floor of the virtual subway station (Multimedia Appendix 1).

To create a realistic VR experience, sound effects related to a subway station environment, including those of animated pedestrians, were added. As shown by prior studies, integrating sounds related to the visual content enhances the sense of presence of participants in a virtual environment [52,53]. To integrate realistic audio effects in the VR application, we used FMOD, an end-to-end solution for sound that integrates seamlessly with Unity. It simplifies the process of creating sound behaviors, with a comprehensive set of features that allows one to quickly and easily build adaptive audio.

In VR, we simulated different aspects of VI, including peripheral vision loss, reduced contrast sensitivity, altered color perception, and glare [54], as shown in Figure 2 (refer to Multimedia Appendix 2 for more details). Impairment severity was based on the extent of peripheral vision loss and the intensity of the simulated symptoms. Our simulation of peripheral vision loss specifically targeted the severe stage of glaucoma, a prevalent cause of VI among adults in the United States, which is known for its substantial impact on mobility [55]. This progressive reduction of the peripheral visual field in glaucoma impedes the clear identification of objects, which is crucial for obtaining wide-field information about the environment [56,57]. Realistic simulation of such symptoms was accomplished by combining postprocessing effects and C# scripts coded in Unity. Specifically, we combined rendering and graphic tools provided by Unity, such as shader and culling mask. A shader is a mini-program that provides a flexible way of dynamically tweaking the appearance of any components in the scene (such as models and lights). A culling mask is a camera’s property that allows one to selectively render objects in the scene. We used a Gaussian blur shader to reproduce the symptoms of glare and blurriness and a culling mask to create the visual effects of peripheral vision loss, reduced contrast sensitivity, and altered color perception.

To ensure the realism and accuracy of our simulations, we sought the expertise of 2 professionals familiar with low-vision conditions. Specifically, a certified orientation and mobility specialist (also a certified low-vision therapist) with ≥30 years of experience in the field and the chief research officer at an American nonprofit organization dedicated to vision rehabilitation and advocacy for the blind, who is also a research professor of ophthalmology at New York University Grossman School of Medicine, provided their expertise.

Obstacle detection was implemented using the UnityEngine.PhysicsModule. Specifically, the Spherecast function was used to project a sphere of a given radius into the scene. The function returns a true Boolean value when an object in the virtual environment is hit by the sphere, and it provides information about the distance between the projection point and the object.

The haptic feedback was provided by a wearable device in the form of a belt that improves on our team’s previous effort [37-39]. The belt was equipped with 10 cylindrical eccentric rotating mass actuators (Precision Microdrives Ltd, model number 307-103) with a diameter of 9 mm and a length of 25 mm. We opted for this type of actuator as it is widely available, simple to use, and inexpensive. The actuators were arranged on 6 distinct modular units that could be added or removed easily based on users’ preference, ability, and experience with the device (Figure 3). The units were designed in SolidWorks (version 2019) and 3D printed on a Bambu Lab X1C. Precisely, the 4 central modules had 2 actuators each disposed horizontally and separated by a vertical distance of 85 mm. In these central modules, each actuator was enclosed in a parallelepipedal housing of dimensions 35 mm × 42 mm × 10 mm. The housing was made of polylactic acid. To minimize the vibrations inside the modules, each actuator was connected through springs to a flexible element of thermoplastic polyurethane. The 2 modules at the ends of the belt each had a single actuator positioned vertically in the center. In these lateral modules, each actuator was enclosed in a parallelepipedal housing of dimensions 45 mm × 60 mm × 12 mm.

Once assembled, the modules were evenly inserted on the 2 straps of a commercial waist bag, which was secured above the user’s hips through a buckle. Inside the waist bag, we placed all the electronic components needed to control and power the belt, namely, a custom printed circuit board, an EPS32 microcontroller (Espressif Systems), and a Krisdonia 50,000 mAh power bank. The function of the actuators on the belt was to provide environmental information through vibration feedback on the users’ abdomen. Specifically, the vibration indicated the presence and location of obstacles near the user in the virtual environment. The amplitude and frequency of the vibration were programmed to vary on 3 levels based on the distance from the obstacles; information about the position and location of closer obstacles was conveyed through higher amplitude and frequency. The belt was connected to the laptop via Wi-Fi using the EPS32 microcontroller. The interface between the belt and the VR environment was enabled through a server or client transmission control protocol established in a C# script.

The user’s field of view in VR, characterized by a horizontal span of 89° and a vertical span of 93° (per the Meta Oculus Quest 2 specifications), was discretized into a grid comprising 10 sectors. This grid layout closely mirrored the configuration of actuators on the haptic feedback belt. Each sector was then associated with a virtual sphere projected from the user’s body. The 10 resulting spheres were positioned to align with the 10 field of view sectors. Anytime an obstacle fell into a sector, it was detected by a specific sphere, and information to activate the actuators was sent through the transmission control protocol to the EPS32 microcontroller. The latter used pulse width modulation to control a metal-oxide-semiconductor field-effect transistor driver (Texas Instruments) placed in the printed circuit board, which fed the actuators. The maximum hit distance of the spheres was set to 2.5 m based on pilot testing of the haptic feedback system. This value determined the range of action of the belt. The frequency of vibration was regulated on 3 levels based on the distance of the object from the user in VR by means of a C++ code.

The audio feedback was provided through the VR headset, and it consisted of a beep sound added to the VR application using an FMOD sound effect engine. Similar to haptic feedback, audio feedback serves the purpose of alerting users of the presence of obstacles in their surroundings via a beep sound. The sound was played at increasingly short intervals as the user approached an obstacle. The VR device was connected to the laptop via Wi-Fi using the Oculus application and Quest Link.

Obstacle detection through audio feedback was again implemented in a C# script using the Spherecast function. However, in this case, only 1 sphere was designed to be projected from the user’s head in the virtual environment. Anytime an object was in the direction the user was facing, it was detected by the sphere and a beep sound was emitted by the VR headset to alert the user about the presence of an obstacle. Similar to the haptic feedback, the maximum hit distance of the sphere was set to 2.5 m. The rationale behind this audio feedback design was to enhance users’ residual vision while exploring the environment with their head movement via simple and intuitive audio feedback.

Moving forward, future implementations could explore additional sensory cues to further enrich the user experience in virtual environments. For example, synchronized footstep sounds tailored to users’ movements have been shown to significantly elevate perceived presence in the virtual environment. This heightened presence fosters greater awareness of one’s gait and posture, resulting in more authentic interactions and enhanced movement control [58]. The efficacy of echo-acoustic cues in navigating virtual environments has also been previously assessed [59]; not only could these cues improve collision avoidance and navigation efficiency, but they may also enhance the perception and evaluation of different routes after training.

A total of 72 healthy participants with a mean age of 25.93 (SD 4.48) years were recruited from New York University Tandon School of Engineering. Of these 72 participants, 26 (36%) self-identified as women and 46 (64%) as men. To reduce the risk of injury or discomfort associated with the use of a VR device, we excluded people who were pregnant; older adults; had preexisting binocular vision abnormalities or psychiatric disorders; had a heart condition, seizures, or other serious medical conditions; and used medical devices. We opted for self-reported visual acuity to exclude persons with preexisting binocular vision abnormalities, as conducting objective screenings for all participants would have required additional resources, including time and personnel. Given the nature of our research and the characteristics of our target population, we felt self-reporting was a practical and feasible approach, allowing us to efficiently gather relevant data without significantly extending the duration of participant recruitment and data collection. Participants with normal or normal corrected vision were included in the study.

The experimental study took place in a multipurpose production space at New York University’s Media Commons, consisting of 4 bays, each of which was 6 m long and 2 m wide with a total area of 178 m². Other than 4 curtains positioned along the sidewall of the bays, the environment was free from obstructions. Thus, participants were able to walk freely during the experiment (Figure 4).

There was no training provided for using the haptic feedback device or the VR platform; participants completed the experiment in a single session.

Participants performed an obstacle avoidance task on the 2 floors of the virtual subway station environment while experiencing the most common symptoms and signs of a VI. Specifically, participants were asked to physically walk from a starting point until they reached a virtual elevator and then turn 180° and walk back until they reach the train platform. To help participants understand that they had reached the final destination, arrival was signaled through the sound of a turnstile opening and the animation of a train passing by. Immediately after the completion of each condition, participants were asked to fill out a questionnaire concerning their overall experience and the 2 types of feedback (refer to the Questionnaire subsection).

During the experiment, the belt and the VR headset alerted users about the presence of obstacles in the surrounding environment through vibration feedback on the abdomen and audio feedback, respectively, to minimize the possibility of a collision. The right Oculus Touch controller vibrated any time a user hit an obstacle in the virtual environment to reproduce the sensation of touching an object. The left Oculus Touch controller was attached to the haptic feedback device vertically to track the position of the users during the experiment (refer to the Data Collection subsection). The experiment was aimed at realistically recreating a path from the entrance of a subway station to the train platform, with a maximum duration of 30 minutes to prevent distress associated with extended VR sessions [60].

A total of 4 experimental conditions were tested to elucidate the individual and combined effects of haptic and audio feedback on movement behavior, navigation performance, and self-reported ratings. Each participant performed the task in 4 different conditions: no feedback, haptic feedback only, audio feedback only, and both feedbacks. Apart from the type of feedback provided, all conditions were identically structured. Each participant was assigned to only 1 (4%) of the 24 possible combinations for the following purposes: (1) preventing fatigue from potentially diminishing the impact of the feedback on users’ performance in the later stages of the experiment and (2) mitigating biases related to increased familiarity with the devices. During the obstacle avoidance task, data on the navigation performance (task completion time, number of collisions, and trajectory) and movement behavior (head and body orientation) of the participants were collected (refer to Data Collection subsection).

This study aimed to answer the following research questions (RQs) based on the collected data:

During each experiment, we collected the following metrics: number of collisions, completion time, head orientation, and body position and orientation. To save these metrics, we used 2 C# scripts. The first script was used to start and reset a stopwatch at the beginning of each experiment and to collect the following data: (1) head orientation (Euler angles) from the VR headset, (2) body position from the user’s body in VR, and (3) body orientation from the left Oculus Touch controller. Specifically, to collect data on users’ body position, we provided the player with a CapsuleCollider and a RigidBody component. The former is an invisible capsule-shaped primitive that represents the user’s body in VR, while the latter provides the user’s body with physics properties. These 2 components moved in the virtual environment according to the movement of the user in the real environment. The left Oculus Touch controller was secured vertically on the belt by means of an element 3D-printed in carbon fiber reinforced polylactic acid and used for collecting users’ body orientation. The game object representing the left Oculus Touch controller in VR moved in the virtual environment according to the movement of the physical controller in the real environment.

The second script was used to simulate the collision with obstacles and to alert the user through a vibration provided by the right Oculus Touch controller. To enable the vibration of the controller, each virtual object was provided with a RigidBody and a Collider component. In this case, we used a BoxCollider, an invisible box-shaped primitive that encloses the object. When a BoxCollider of an object came in contact with the collider of the player, the script initiated the vibration of the right Oculus Touch controller and registered a collision.

A questionnaire was created to collect participants’ opinions on the overall experience and the 2 types of feedback. The questionnaire (Multimedia Appendix 3) included 8 items. Questions 1 to 3 were designed to investigate participants’ familiarity with VR, emotional reaction, and potential motion sickness felt during the experiment. Question 4 sought to understand participants’ personal perception of their navigation performance during the 4 experimental conditions. Question 5 asked for an explanation about their answer to question 4. Question 6 was designed to explore participants’ preference toward 1 specific condition. Question 7 required an explanation about that preference. Finally, the participants were asked to give an overall evaluation of the experience using a 5-point scale (not at all interesting, slightly interesting, moderately interesting, fairly interesting, and extremely interesting). The questionnaire was developed in a Google form, and it was accessible to participants by scanning a QR code. Participants filled out the questionnaire only after they completed all the 4 experimental conditions.

The data processing was performed in MATLAB (MathWorks, version 2021b). The body position was defined in a coordinate system CS0 whose origin was set at the experiment starting position, as shown in Figure 5A. The x- and y-axes were oriented along the main dimensions of the room, while the z-axis was aligned with the direction of gravity. Euler angles (ψ_b, θ_b, ϕ_b) were used to describe the orientation of the trunk, and Euler angles (ψ_h, θ_h, ϕ_h) were used to describe the spatial orientation of the head; coordinate systems are shown in Figure 5B. Raw data of the Euler angles and body position were smoothed using a quadratic regression method over a window of 20 samples to minimize noise from the measured data.

We computed participants’ trajectory length and smoothness. The trajectory length of each participant was calculated as follows:

where nF is the number of frames, p_t = [X_0,t,Y_0,t] is the body position in 2 dimensions at time step t, and ||・|| is the Euclidean norm.

Smoothness was estimated through the spectral arc length (SPARC) [61] and computed as follows. First, we performed a numeric derivative on the speed profile v. Then, we computed the fast Fourier transform on the speed to obtain the spectrum magnitude V(f) as a function of the frequency f, which we normalized with respect to its maximum to obtain.

where f_i is the i-th frequency component of the spectrum.

We determined the cut-off frequency f_c as the maximum frequency where the spectral magnitude is above a threshold V and below a maximum frequency limit f_max,

f_c = {f_i < f_max, V_norm(f_i) > V}

Finally, we computed the SPARC,

where N_fc is the number of frequency components up to f_c and ΔV(f_i) is the difference in the normalized spectrum magnitude between adjacent frequency components, calculated as ΔV(f_i) = V_norm(f_i+1) − V_norm(f_i). We set V = 0.05 and f_max = 10 Hz. The SPARC is related to the frequency content of the velocity, and therefore, a smoother movement presents a higher value of SPARC.

To evaluate how each condition affected the user’s head motion, we performed an analysis of the variability of the pitch angle of the head θ_h and the difference between the head yaw angle ψ_h and body yaw angle ψ_t, defined as χ.

The angle variability was calculated by computing Shannon entropy, defined as

where p(∙) denotes probability and λ is a realization of Λ in the sample space of all the possible realizations Ω. The entropy H(Λ) is expressed in bits because a logarithm with base 2 was used. To compute the entropy for the aforementioned angles, we split the range of motion into single-degree intervals and computed the probability for each bin.

Experimental results on completion time and number of collisions are reported in Figure 6A and Figure 6B, respectively. The haptic feedback through the belt was conducive to an increase in the completion time of the task (F_1,207.5=4.7962; P=.03) and a decrease in the number of collisions (test statistic from the Scheirer-Ray-Hare test, H=3.8285; P=.05). The audio feedback, instead, was not found to modulate the completion time and number of collisions; neither did we find a main effect of the audio feedback (completion time: F_1,207.5=0.1467; P=.70 and collisions: H=0.6110; P=.43), nor did we observe a significant interaction between the audio and haptic feedback (completion time: F_1,207.5=1.7725; P=.18 and number of collisions: H=0.8518; P=.35).

Experimental results on L and SPARC are reported in Figure 6C and Figure 6D, respectively. The haptic feedback through the belt was linked to a notable increase in trajectory length (F_1,188.12=7.3482; P=.007), though it did not yield a significant variation of the trajectory smoothness (F_1,213=0.0127; P=.91). In contrast, audio feedback yielded a significant enhancement in the trajectory smoothness (F_1,213=7.6342; P=.006), but it did not influence the trajectory length (F_1,188.09=0.2972; P=.58). A significant interaction between haptic and audio feedback was observed with respect to the trajectory length (F_1,186.73=4.20092; P=.04) but not with respect to the trajectory smoothness (F_1,213=1.2684; P=.26).

Our investigation of the efficacy of the haptic feedback device indicated notable improvements in participants’ navigation performance, specifically in reducing the number of collisions. However, these positive effects did not extend to task completion time, trajectory length, or trajectory smoothness. Contrary to our expectations, the introduction of the haptic feedback device led to a significant increase in task completion time. Participants exhibited hesitancy in their walking behavior when relying only on the haptic feedback device, as evidenced by observable delays in reacting to stimuli. Such an outcome diverged from our earlier work [42], where the haptic feedback device was found to reduce task completion time. This disparity can be attributed to the increased difficulty and duration of the obstacle avoidance task in this study as well as the distinct walking modality used. In this experiment, participants navigated a dynamic and complex urban environment, whereas in our previous study, they traversed a simpler and smaller outdoor environment using a controller. The prolonged task completion time resulted in longer trajectories in response to haptic feedback.

The spatial resolution of the haptic feedback device played a crucial role in these findings, as the detailed environmental information prompted participants to navigate cautiously, resulting in intricate trajectories. Examining participants’ movement behavior, we observed a significant reduction in the entropy of the pitch angle of the head due to haptic feedback. Just as participants moved more smoothly in the environment, they also maintained a more constant and less variable head orientation. However, the device did not affect the entropy of the difference between the yaw angle of the head and the yaw angle of the body. This result may be attributed to the spatial information provided by the haptic feedback device, which guided users based on their body orientation and prompted them to reduce their vertical head movements.

We registered an effect of audio feedback on participants’ navigation performance with respect to trajectory smoothness. Using audio feedback, participants were likely to favor straight paths, as seen from reduced instances of halted movement and a reduced tendency to course-correct during navigation. Moreover, the design of the audio feedback system, which alerted users to obstacles in their line of sight, may have facilitated the exploration of the environment with just the movement of their head. We did not register a variation in the number of collisions, likely due to the modality used by the audio feedback device for obstacle detection. In contrast to the haptic feedback device, which detected obstacles using 10 vibrating actuators on the user’s abdomen, the audio feedback device signaled the presence of obstacles in the user’s line of sight through a distinctive beep sound emitted by the VR headset. The lower spatial resolution of the audio feedback device may have been less effective in aiding users to avoid obstacles compared to the haptic feedback.

While not effective in reducing the trajectory length alone, audio feedback had a positive effect on haptic feedback in the form of a significant interaction between the 2 modalities. In fact, the increase in trajectory length due to haptic feedback alone is mitigated by the concurrent use of audio feedback, thereby suggesting that participants were able to leverage both information cues and make informed decisions as they negotiated haptic versus audio cues. However, a positive role of combined feedback was not observed for all metrics, likely due to the increased cognitive load resulting from the use of both feedbacks. It is tenable that the delivery of multiple feedback cues poses some difficulties in terms of assimilation and requires a learning curve for users to adapt to new approaches. In principle, this may be mitigated by increasing the training time for users to become more proficient with combined feedback. Overall, the combined use of audio and haptic feedback enhances safety by facilitating informed decision-making, and it contributes to travel efficiency by addressing trajectory length and smoothness. This underscores the potential of blending feedback modalities to optimize both safety and travel efficiency.

The results of the questionnaire offer valuable insights into participants’ experiences and preferences during the experiment. The high engagement reported by 76% (55/72) of the participants suggests that the multisensory feedback, comprising both haptic and audio cues, contributed to an immersive and captivating experience. Notably, only a minimal percentage of participants (4/72, 6%) experienced nausea or motion sickness, indicating that the implemented feedback modalities were well tolerated. Participants’ perceptions of navigation performance revealed a preference for the combined haptic and audio feedback condition, with 50% (36/72) of them believing that it enhanced their performance. Participants emphasized that haptic and audio cues offer distinct information. Many participants note that having both types of feedback provides a more complete and nuanced understanding of their surroundings, aiding in better decision-making and spatial awareness. Finally, others found the combination more intuitive, with haptic feedback offering directional cues and audio feedback providing information on the proximity of obstacles. The preference for both modalities suggests that, when used together, they complement each other, addressing potential limitations or confusion that might arise when using either haptic or audio feedback alone. Interestingly, the preferred condition did not always align with perceived performance, highlighting the complexity of user preferences. Finally, most of the participants (42/72, 58%) found the overall experiment extremely interesting, emphasizing the potential of multisensory, AT-integrated VR systems in maintaining user engagement. These findings underscore the importance of considering user preferences and experiences when developing and refining multisensory ATs, ensuring that future iterations are tailored to meet the needs of individuals with visual impairments.