Introducing a sense of physicality to the application of virtual and augmented reality, collectively termed as extended reality (XR), still remains one of the main challenges for adding realism to the virtual world. Although significant advancements have been made through the addition of differing sensing modalities to increase auditory and visual realism, closing the loop between the real-world physicality and immersive virtual environments has been seen as the next chapter in development.
Closing the loop requires an interaction between the physical forces generated by the user in the virtual environment and the feedback of haptic sensations to the user. The problem exists that, in current XR solutions, the force input has either been completely neglected or the feedback has relied upon overly obtrusive sensors that limit, or otherwise compromise, the user experience with the virtual environment.
Authors from New York University and the Chinese University of Hong Kong recently published a new paper looking to address some of these issues with the novel application of Trigno Avanti EMG sensors combined with a force sensitive pressure pad to create a learning-based neural interface for natural and intuitive force inputs in the virtual world.
The authors were able to show that, by using the lightweight and non-intrusive Trigno Avanti EMG sensors placed on the subject’s forearm, subjects could inform and establish a robust understanding of their complex hand activities. By employing a neural-network based model, the interface was able to decode the finger wise forces in real-time with an average 3.3% mean error and with the ability for it to be generalized to new users with no, or very little, additional calibration.
The authors, through the interactive study, were able to show that human perception of virtual objects’ physical properties, such as stiffness, can be significantly
enhanced by the interface. It was further demonstrated that the interface enables ubiquitous control via finger tapping. Ultimately, it is envisioned that the author’s findings will push forward research towards more realistic physicality in future XR.
Reference
Yunxiang Zhang, Benjamin Liang, Boyuan Chen, Paul Torrens, S. Farok Atashzar, Dahua Lin, and Qi Sun. 2022. Force-Aware Interface via Electromyography for Natural VR/AR Interaction. ACM Trans. Graph. 41, 6, Article 268 (December 2022), 18 pages. https://doi.org/10.1145/3550454.3555461