A Labor Of (G)Love: A Mini-Review of Haptic Glove- Controlled Robotic Hands in the ANA Avatar XPRIZE for Fine Telemanipulation

Telepresence technology allows people to physically interact with distant environments using a robotic surrogate body-commonly known as a robot avatar. A human operator furnished with various Virtual Reality (VR) equipment controls the faraway telerobot while receiving visual and tactile feedback. Beyond connecting people to remote situations tangibly, telepresence shows incredible promise in driving robots to accomplish useful work. This is especially true for physically demanding gross manipulation tasks, like material handling, construction, or pushing carts [1]. In contrast, the robots’ fine manipulation Capabilities-encompassing tasks primarily using the fingers instead of the whole-arm-remain limited. Moreover, fine manipulation is often prerequisite for completing many gross manipulation tasks. For instance, drilling holes in a wall (gross manipulation) often demands several finger picking and pressing actions (fine manipulation) to insert batteries, change drill bits, and control the trigger. Thus, tasking telepresence robots for generalpurpose use necessitates a solution for fine robotic manipulation.

State-of-the-Art Telepresence Showdown: The ANA Avatar XPRIZE

A driving factor behind struggles with fine manipulation resides in manipulator control strategies. For instance, many telepresence systems supply human operators with data gloves to control anthropomorphic robots with multifigured dexterous hands. The 2022 ANA Avatar XPRIZE, a global telepresence robotics competition with a $10M prize pool, was a prime demonstrator of this strategy. Most notably was the competition’s televised finals event, which subjected each telepresence system to a space-themed mission requiring locomotion and advanced manipulation tasks to be performed live by novice operators without team intervention. Of the 17 teams that qualified for the live finals event, 11 used the data glove and robotic hand combination, including the winner of the $5M grand prize [2].

Despite the ANA Avatar XPRIZE Finals’ ability to both catalyze and showcase avatar robot development, some of the competition requirements suited some systems far better than others. For instance, on flat tile indoors, bipedal robots simply cannot compete with their wheeled competitors in terms of speed or stability. As such, team rankings are not perfect. comparative descriptors of each robots’ general prowess. Nonetheless, analysis of the robots and technologies developed can shed useful insight on the current state-of-the-art in robotic telepresence, especially in terms of robotic hands, haptic feedback, and fine manipulation.

Hand Control via Haptic Exoskeletons

Several systems during the competition utilized the commercial Sense Glove DK1 data glove-a haptic exoskeleton glove with 20 Degrees of Freedom (DoFs), 4 angular position sensors per finger, vibrotactile motors, and 5 electronic braking modules for force feedback-to command various robotic hands while. providing force feedback. Team JANUS utilized a custom, underactuated, 1 DoF anthropomorphic hand with force sensors in the fingertips that drove haptic actuation on the DK1 gloves [3]. This simplified system allowed their avatar to grab several objects of varying shape and size, however, binary control of the fingers (e.g. all open or all closed) makes any complex fine or in-hand manipulation impossible.

On the other hand, Team AVATRINA’s approach documented in [4] was more sophisticated, using finger measurements from the DK1 in tandem with an extensive calibration procedure to retarget operator finger motions to their Psyonic Ability Hand (a rugged 6 DoF prosthetic hand). Moreover, the Ability Hand’s inherent fingertip force sensors can binarily trigger force feedback on the DK1 when a threshold is surpassed. This overall strategy allowed their avatar robot to successfully use a drill to remove a bolt in the penultimate task of the competition. While successful in many regards, limited in-hand manipulation can be accomplished with this setup due to an inherent lack of finger addiction/abduction, or “splay”. Without moving the finger’s plane of flexion, fine manipulation tasks, especially those requiring rotating an object or readjusting a grasp, are often perilous. In contrast, Team iCub used the DK1 to control their custom 9 DoF hands which included finger splay motions [5], although in-hand manipulation with this system has never been publicly.

documented, to the best of the author’s knowledge. Similarly, Team SNU developed their own 8 DoF robotic hand in [6] and exoskeleton glove (similar to the DK1) but has not publicly shared their control techniques.

Finally, Team NimbRo, the winners of the competition, used the aforementioned glove to retarget motions onto their 9 DoF SCHUNK SVH humanoid hand. Their setup in [7] involved joint-level mapping of exoskeleton movements to their dexterous manipulator, and haptic actuation driven by a mixture of robot’s finger joint currents and distal sensor feedback. This setup displayed prowess over much of the competition, aiding in the team winning 1st place at both the semifinals and finals testing events. While 9 DoF hands permit limited in-hand/fine manipulation, Team NimbRo’s avatar could successfully use a drill, infrared thermometers, and even set up a blood pressure monitor, demonstrating that some fine manipulation tasks can be achieved with lower DoF hands and haptic feedback gloves.

Other Approaches for Haptic Robot Hands

Beyond exoskeleton-based retargeting, the next dominant strategy involved using tendon-driven haptic gloves for tactile feedback. The most used gloves in this regard were the HaptX DK1 and the Sense Glove Nova, with the former being used by Team Cyberselves (did not publicly share methods), i-Botics, and Tangible Research. Contrarily, the Sense Glove Nova was only used by Team UNIST [8], whose hardware and control strategy were similar to that of Team AVATRINA. i-Botics also used an AVATRINAreminiscent strategy, mapping 6-DoF fingertip poses from the operator’s gloves to their own Psyonic Ability Hand, while using their robot’s fingertip sensing to drive haptic actuation signals [9].

Team Avatar-Hubo, of which the Authors were a part of, had a different approach, opting to use separate interfaces for tracking finger motions and rendering haptic feedback. Motion retargeting was accomplished using MANUS Quantum Metagloves, which are magnetically tracked motion capture gloves that offer sub-millimeter precision in fingertip pose estimation, for motion capture, while a digital twin of their custom 20 DoF robotic hand [10] recreated the motions by solving inverse kinematics. Simultaneously, hand joint currents and fingertip piezoresistive sensor feedback were haptically rendered using a multimodal haptic finger thimble (WEART Touch DIVER) [11]. Despite this hand having more DoF per finger than any other hand brought to the competition, in-hand manipulation has yet to be documented with it.

Finally, Tangible Research (formerly Converge Robotics) in [12] used a pair of HaptX DK1 gloves with a (20 DoF) Shadow Robot Dexterous Hand with Syntouch BioTac sensors installed on each fingertip. This particular setup grants operators restrictive force feedback, cutaneous force feedback, and a hand with similar DoF to its human counterpart. Such sophisticated hardware culminates in the competition’s only system demonstrating in-hand manipulation, revealing the ability to twist bottle caps, open boxes, or even plug-in USB cables. The heavy emphasis on sensing comes at a staggering price, however; prices of components are estimated at ~$100,000 USD per dexterous hand (2 total), ~$15,000 USD per fingertip sensor (10 total), and ~$100,000 USD per pair of HaptX gloves. Thus, telepresence robot in-hand manipulation appears to currently be locked behind a staggering technological gap, only having been scaled by those with extremely specialized and expensive equipment.

The capabilities fine manipulation capabilities of telepresence robots are limited by robotic hand dexterity, human correspondence, motion retargeting, sensing, and haptic feedback. While commercialized solutions exist for some of these issues, many come at debilitating costs, creating a notable gap in the field of robotic telepresence. Methods to cut costs by reducing robotic hand DoF, sensor capabilities, and haptic actuation can quickly make intuitive in-hand manipulation cumbersome, if not impossible, as demonstrated by many of the telepresence systems with less sophisticated hardware than Tangible Research. Thus, for robotic in-hand manipulation to become ubiquitous within telepresence, the issue of cost must be addressed. By solving these existing limitations, telepresence systems offering anthropomorphic hands and haptic gloves could realize truly general-purpose applications, enabling people commandeering robots to complete any physical task, as if they were physically present.

The authors gratefully acknowledge the contributions of Truc Tran, Son Tran, Baekseok Kim, and Nathan Kassai from the University of Nevada, Las Vegas, for their help in developing the HDHM robot hand platform and its controller.

No conflict of interest.

1. JC Vaz, N Kosanovic, P Oh (2024) Avatar Robotics Telepresence-the future of humanoid material handling loco-manipulation. Intel Serv Robotics 17(1) 237: 237-250.
2. M Schwarz, C Lenz, R Memmesheimer, B Pätzold, A Rochow, et al. (2023) Robust Immersive Telepresence and Mobile Telemanipulation: NimbRo wins ANA Avatar XPRIZE Finals. 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), USA.
3. R Cisneros-Limo´n, A Dallard, M Benallegue, K Kaneko, H Kaminaga, et al. (2024) A Cybernetic Avatar System to Embody Human Telepresence for Connectivity, Exploration, and Skill Transfer. Int J of Soc Robotics.
4. J M Correia Marques, P Naughton, JC Peng, JS Nam, et al. (2024) Immersive Commodity Telepresence with the AVATRINA Robot Avatar. Int J of Soc Robotics.
5. S Dafarra, U Pattacini, G Romuladi, L Rapetti, R Grieco, et al. (2014) iCub3 avatar system: Enabling remote fully immersive embodiment of humanoid robots. Sci. Robot 9(86).
6. E Sung, S You, S Kim, J Park (2023) SNU-Avatar Robot Hand: Dexterous Robot Hand with Prismatic Four- Bar Linkage for Versatile Daily Applications. 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), USA.
7. C Lenz, M Schwarz, A Rochow, B Pätzold, R Memmesheimer, et al. (2023) NimbRo Wins ANA Avatar XPRIZE Immersive Telepresence Competition: Human-Centric Evaluation and Lessons Learned. Int J of Soc Robotics (2023).
8. S Park, J Kim, H Lee, M Jo, D Gong, et al. (2023) A Whole-Body Integrated AVATAR System: Implementation of Telepresence with Intuitive Control and Immersive Feedback. IEEE Robotics & Automation Magazine.
9. JV Bruggen, C Brekelmans, R Lieftink, D Dresscher, JV Erp (2023) I-Botics avatar system: Towards robotic embodiment. 2nd Workshop Toward Robot Avatars, IEEE International Conference on Robotics and Automation (ICRA), UK.
10. T Tran, JC Vaz, N Kosanovic, P Oh (2023) Highly Dexterous Humanoid Manipulator (HDHM) Designed for Avatar Systems. 2023 IEEE International Conference on Consumer Electronics (ICCE), USA.
11. N Kosanovic, JC Vaz, P Y Oh (2023) Biomimetic Real-Time Multimodal Tactile Perception and Haptics for Telepresence Humanoids. 2023 21st International Conference on Advanced Robotics (ICAR), UAE.
12. J A Fishel, T Oliver, M Eichermueller, G Barbieri, E Fowler, et al. (2020) Tactile Telerobots for Dull, Dirty, Dangerous, and Inaccessible Tasks. 2020 IEEE International Conference on Robotics and Automation (ICRA). France.