Abstract

In daily life, the human hand exhibits remarkable abilities, such as fine in-hand manipulation and multimodal sensing, which are crucial for complex tasks like multiobject manipulation. However, current robotic dexterous hands have not yet achieved this level of proficiency due to limitations in hardware, perception algorithms, control strategies, and data collection. In this work, we present a dexterous in-hand teleoperation framework, DIH-Tele, designed to enable such complex tasks. The framework includes the tactile dexterous hand (T-DexCo hand), an accurate dexterous teleoperation system, multimodal data collection, and an imitation learning algorithm based on discrete control space and fused training. The in-hand counting task is selected as a common example of multiobject manipulation, which involves counting a set of objects held in hand and selectively removing a specified number of them by in-hand manipulation. Our experimental results demonstrate that the DIH-Tele framework effectively leverages multimodal perception to perform multiobject manipulation tasks with a success rate approaching that of human teleoperation. Additionally, the learned fingertip behaviors are highly versatile, often utilizing every degree of freedom of the dexterous hand. Finally, ablation studies confirm the significant impact of multimodal perception and fused training on enhancing multiobject manipulation tasks.

Publications

J. Huang, K. Chen, J. Zhou, X. Lin, P. Abbeel, Q. Dou, & Y. Liu. “DIH-Tele: Dexterous In-Hand Teleoperation Framework for Learning Multi-Objects Manipulation With Tactile Sensing”. IEEE/ASME Transaction on Mechatronics, 2025. Paper