Transformer Driven Visual Servoing for Fabric Texture Matching Using Dual-Arm Manipulator
Published:
F. Tokuda1, 2, A. Seino1, 2, A. Kobayashi1, 2, T. Kai1, 2, K. Kosuge1, 2, 3
1. Centre for Transformative Garment Production, Unites 1215 to 1220, 12/F, Building 19W, SPX1, Hong Kong Schience Park, Pak Shek Kok, N. T., Hong Kong SAR
2. Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR
3. Director of the JC STEM Lab of Robotics forSoft Materials, Department of Electrical and Electronic Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR
Published in IEEE Robotics and Automation Letters (RA-L), 2 (2025) 1522 - 1529
DOI: 110.1109/LRA.2025.3643335
Download here.
- Abstract
- In this paper, we propose a method to align and place a fabric piece on top of another using a dual-arm manipulator and a grayscale camera, so that their surface textures are accurately matched. We propose a novel control scheme that combines Transformer-driven visual servoing with dual-arm impedance control. This approach enables the system to simultaneously control the pose of the fabric piece and place it onto the underlying one while applying tension to keep the fabric piece flat. Our transformer-based network incorporates pre-trained backbones and a newly introduced Difference Extraction Attention Module (DEAM), which significantly enhances pose difference prediction accuracy. Trained entirely on synthetic images generated using rendering software, the network enables zero-shot deployment in real-world scenarios without requiring prior training on specific fabric textures. Real-world experiments demonstrate that the proposed system accurately aligns fabric pieces with different textures.