Full Text
Transmitting Mastery: A Spatial Computing-Based Skill Transfer Framework Using XR and AI for Remote Manufacturing
This study investigates whether expert skills can be effectively transmitted and archived through spatial computing technologies—specifically XR and AI—and whether such processes can be evaluated quantitatively. We present SHUGI, a skill transfer framework developed by USEYA ADVANCED INDUSTRY (UAI), a Japanese design-tech firm with over 13 years of XR expertise. SHUGI integrates smart glasses (e.g., Apple Vision Pro, Meta Quest 3), AWS cloud services, WebSocket, and WebRTC to facilitate remote manufacturing support and digital skill archiving.
To evaluate the framework’s efficacy, we conducted a controlled experiment in UAI’s digital workshop using laser cutters and 3D printers. Nine participants unfamiliar with digital fabrication tools were divided into three groups (A–C).
Group A received direct, in-person instruction from experts.
Group B was trained through remote XR guidance and live expert support.
Group C relied entirely on pre-recorded SHUGI XR guidance and AI-generated feedback without any real-time human assistance.
Results show that for laser cutting, Group C achieved task completion in ~1 hour compared to ~3 hours in Groups A and B. For 3D printing, Group C required ~9 hours, outperforming Group A (~10h) and Group B (~12h). Although memory retention after initial training showed similar decay across all groups, Group C required no retraining due to the availability of on-demand digital guides. Notably, Group C’s training time improved 10–30% in initial tasks, and AI-powered skill synchronization scoring enabled users to reach over 90% alignment with expert motion data after three self-guided sessions—up from an initial 20%.
In contrast, Groups A and B depended on subjective human judgment to assess proficiency. Group B did benefit from digitized training records, leading to 30–50% reductions in retraining effort, yet lacked consistent, objective evaluation tools. SHUGI’s ability to visualize and quantify tacit knowledge through real-time feedback contributed to greater learner motivation, eliminated the need for instructor presence, and enabled repeatable, location-independent training.
Technologically, the SHUGI system incorporates:
Input devices and data acquisition:
Smart glasses (Apple Vision Pro, Meta Quest 3), MANUS Prime 3 motion gloves, and RGB cameras installed at multiple points in the UAI stage. Expert performance is recorded in BVH/FBX format via Unity or Unreal Engine, along with audio instruction.
AI processing:
AWS SageMaker is used for training and deploying models to analyze user performance, with MediaPipe applied for full-body motion tracking based on RGB footage.
Output and user feedback:
Vision Pro displays real-time overlays of motion discrepancies and finger movement paths using Unity (URP + ShaderGraph) or RealityKit. Planned development includes haptic feedback on a per-finger basis to further enhance embodied learning.
Future development includes expanding participant diversity beyond UAI facilities, applying SHUGI across different industries, implementing full-body motion tracking to capture complex movements, and creating an inclusive, multilingual digital skills archive to support global accessibility. UAI aims to finalize a universal, scalable skill transfer UI/UX system by 2027 to foster sustainability across traditional and emerging industries.
This study investigates whether expert skills can be effectively transmitted and archived through spatial computing technologies—specifically XR and AI—and whether such processes can be evaluated quantitatively. We present SHUGI, a skill transfer framework developed by USEYA ADVANCED INDUSTRY (UAI), a Japanese design-tech firm with over 13 years of XR expertise. SHUGI integrates smart glasses (e.g., Apple Vision Pro, Meta Quest 3), AWS cloud services, WebSocket, and WebRTC to facilitate remote manufacturing support and digital skill archiving.
To evaluate the framework’s efficacy, we conducted a controlled experiment in UAI’s digital workshop using laser cutters and 3D printers. Nine participants unfamiliar with digital fabrication tools were divided into three groups (A–C).
Group A received direct, in-person instruction from experts.
Group B was trained through remote XR guidance and live expert support.
Group C relied entirely on pre-recorded SHUGI XR guidance and AI-generated feedback without any real-time human assistance.
Results show that for laser cutting, Group C achieved task completion in ~1 hour compared to ~3 hours in Groups A and B. For 3D printing, Group C required ~9 hours, outperforming Group A (~10h) and Group B (~12h). Although memory retention after initial training showed similar decay across all groups, Group C required no retraining due to the availability of on-demand digital guides. Notably, Group C’s training time improved 10–30% in initial tasks, and AI-powered skill synchronization scoring enabled users to reach over 90% alignment with expert motion data after three self-guided sessions—up from an initial 20%.
In contrast, Groups A and B depended on subjective human judgment to assess proficiency. Group B did benefit from digitized training records, leading to 30–50% reductions in retraining effort, yet lacked consistent, objective evaluation tools. SHUGI’s ability to visualize and quantify tacit knowledge through real-time feedback contributed to greater learner motivation, eliminated the need for instructor presence, and enabled repeatable, location-independent training.
Technologically, the SHUGI system incorporates:
Input devices and data acquisition:
Smart glasses (Apple Vision Pro, Meta Quest 3), MANUS Prime 3 motion gloves, and RGB cameras installed at multiple points in the UAI stage. Expert performance is recorded in BVH/FBX format via Unity or Unreal Engine, along with audio instruction.
AI processing:
AWS SageMaker is used for training and deploying models to analyze user performance, with MediaPipe applied for full-body motion tracking based on RGB footage.
Output and user feedback:
Vision Pro displays real-time overlays of motion discrepancies and finger movement paths using Unity (URP + ShaderGraph) or RealityKit. Planned development includes haptic feedback on a per-finger basis to further enhance embodied learning.
Future development includes expanding participant diversity beyond UAI facilities, applying SHUGI across different industries, implementing full-body motion tracking to capture complex movements, and creating an inclusive, multilingual digital skills archive to support global accessibility. UAI aims to finalize a universal, scalable skill transfer UI/UX system by 2027 to foster sustainability across traditional and emerging industries.