Validating immersive technologies for enhanced decision-making data in surgical robotics via agent-based learning

This thesis addresses the critical challenge of acquiring high-quality decision-making data for surgical robotics by uniting immersive technologies with advanced machine-learning frameworks. Existing data-collection methods often fail to capture the rich multimodal signals necessary for training competent robotic agents. To overcome these limitations, a synchronized pipeline was developed within the DaVinci Research Kit (dVRK) simulator that combines stereoscopic virtual-reality (VR) visualization, real-time haptic feedback, electroencephalography (EEG)–based cognitive monitoring, motion tracking, and task performance metrics. A novel Reinforcement-and-Imitation Diffusion Learning (RIDL) approach was introduced, employing a stable-diffusion pre-training stage for a diffusion-policy network followed by Proximal Policy Optimization (PPO) fine-tuning with expert demonstrations. Contributions include (1) a reproducible multimodal data-acquisition pipeline for surgical robotics, (2) an open-source implementation of the VR–haptic–EEG–RIDL framework, and (3) a comprehensive evaluation protocol integrating behavioral, neurophysiological, and subjective metrics. Future work will validate the system on physical dVRK hardware, integrate additional biosignals (e.g., heart-rate variability, biomechanics via body tracking), and explore metareinforcement learning for rapid policy adaptation. By elevating the quality of decisionmaking datasets and enabling neuroadaptive training, this research aims to accelerate the deployment of capable, human-centered surgical robots in clinical practice. For readers and practitioners drawn to the intersection of immersive interface design, cognitive neuroscience, and robotic autonomy, this work offers both a urnkey framework and a rigorous evaluation methodology. Expect detailed guidance on implementing synchronized VR/haptic/EEG data streams, reproducible code for iffusion-policy pretraining, and a suite of metrics that link neurophysiological engagement to task performance. Whether the goal is to replicate these experiments, extend the pipeline to new robotic platforms, or inform the design of next-generation neuroadaptive systems, this thesis provides the conceptual foundations, open-source tools, and empirical benchmarks necessary to advance the state of surgical robotics and human-centered automation.

https://orcid.org/0000-0002-5845-6663