Close

Haoying(Jack) Zhou

Researcher, Robotics and Software Engineer

Download Resume

About Me

portrait image

I' m a fourth-year Robotics Engineering Ph.D.student working in AIM Lab at Worcester Polytechnic Institute. I have worked with dVRK (da Vinci Research Kit) closely.
My concentration is robotics, system control and learning from demonstration. My research mainly focus on exoskeleton and surgical robot control system simulation, optimization and automation. I also have various experience on medical imaging, such as GUI construction using PyQt and data analysis on DICOM datasets. I am currently working on imitation learning with da Vinci surgical system.
I'm seeking for summer 2024 Intern.

Experience

Laboratory for Computational Sensing and Robotics, Johns Hopkins University

Visiting Graduate Scholar

- Construct research on suturing tasks automation based on skills learned from demonstrations in simulation
- Develop the software infrastructure of the 2021-2022 & 2023-2024 AccelNet Surgical Robotics Challenge .
- Build and test the whole robot system model for daVinci Si surgical system in ROS.
- Generate a synthetic 6D pose estimation dataset using multi-threading programming in simulation and train deep learning model on the dataset.
- Construct novel phantom & needle meshes for the simulation using Blender.
- Leverage Magic Leap 1 as an alternative stereo viewer and enable eye focus tracking

Philips Research North America

Image-Guided Therapy Robotics Intern

- Design a synthetic motion simulator with GUI in Python using 3D DICOM data as the only input.
- Implement phantom feature extraction, volume rendering and 3D volume visualization with VTK, ITK and VMTK.
- Construct data auto-generator based on the synthetic motion simulator with flexible configuration inputs.
- Integrate the simulator with Xbox controller as motion control input.
- Improve the refreshing rate of the simulator from 0.15 fps to 5 fps.
- Implement analytical analysis on the generated data.

Automation and Interventional Medicine Robotics Research Laboratory, Worcester Polytechnic Institute

Research Assistant

- Manage and lead all da Vinci Research Kit(dVRK) related projects, including suturing automation, dynamic identification, customized controller teleoperation, kinematic & dynamic controller design and customized tool integration.
- Reactivate a full da Vinci surgical system with dVRK software framework; actively repair, maintenance and improve both hardware and software infrastructures.
- Integrate probe for photoacoustic scanning on dVRK PSM (patient side manipulator) with auto-scanning enabled.
- Lead and manage user study for collecting human motion patterns on the physical dVRK.
- Implement suturing subtask automation with learning from demonstrations in simulation.

Worcester Polytechnic Institute

Teaching Assistant

- TA for Control Engineering, Introduction to Dynamic Systems, Design of Machine Elements
- Design and Construct lab documents and GitHub repository for Control Engineering course
- Lead conference lectures for undergraduate courses
- Hold TA session to answer students' questions about homework assignments, labs and lectures

Education

Worcester Polytechnic Institute

Sep 2020 - Present

Doctor of Philosophy in Robotics Engineering

GPA: 3.95/4.00

Boston University

Sep 2018 - May 2020

Master of Science in Mechanical Engineering

GPA: 3.78/4.00

University of California, Berkeley

Sep 2017 - May 2018

Bachelor of Science in Mechanical Engineering

GPA: 3.95/4.00

Beijing Institute of Technology

Sep 2014 - May 2018

Bachelor of Science in Mechanical Engineering

Ranked: 10/33

Senior year exchanged to University of California, Berkeley

Research Projects

Suturing Automation based on Imitation Learning

WPI AIM Lab & JHU LCSR

- Construct user study for collecting human demonstration in both simulation and physical dVRK
- Implement learning from demonstrations with dynamic movement primitives(DMP) for suturing automation
- Achieve high generality for suturing automation, on the order of 91.5%, from experienced human subjects' demonstrations

Suture Needle Pose Estimation in Simulation

WPI AIM Lab & JHU LCSR

- Implement suturing needle 6D pose estimation algorithm using keypoint-based method and deep learning(fast RCNN)
- Achieve estimated position errors less than 1mm and orientation errors around 2 degrees
- Win second prize on 2021 AccelNet Surgical Robotics Challenge
- Generate a synthetic segmented 6D pose estimation dataset in simulation and train deep learning models on the dataset
- Related publication: Publication 3

Point Cloud Completion

WPI VISLab

- Purpose a novel chamfer distance loss function for point cloud completion task using Landau distribution
- Achieve new state-of-the-art results on some benchmark datasets

dVRK Customized Photoacoustic Scanning Instrument Integration and Automation

WPI AIM Lab & WPI FUSION Lab

- Integrate a customized surgical instrument using ultrasound probe with dVRK for photoacoustic scanning
- Utilize April Tag and computer vision technique to find the transformation between the probe and the endoscope
- Construct kinematic model for the customized instrument
- Build a ROS network to enable scanning automation
- Related publication: Publication 1 & 2

dVRK PSM Teleoperation using sEMG Proportional Control

WPI AIM Lab

- Utilize sEMG bio-signal differences on muscles to control the surgical instrument grippers open or close
- Leverage the trackers of a motion capture system to implement dVRK PSM teleoperation for human subjects
- Related publication: Publication 4

Lower Limb Exoskeleton Automation based on Learning from Demonstration

WPI AIM Lab

- Construct and optimize an algorithm on imitation learning with Task-Parameterized Gaussian Mixture Model (TPGMM) applied to human walking strategies for lower-limb exoskeleton
- Collect data using motion capture technique with real-life human motion
- Leverage AMBF for simulating the exoskeleton and human lower limb movements
- Design and implement iLQR controller to above algorithm and managed to find the optimal weight matrix
- Related publication: Publication 5

Reach-to-Grasp Imitation Learning

BU Robotics Lab

-Implement generalized research-to-grasp automation using imitation learning with dynamic movement primitives(DMP) in Python
-Leverage VREP for simulating robot arm motions
-Collect human demonstration data using joystick controller
-Accomplish 6D pose imitation learning for the end-effector of Baxter Robot
-Write and defend my Master’s Thesis based the research project

View Project

Technical Projects

Physical dVRK Reactivation

Worcester Polytechnic Institute

-Deploy both hardware and software infrastructures of dVRK
-Resolve multiple mechanical failures for components of the da Vinci Surgical System, including broken joint encoders and joint brakes
-Upgrade the stereo viewer of the da Vinci Surgical System
-Implement full dVRK system teleoperation using MTM and dVRK control boxes
-Enable Geomagic Touch device and Razer Hydra controller as alternative teleoperation options for dVRK
-Implement dynamic identification on dVRK PSM

Surgical Instrument Lubrication

Worcester Polytechnic Institute

-Develop a practical method to lubricate the surgical instruments of the first generation da Vinci Surgical system
-Tighten the internal cables of the instruments to enable solid grasping solutions
-Decrease the mechanical noise due to the instruments by two or three orders of magnitude

Selected Course Projects

Adaptive Robustness Control Design for UAV with ROS Gazebo

Worcester Polytechnic Institute

-Construct dynamic model for Crazyflie 2.0 drone and simulate the drone in ROS Gazebo
-Implement robustness control algorithm for UAV hovering
-Design adaptive robustness controller for UAV following specific trajectories

Autonomous Race Car Modelling and Control

University of California, Berkeley

-Construct the vehicle model and its tire model under different road conditions for predicting the vehicle movements
-Utilize MATLAB, Simulink, and ROS Turtlesim to simulate BARC movements under different conditions
-Remotely connect to the Linux-based vehicle operating system via VNC Viewer and SSH
-Implement lane keeping, drift parking, and adaptive cruise control on BARC with linear controllers such as PID and LQR
-Analyze data in MATLAB, including camera calibration and result data analysis utilizing methods such as FFT interpolation and least square method to obtain and discern relationships between variables

Tri-Bot Design and Manufacturing

University of California, Berkeley

-Utilize MyRio as the microprocessor and program in LabVIEW for target detection and tracking algorithm in the upper camera system
-Leverage Arduino board as the processor and code in Arduino for lane keeping algorithm in the lower moving system
-Champion full design and manufacturing of product from conception to delivery

View Project

Other Selected Projects

Academic Publications

  1. Gao, Shang, Yang Wang, Xihan Ma, Haoying Zhou, Yiwei Jiang, Kehan Yang, Liang Lu et al. "Intraoperative laparoscopic photoacoustic image guidance system in the da Vinci surgical system." Biomedical optics express 14, no. 9 (2023): 4914-4928.
  2. Gao, Shang, Yang Wang, Haoying Zhou, Kehan Yang, Yiwei Jiang, Liang Lu, Shiyue Wang et al. "Laparoscopic photoacoustic imaging system integrated with the da Vinci surgical system." In Medical Imaging 2023: Image-Guided Procedures, Robotic Interventions, and Modeling, vol. 12466, pp. 62-70. SPIE, 2023.
  3. Jiang, Yiwei, Haoying Zhou, and Gregory S. Fischer. "Markerless Suture Needle Tracking From A Robotic Endoscope Based On Deep Learning." In 2023 International Symposium on Medical Robotics (ISMR), pp. 1-7. IEEE, 2023.
  4. Yang, Kehan, Tess B. Meier, Haoying Zhou, Gregory S. Fischer, and Christopher J. Nycz. "A sEMG Proportional Control for the Gripper of Patient Side Manipulator in da Vinci Surgical System." In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 4843-4848. IEEE, 2022.
  5. Goldfarb, Nathaniel, Haoying Zhou, Charles Bales, and Gregory S. Fischer. "Control of a lower limb exoskeleton using Learning from Demonstration and an iterative Linear Quadratic Regulator Controller: A simulation study." In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 4687-4693. IEEE, 2021.

** Some publications submitted in 2023 are still under review. The list of publications will be updated when the decisions are made.

Skills

Contact Information

hzhou6@wpi.edu / zhyjack0718@gmail.com
(774)-823-0984
Worcester, Massachusetts, USA