Evaluation of Augmented Reality Assembly Guidance in the Operation Room

19.03.2025, Abschlussarbeiten, Bachelor- und Masterarbeiten

The Human-Centered Computing and Extended Reality Lab of the Professorship for Machine Intelligence in Orthopedics seeks applicants for Bachelor/Master Thesis for the Summer Semester 2025.

Abstract

Handling manual assembly tasks is a crucial part of everyday life. Especially with surgical instruments, proficient handling is crucial for flawless workflows, efficiency, and overall patient safety. Our research project introduced an AI-based pose estimation-driven assembly guidance with gaze interaction to trigger step-by-step instructions for scrub nurses. This thesis seeks to support empiric user studies with surgeons and scrub nurses to assess the usability and benefits for surgical crews.

Background & Motivation

The human hands remain indispensable for manual tasks in many everyday life scenarios, e.g., cooking, packing one's bag, or putting household items away in the cupboard. In a professional context, however, proficient handling is crucial, and errors can entail failures with great consequences. In complex and demanding medical environments, errors in handling surgical instruments can break the surgical workflow and endanger patient safety in the operation room (OR). Surgical crews are trained in this work environment, but they also have to assemble and operate specialized tools used only for rare procedures. The responsible scrub nurses cannot have experience with all instruments and might not have seen it before in the case of standing in. We tackle a part of this problem by investigating surgical tool assembly with augmented reality (AR) guidance to reduce interruptions. AR assembly guidance can generally support assembly processes so that users not only experience less cognitive load but also make fewer mistakes and finish the task in a shorter time. Therefore, they must be given precise and easy-to-follow visual advice at the right time and position, and the GUI must be easy and efficient to operate. Please see our project website for more details: https://hex-lab.io/projects/karvimio/.

Student’s Task

First, the student is expected to familiarize with the existing hardware (HoloLens, Kinect) and software (Unity, C#, C++) stack. Second, they should read up on user study design and empiric HCI evaluation to support running an extensive user study with scrub nurses and surgeons. Depending on the commitment and scope of the work, the project will aim for a scientific publication.

Technical Prerequisites

Applicants should be familiar with Unity, Augmented Reality (HoloLens), and basic software development. They should know human-computer interaction and have an intrinsic interest in empiric experiments and working independently. Students should be able to speak German confidently and be communicative.

Please send your transcript of records, CV and motivation to: Julian Kreimeier (julian.kreimeier@tum.de) with CC to hex-thesis.ortho@mh.tum.de

Literatur

Blattgerste, J., Strenge, B., Renner, P., Pfeiffer, T., & Essig, K. (2017, June). Comparing conventional and augmented reality instructions for manual assembly tasks. In Proceedings of the 10th international conference on pervasive technologies related to assistive environments (pp. 75-82).
Kleinbeck, C., Schieber, H., Andress, S., Krautz, C., & Roth, D. (2022, March). ARTFM: Augmented reality visualization of tool functionality manuals in operating rooms. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 736-737). IEEE.
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023). A survey on measuring cognitive workload in human-computer interaction. ACM Computing Surveys, 55(13s), 1-39.
Merino, L., Schwarzl, M., Kraus, M., Sedlmair, M., Schmalstieg, D., & Weiskopf, D. (2020, November). Evaluating mixed and augmented reality: A systematic literature review (2009-2019). In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 438-451). IEEE.
Field, A. (2002). How to design and report experiments.

Kontakt: hex-thesis.ortho@mh.tum.de, julian.kreimeier@tum.de