Bridging the Domain Gap: Rendering and Testing Synthetic Training Data for Artifical Intelligence in the Operation Room

02.08.2024, Abschlussarbeiten, Bachelor- und Masterarbeiten

Overview

To ensure that artificial intelligence can handle real data as well as possible, the training data must be as similar as possible to the real data. This is particularly challenging in medical applications, as the image data is very different and diverse. This project aims to create the most realistic data from the instruments before and during an operation to learn to estimate their positions. Blood-immersed and metallic reflective instruments are particularly challenging. This project is based on preliminary work and includes creating such synthetic training data, the added value of which is measured with real data.

Background & Motivation

The situation in the operating room is often very complex, as many different instruments have to be prepared and used. To support the staff here, the KARVIMIO project is researching how a RGB-D camera can learn to identify the individual parts and their poses without any optical markers. This information is passed on to Augmented Reality head-mounted display so that it can be seen directly which parts need to be picked and how to assembled and applied. However, this only works well if the parts are well recognized. In order to train this AI, training data needs to be as realistic as possible, including, for example, instruments with bloody and metallic reflective surfaces. The proposed project can build on preliminary work and should optimize and evaluate it to this end.

Student’s Task

After familiarization with the framework used to date and the current state of the literature, an implementation strategy is defined. The aim is to integrate bloody and metallic reflective instrument parts into the synthetic rendering. Finally, the precision and robustness of the pose estimation will be evaluated with synthetic and real data. The work aims to integrate bloody and metallic-mirrored instruments into the rendering of synthetic training data for the training of pose estimation in an OR setting. The added value, as well as gained insights and limitations, are shown in a final evaluation.

Technical Prerequisites

Students should have basic knowledge of deep learning and be familiar with Pytorch. They should also have the ability to program with python. They should have strong motivation for medical-related AI.

Please send your transcript of records, CV and motivation to: Shiyu Li (shiyu.li@tum.de) with CC to hex-thesis.ortho@mh.tum.de

Literature
[1] Qiyu Dai et al. “Domain randomization-enhanced depth simulation and restoration for perceiving and grasping specular and transparent objects”. In: European Conference on Computer Vision. Springer. 2022, pp. 374–391.
[2] Paul Debevec. “Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography”. In: Acm siggraph 2008 classes. 2008, pp. 1–10.
[3] Maximilian Denninger et al. “Blenderproc2: A procedural pipeline for photorealistic rendering”. In: Journal of Open Source Software 8.82 (2023), p. 4901.
[4] Ori Gordon, Omri Avrahami, and Dani Lischinski. “Blended-nerf: Zero-shot object generation and blending in existing neural radiance fields”. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023, pp. 2941–2951.
[5] Yingwenqi Jiang et al. “Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 5322–5332.
[6] Shiyu Li et al. “GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance”. In: arXiv preprint arXiv:2402.07677 (2024).
[7] Zhen Li et al. “Multi-view inverse rendering for large-scale real-world indoor scenes”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 12499–12509. [8] Hui Tang and Kui Jia. “A new benchmark: On the utility of synthetic data with blender for bare supervised learning and downstream domain adaptation”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 15954–15964.
[9] Haoyuan Wang et al. “Inverse Rendering of Glossy Objects via the Neural Plenoptic Function and Radiance Fields”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 19999–20008.

Kontakt: hex-thesis.ortho@mh.tum.de, shiyu.li@tum.de