Neural 3D Reconstruction and Region of Interest Real-time Speedup for Drone Rescue Teleguidance

09.09.2024, Abschlussarbeiten, Bachelor- und Masterarbeiten

Overview

This project aims to combine knowledge of classical reconstruction methods and modern neural rendering to provide high-fidelity reconstructions from accidents and traffic scenes.

Background & Motivation

The aim of XR teleguidance in medical scenarios is to address the shortage of specialists and to be able to act more quickly in emergency situations without the physical presence of the emergency personnel, i.e. the physician [5,6]. However, the real emergency begins before interacting with the patient. Not only do operations often have to be planned for patients, but their rescue is a crucial point which happens before the interaction with the physician. By using drones with built-in sensors, a mapping of the surroundings can be carried out before the rescuers reach the scene of the accident. Based on this mapping, a 3D reconstruction can be created that can be evaluated by the operations center and rescue strategies can be planned before the rescue personnel arrive on the scene.

Student’s Task

Initial aerial data could be generated using: https://www.google.com/intl/de/earth/about/versions/ the student should develop a streaming algorithm, streaming the aerial data retrieved by real or synthetic images to Unity/WebGL to enable a 3D visualization of the environment. The idea is to provide point cloud streaming and only provide high-fidelity reconstruction using NeRF or Gaussian Splatting on user annotated areas of interest. For the project these areas could be cars or parts of the street. Ideally the final approach is interactable or visualizable in Virtual Reality using Unity directly or WebGL. There exists by now several Gaussian Splatting + SLAM approaches [10, 11]; the idea is to optimize them towards Regions-Of-Interest and reduced preprocessing times.

Technical Prerequisites

The student should have some experience in data processing and deep learning. Knowledge in 3D computer vision is beneficial. Additional knowledge in terms of different sensor types etc are beneficial as well but not mandatory.

Please send your transcript of records, CV and motivation to: Hannah Schieber (hannah.schieber@tum.de) with CC to hex-thesis.ortho@mh.tum.de

Literature
[1] Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics, 42(4).
[2] Fu, Y., Liu, S., Kulkarni, A., Kautz, J., Efros, A. A., & Wang, X. (2023). COLMAP-Free 3D Gaussian Splatting. arXiv preprint arXiv:2312.07504.
[3] Charatan, D., Li, S., Tagliasacchi, A., & Sitzmann, V. (2023). pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. arXiv preprint arXiv:2312.12337.
[4] Wu, G., Yi, T., Fang, J., Xie, L., Zhang, X., Wei, W., ... & Wang, X. (2023). 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528.
[5] Roth, D., Yu, K., Pankratz, F., Gorbachev, G., Keller, A., Lazarovici, M., ... & Eck, U. (2021, March). Real-time mixed reality teleconsultation for intensive care units in pandemic situations. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 693-694). IEEE.
[6] Yu, K., Roth, D., Strak, R., Pankratz, F., Reichling, J., Kraetsch, C., ... & Eck, U. (2023, October). Mixed Reality 3D Teleconsultation for Emergency Decompressive Craniotomy: An Evaluation with Medical Residents. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 662-671). IEEE.
[7] Naumann, J., Xu, B., Leutenegger, S., & Zuo, X. (2023). NeRF-VO: Real-Time Sparse Visual Odometry with Neural Radiance Fields. arXiv preprint arXiv:2312.13471.
[8] Karaoglu, M. A., Schieber, H., Schischka, N., Görgülü, M., Grötzner, F., Ladikos, A., ... & Busam, B. (2023). DynaMoN: Motion-Aware Fast And Robust Camera Localization for Dynamic NeRF. arXiv preprint arXiv:2309.08927.
[9] Zollmann, S., Dickson, A., & Ventura, J. (2020, November). Casualvrvideos: Vr videos from casual stationary videos. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology (pp. 1-3). [10] Matsuki, H., Murai, R., Kelly, P. H., & Davison, A. J. (2023). Gaussian splatting slam. arXiv preprint arXiv:2312.06741.
[11] Tosi, F., Zhang, Y., Gong, Z., Sandström, E., Mattoccia, S., Oswald, M. R., & Poggi, M. (2024). How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Survey. arXiv preprint arXiv:2402.13255.
[12] Deng, T., Chen, Y., Zhang, L., Yang, J., Yuan, S., Wang, D., & Chen, W. (2024). Compact 3D Gaussian Splatting For Dense Visual SLAM. arXiv preprint arXiv:2403.11247.

Kontakt: hex-thesis.ortho@mh.tum.de, hannah.schieber@tum.de

Mehr Information

https://hex-lab.io