Realtime Updates for Scene Changes in Gaussian Splatting Scenes

16.03.2025, Abschlussarbeiten, Bachelor- und Masterarbeiten

This project aims to combine knowledge of classical reconstruction methods, real time capable algorithms and modern neural rendering to provide high-fidelity reconstructions from accidents and traffic scenes.

The Human-Centered Computing and Extended Reality Lab of the Professorship for Machine Intelligence in Orthopedics seeks applicants for Bachelor/Master Thesis for the Summer Semester 2025.

Project Description

The aim of XR teleguidance in medical scenarios is to address the shortage of specialists and to be able to act more quickly in emergency situations without the physical presence of the emergency personnel, i.e. the physician [5,6]. However, the real emergency begins before interacting with the patient. Not only do operations often have to be planned for patients, but their rescue is a crucial point which happens before the interaction with the physician. By using drones with built-in sensors, a mapping of the surroundings can be carried out before the rescuers reach the scene of the accident. Based on this mapping, a 3D reconstruction can be created that can be evaluated by the operations center and rescue strategies can be planned before the rescue personnel arrive on the scene. As the scene changes and the current part of the scene is not constantly captured the scene representation needs to be updated. The student should build upon real time-capable Gaussian Splatting SLAM algorithms and develop and integrate real-time capable updates.

Technical Prerequisites

The student should have some experience in python and deep learning.

Please send your transcript of records, CV and motivation to: Hannah Schieber (hannah.schieber@tum.de) with CC to hex-thesis.ortho@mh.tum.de

Literatur

[1] Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics, 42(4).
[2] Fu, Y., Liu, S., Kulkarni, A., Kautz, J., Efros, A. A., & Wang, X. (2023). COLMAP-Free 3D Gaussian Splatting. arXiv preprint arXiv:2312.07504.
[3] Charatan, D., Li, S., Tagliasacchi, A., & Sitzmann, V. (2023). pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. arXiv preprint arXiv:2312.12337.
[4] Wu, G., Yi, T., Fang, J., Xie, L., Zhang, X., Wei, W., ... & Wang, X. (2023). 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528.
[5] Roth, D., Yu, K., Pankratz, F., Gorbachev, G., Keller, A., Lazarovici, M., ... & Eck, U. (2021, March). Real-time mixed reality teleconsultation for intensive care units in pandemic situations. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 693-694). IEEE.
[6] Yu, K., Roth, D., Strak, R., Pankratz, F., Reichling, J., Kraetsch, C., ... & Eck, U. (2023, October). Mixed Reality 3D Teleconsultation for Emergency Decompressive Craniotomy: An Evaluation with Medical Residents. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 662-671). IEEE.
[7] Naumann, J., Xu, B., Leutenegger, S., & Zuo, X. (2023). NeRF-VO: Real-Time Sparse Visual Odometry with Neural Radiance Fields. arXiv preprint arXiv:2312.13471.
[8] Karaoglu, M. A., Schieber, H., Schischka, N., Görgülü, M., Grötzner, F., Ladikos, A., ... & Busam, B. (2023). DynaMoN: Motion-Aware Fast And Robust Camera Localization for Dynamic NeRF. arXiv preprint arXiv:2309.08927.
[9] Zollmann, S., Dickson, A., & Ventura, J. (2020, November). Casualvrvideos: Vr videos from casual stationary videos. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology (pp. 1-3). [10] Matsuki, H., Murai, R., Kelly, P. H., & Davison, A. J. (2023). Gaussian splatting slam. arXiv preprint arXiv:2312.06741.
[11] Tosi, F., Zhang, Y., Gong, Z., Sandström, E., Mattoccia, S., Oswald, M. R., & Poggi, M. (2024). How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Survey. arXiv preprint arXiv:2402.13255.
[12] Deng, T., Chen, Y., Zhang, L., Yang, J., Yuan, S., Wang, D., & Chen, W. (2024). Compact 3D Gaussian Splatting For Dense Visual SLAM. arXiv preprint arXiv:2403.11247.

Kontakt: hex-thesis.ortho@mh.tum.de, hannah.schieber@tum.de