MA: Cognitively-realistic Spatial Visual Search Behavior for Embodied AI

Human gaze tracking during visual search

14.07.2026, Abschlussarbeiten, Bachelor- und Masterarbeiten

Embodied AI uses Large Vision Models to synthesize training data and validate performance in new tasks and environments. Crucially, models which learn from purpose-driven human demonstrations are assuming the goal is known beforehand, the task sequence is optimal and the steps are well-structured. Such models struggle in situations of uncertainty, e.g. during open-ended exploration. The lack of uncertain, search and exploration demonstration data inhibits the progress.  

Humans, on the other hand, are excelling in solving uncertain situations and unknown environments, e.g. through visual search and exploration. To that end, human gaze is a powerful tool to capture and describe an implicit cognitive process. For instance, when solving problems or acting under uncertainty, the gaze scan paths can indicate the sequence of attention and potential targets.

In this thesis, we aim to demonstrate a generalization of search behavior in Visual Language Action Models, using first-person, gaze-overlaid videos from eye-tracking glasses as input. Steps to that end include:

• Defining an environment, e.g. a cluttered room, the hidden objects, and the language/image instructions to be followed.
• Collecting the first-person data in the environment from diverse participants wearing gaze-tracking glasses.
• Fine-tuning an action-generation model to synthesize discrete motion based on language or image goals.
• Validating the synthetic data in terms of cognitive realism.

Prerequisites:

• Strong machine learning and deep learning background
• Excellent programming skills in Python
• Preferable experience in PyTorch or TensorFlow
• Background in Informatics or Robotics (CIT School)
• Independent and self-motivated working

If you are interested, just send an email to andrey.rudenko(at)tum.de with a short CV and your grade report.

Kontakt: andrey.rudenko@tum.de