Abschlussarbeiten, Bachelor- und Masterarbeiten

02.12.2025, Abschlussarbeiten, Bachelor- und Masterarbeiten

Vision-Language-Model-Driven RL for End-to-End Autonomous Driving on F1TENTH

End-to-end autonomous driving has seen rapid progress through deep reinforcement learning (RL) and large vision-language models (VLMs).
While RL policies excel at learning control behavior from interaction, they typically lack a higher-level understanding of complex driving scenes.
Modern VLMs, on the other hand, can interpret images, describe environments, and identify potential risks or relevant traffic elements.

This thesis explores how VLM-based scene analysis can be combined with RL-based end-to-end driving to improve decision-making in autonomous driving tasks.
The project will be developed in CARLA and later tested on a real F1TENTH racing car, offering the student a rare opportunity to deploy advanced algorithms on physical hardware such as Jetson Orin / Nano.

Objective

Design, implement, and evaluate a VLM-guided RL pipeline for end-to-end autonomous driving, focusing on:

1. VLM-based scenario analysis

Use a VLM to process CARLA camera views and extract semantic scene information.

Identify elements such as road layout, traffic participants, intentions, and potential hazards.

Produce interpretable semantic outputs that can support downstream RL tasks.

2. RL-based end-to-end driving policy

Train an RL agent that maps observations (raw images + optional high-level features) to steering and throttle.

Compare different integration strategies, such as:

Using VLM embeddings as additional observations,

VLM-informed reward shaping or auxiliary tasks,

hierarchical systems with VLM providing high-level cues.

3. Simulation experiments in CARLA

Implement a diverse set of training scenarios (e.g., overtaking, intersections, dynamic obstacles).

Provide quantitative and qualitative evaluation of performance and robustness.

4. Deployment on the F1TENTH car

Transfer the trained policy pipeline to an F1TENTH autonomous race car.

Evaluate real-world performance on a physical track.

Analyze differences between simulation and real-world behavior.

This thesis offers a unique opportunity to work across simulation, machine learning, robotics, and embedded deployment, culminating in running a real working system on the physical F1TENTH platform.

We Offer

A dynamic, research-oriented environment at the intersection of VLMs, RL, and autonomous racing.

Hands-on experience with:

CARLA simulator,

Deep reinforcement learning frameworks,

Vision-Language Models (e.g., BLIP, CLIP, LLaVA-style models),

F1TENTH autonomous racing platform,

Jetson Orin/Nano edge-compute hardware.

Opportunity to contribute to new research and potentially publish a scientific paper (based on merit).

Thesis can be written in English or German.

Requirements

Strong motivation and a creative, problem-solving mindset.

Excellent English or German proficiency.

Solid Python skills; experience with PyTorch preferred.

Background in at least one of:

Reinforcement learning (PPO, SAC, DQN, etc.),

Computer vision or VLMs/LLMs,

Robotics or autonomous driving.

Familiarity with development tools (Git, Ubuntu).

Nice to have

Experience with CARLA, MetaDrive, or other driving simulators.

Experience with F1TENTH, ROS2, or embedded GPU platforms.

Knowledge of scenario analysis, perception, or safety evaluation.

Start

Work can begin immediately.
If you are interested in this topic, please first have a look at our recent survey paper:
https://arxiv.org/abs/2506.11526

Then send a brief cover letter explaining why this topic interests you, along with your CV and current transcript of records to:
📧 yuan_avs.gao@tum.de

Kontakt: yuan_avs.gao@tum.de