Master thesis or research internship - Explainability Assessment for Video Synthesis

22.01.2025, Abschlussarbeiten, Bachelor- und Masterarbeiten

External master thesis or research internship supervised by the Chair of Media Technology and Sureel.ai

Generative AI models are demonstrating strong performance in various domains. Models such as Stable Diffusion, trained using billions of images, are capable of generating highly realistic images based on text prompts or input images. More recently, video synthesis models such as Sora by OpenAI have demonstrated that temporally consistent image sequences can be generated as well.

At Sureel Inc., an explainability framework is available that calculates the influence of training data for both image and audio synthesis. For video synthesis, such a framework is not available yet. To obtain an assessment of the influence training material had on video synthesis models, the impact of both audio and image data needs to be considered.

In this thesis, existing explainability techniques should be extended and applied to video synthesis. Using an open-source model such as Mochi 1, the challenges of extending image synthesis explainability to video synthesis should be identified and solutions for them should be developed. That includes considering how many visually distinct scenes a video contains and how to incorporate the temporal dimension into the influence computation. Such techniques could include identifying sequences of key frames from each training and testing scene, as well as identifying critical audio content such as copyrighted music or copyrighted text from spoken word. The goal of this thesis is an algorithm that computes the influence of a given training video on a generated video in a temporal fashion, both for the audio and the visual component of the video.

This thesis will be conducted externally at Sureel Inc., a silicon-valley-based startup with an office in Munch specializing in explainable and legal generative AI content.