PhD Multimodal Understanding of Visual and Textual Information through Large Language Models for Advanced Cognitive Visual Tasks

The aim of this Ph.D. project is to develop a comprehensive framework that seamlessly integrates Large Language Models (LLMs) with advanced computer vision techniques to enhance the understanding, generation, and retrieval of multimedia content.Example Research Directions:1. Multimodal Representation Learning: Investigate methods for joint representation learning that fuses textual and visual information. This could involve adapting pre-trained LLMs to handle multimodal input, and training models to map between the semantic space of language and the visual space.2. Content Generation: Explore how LLMs can be used to generate descriptive and contextually relevant text for images and videos. This could involve conditioning the language model on visual features extracted by a convolutional neural network (CNN).3. Semantic Understanding: Explore how the combined model can achieve a deeper semantic understanding of content. This may involve incorporating domain-specific knowledge into the LLM or fine-tuning the model on specific datasets.Outcomes and Impact: The developed framework and algorithms will be validated in real-world applications, such as healthcare, sports, robotics, content creation platforms, or assistive technologies. The model and learning performance will be evaluated in terms of accuracy, robustness, and user subjective satisfaction. The scalability and efficiency of the proposed multimodal approach will be evaluated, allowing its potential it to handle large datasets and operate in real-time systems. The project will evaluate the model's performance on benchmark datasets and its application potential in collaboration with industry partners.

Applicants should have, or expect to achieve, at least a 2:1 Honours degree in computer science or a related science and engineering subject. A relevant Master’s degree, experience in machine learning, deep learning and computer vision, strong programming skills, and passion in interdisciplinary research and innovation will be advantages.