Fine grained visual analysis using natural language

Vestéinn Snaebjarnarson (Ph.D. Student)

The project is based around analysis of images by use of natural language, in particular for fine grained categorization and analysis. The idea is that the use of natural language will serve both as grounding for direct labelling as well as for comparison to other sources of information. Adaptation of existing general purpose multimodal systems for efficient and targeted use will be explored. Generation of captions from images with the purpose of explaining labels and scenes will be considered. The use of purely textual information for the purpose of analysing images will also be investigated, in particular for novel situations where no labelled images exist.

Primary Host:	Serge Belongie (University of Copenhagen & Cornell University)
Exchange Host:	Ryan Cotterell (ETH Zürich & University of Cambridge)
PhD Duration:	01 September 2022 - 31 August 2025
Exchange Duration:	01 June 2024 - 31 December 2024 - Ongoing