Projects

Maritime Open-vocabulary Panoptic Segmentation

Duration: 1. 9. 2024 – 31. 8. 2026

The ability to recognize objects beyond the training taxonomy is essential for computer vision models in safety-critical applications such as Unmanned Surface Vehicles (USVs). USVs are exposed to a wide variety of possible obstacles which is impossible to cover by collecting and labeling a dataset. Recent advances in multimodal vision and language learning have enabled training models that are capable of recognition on open vocabulary. This research proposal aims to investigate open-vocabulary panoptic segmentation within the maritime domain. This domain presents unique challenges due to limited labeled examples and a constrained training taxonomy in maritime datasets. To address these challenges, we intend to explore semi and weakly supervised learning techniques for open-vocabulary models. Moreover, USVs cannot depend on cloud processing due to the high latency and unreliable nature of mobile networks. As a result, all control critical computations must be conducted using embedded on-vehicle hardware. This prevents the deployment of current open-vocabulary segmentation models due to computational inefficiency. Therefore, this research proposal aims to investigate embedded-ready architectures for open-vocabulary panoptic segmentation.