[Research] Multimodal Emotion Recognition for On-device AI

The codebase for our paper on Multi-modal Medical Dialogue Summarization

[FR|EN - Trio] 2023 - 2024 Centrale Méditerranée AI Master | Multimodal retranscription with text, audio and video

This study presents a novel multimodal fusion technique for disaster identification in Bangla, combining text and image data using the "BanglaCalamityMMD" dataset. Employing DisasterTextNet, DisasterImageNet, and DisasterMultFusionNet, the approach addresses a key gap in Bangla disaster research.

A Transferability-guided Protein-Ligand Interaction Prediction Method

We propose Multi-Modal Segmentation TransFormer (MMSFormer) that incorporates a novel fusion strategy to perform multimodal material segmentation.

Repository for context based emotion recognition

Web scraper for Wildberries + simple vectorization/multimodal embedding workflow

The code and data for the Paper 'Inferring Climate Change Stances from Multimodal Tweets' accepted by the Short Paper track of SIGIR 2024

VAPOR: Legged Robot Navigation in Outdoor Vegetation using Offline Reinforcement Learning (ICRA2024)

Repo for "Centaur: Robust Multimodal Fusion for Human Activity Recognition"

FusionBrain Challenge 2.0: creating multimodal multitask model

Source code of a sample iOS app for the paper "Automatic Fused Multimodal Deep Learning for Plant Identification" (Alfreds Lapkovskis, Natalia Nefedova & Ali Beikmohammadi, 2024)

E2E-MFD-HOD

Source code for the paper "Automatic Fused Multimodal Deep Learning for Plant Identification" (Alfreds Lapkovskis, Natalia Nefedova & Ali Beikmohammadi, 2024)

Official implementation of "Multi-scale Bottleneck Transformer for Weakly Supervised Multimodal Violence Detection"

Multimodal sentiment analysis

E2E-MFD-OOD

Few-Shot malware classification using fused features of static analysis and dynamic analysis （基于静态+动态分析的混合特征的小样本恶意代码分类框架）

A generalized self-supervised training paradigm for unimodal and multimodal alignment and fusion.