Skip to content

This is the official repository for Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks.

Notifications You must be signed in to change notification settings

UBC-NLP/peacock

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 

Repository files navigation

Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks

The University of British Columbia, Invertible AI

🔥 Details will be released. Stay tuned 🍻 👍

If you find this work useful for your research, please kindly cite our paper and star our repo.

Updates

Abstract

Multimodal large language models (MLLMs) have proven effective in a wide range of tasks requiring complex reasoning and linguistic comprehension. However, due to a lack of high-quality multimodal resources in languages other than English, success of MLLMs remains relatively limited to English-based settings. This poses significant challenges in developing comparable models for other languages, including even those with large speaker populations such as Arabic. To alleviate this challenge, we introduce a comprehensive family of Arabic MLLMs, dubbed Peacock, with strong vision and language capabilities. Through comprehensive qualitative and quantitative analysis, we demonstrate the solid performance of our models on various visual reasoning tasks and further show their emerging dialectal potential. Additionally, we introduce Henna, a new benchmark specifically designed for assessing MLLMs on aspects related to Arabic culture, setting the first stone for culturally-aware Arabic MLLMs.

Henna Benchmark

This collection of images showcases a curated subset selected from Henna dataset, representing 11 Arab countries, and capturing the essence of traditional food, local customs, historical monuments, everyday activities, and distinctive architecture that characterize the diverse and rich heritage of each region. Henna Samples

Henna Dataset Generation

Dataset Generation Example using GPT-4V. This figure demonstrates the process of generating a question-answer dataset for an attraction in Yemen as an example. For each site, an image and its corresponding Wikipedia article were used to provide GPT-4V with rich contextual information. The model then generated ten contextually relevant questions and answers per image. Henna Pipeline

Evaluation results

example Comparison between the performance of Peacock models on SEED-Benchmark dimensions.

Seed-Bench

Examples

example example

Citation

If you find this work useful for your research, please kindly cite our paper:

@article{alwajih2024peacock,
    title={Peacock: A Family of Arabic Multimodal Large Language Models
and Benchmarks},
    author={Alwajih, Fakhraddin and Nagoudi, El Moatez Billah and Bhatia, Gagan and Mohamed, Abdelrahman and Abdul-Mageed, Muhammad},
    journal={arXiv preprint arXiv:2403.01031},
    year={2024}
}

About

This is the official repository for Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published