Skip to content

Cross-media Structured Common Space for Multimedia Event Extraction (ACL2020)

Notifications You must be signed in to change notification settings

FUNCTIONair-han/m2e2

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Cross-media Structured Common Space for Multimedia Event Extraction

Table of Contents

Overview

The code for paper Cross-media Structured Common Space for Multimedia Event Extraction.

Photo

Requirements

You can install the environment using requirements.txt for each component.

pip install -r requirements.txt

Data

Situation Recognition (Visual Event Extraction Data)

We download situation recognition data from imSitu. Please find the preprocessed data in PreprcessedSR.

ACE (Text Event Extraction Data)

We preprcoessed ACE following JMEE. The sample data format is in sample.json. Due to license reason, the ACE 2005 dataset is only accessible to those with LDC2006T06 license, please drop me an email manling2@illinois.edu showing your possession of the license for the processed data.

Voice of America Image-Caption Pairs

We crawled VOA image-captions to train the common space, the image-caption pairs and images can be downloaded using the URLs (We share image URLs instead of downloaded images due to license issue) using script in dataset_image_download.py. We preprocess the data including object detection, and parse text sentences. The preprocessed data is in PreprocessedVOA.

M2E2 (Multimedia Event Extraction Benchmark)

The images and text articles are in m2e2_rawdata, and annotations are in m2e2_annotation under data directory.

Vocabulary

Preprocessed vocabulary is in PreprocessedVocab.

Quickstart

Training

We have two variants to parse images into situation graph, one is parsing images to role-driven attention graph, and another is parsing images to object graphs.

(1) attention-graph based version

sh scripts/train/train_joint_att.sh 

(2) object-graph based version:

sh scripts/train/train_joint_obj.sh 

Please specify the data paths datadir, glovedir in scripts.

Testing

(1) attention-graph based version

sh test_joint.sh

(2) object-graph based version:

sh test_joint_object.sh

Please specify the data paths datadir, glovedir, and model paths checkpoint_sr, checkpoint_sr_params, checkpoint_ee, checkpoint_ee_params in scripts.

Citation

Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, Shih-Fu Chang. 2020. Cross-media Structured Common Space for Multimedia Event Extraction. Proceedings of The 58th Annual Meeting of the Association for Computational Linguistics.

@inproceedings{li2020multimediaevent,
    title={Cross-media Structured Common Space for Multimedia Event Extraction},
    author={Manling Li and Alireza Zareian and Qi Zeng and Spencer Whitehead and Di Lu and Heng Ji and Shih-Fu Chang},
    booktitle={Proceedings of The 58th Annual Meeting of the Association for Computational Linguistics},
    year={2020}

About

Cross-media Structured Common Space for Multimedia Event Extraction (ACL2020)

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 96.2%
  • Shell 3.8%