Falcon-Release

PyTorch implementation of FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations

FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations

Lingjie Mei, Jiayuan Mao, Ziqi Wang, Chuang Gan, Joshua B. Tenenbaum

ICLR 2022

Paper Website

Getting started

Prerequisites

• Linux
• Python3
• PyTorch 1.6 with CUDA support
• Other required python packages specified by requirements.txt.

Installation

1. Clone this repository

   git clone https://github.com/JerryLingjieMei/FALCON-Release
   cd FALCON-Release

2. Create a conda environment for FALCON Model and install the requirements.

   conda create --n falcon-model
   conda activate falcon-model
   pip install -r requirements.txt
   conda install pytorch=1.6.0 cuda100 -c pytorch #Assume you use cuda version 10.0

3. Change DATASET_ROOT in tools.dataset_catalog to the folder where the datasets are stored. Download and unpack the base CUB, CLEVR and GQA datasets into DATASET/CUB-200-2011, DATASET/CLEVR_v1.0, DATASET/GQA, respectively. Download datasets for fast concept learning.

   . scripts/download_cub_data.sh ${DATASET_ROOT}
   . scripts/download_clevr_data.sh ${DATASET_ROOT}
   . scripts/download_gqa_data.sh ${DATASET_ROOT}

4. Download our weights for FALCON-G model.

   . scripts/download_cub_model.sh
   . scripts/download_clevr_model.sh
   . scripts/download_gqa_model.sh

Experiments(Final Testing)

1. Run the fast concept learning experiments via the config file cub/cub_fewshot_graphical_box.yaml, clevr/clevr_fewshot_graphical_0.yaml or gqa/gqa_fewshot_graphical_box.yaml.

   export NAME=cub/cub_fewshot_graphical_box; python tools/test_net.py --config-file experiments/${NAME}.yaml
   export NAME=clevr/clevr_fewshot_graphical_0; python tools/test_net.py --config-file experiments/${NAME}.yaml
   export NAME=gqa/gqa_fewshot_graphical_box; python tools/test_net.py --config-file experiments/${NAME}.yaml

Experiments(Training)

1. Here we use the CUB dataset as an example. Uncomment in scripts/download_cub_data.sh and scripts/download_cub_data.sh. Re-run them

   . scripts/download_cub_data.sh ${DATASET_ROOT}
   . scripts/download_cub_model.sh

2. Train optionally and test on the parser.

   export NAME=cub/cub_fewshot_build; python tools/train_net.py --config-file experiments/${NAME}.yaml
   export NAME=cub/cub_fewshot_build; python tools/test_net.py --config-file experiments/${NAME}.yaml

3. Train optionally the concept embeddings and feature extractor from the training concepts.

   export NAME=cub/cub_support_box; python tools/train_net.py --config-file experiments/${NAME}.yaml

4. Train optionally the fast concept learning models, e.g. FALCON-G.

   export NAME=cub/cub_fewshot_graphical_box; python tools/train_net.py --config-file experiments/${NAME}.yaml
   export NAME=cub/cub_fewshot_graphical_box; python tools/test_net.py --config-file experiments/${NAME}.yaml

Experiments (Additional)

1. Additional experiments can be configured by specifying:

   • TEMPLATE to represent the training stages, base datasets and embedding spaces.
   • MODEL.NAME to represent the type of fast concept learning models.
   • DATASETS to represent the datasets in the evaluations.

2. For other experiments, please fill free to contact the author via email or GitHub.