Initial commit

.gitignore

@@ -0,0 +1,21 @@
+*.avi
+*.egg-info
+*.pyc
+*.swp
+.vscode
+.idea/
+datasets
+datasets/
+devkit/outputs/
+env/
+eval-data
+eval-data/
+inpainting-results
+inpainting-results/
+inpainting-results-quantitative/
+inpainting-results-quantitative-summary.tsv
+old/
+slurm/logs/
+src/debug/
+video-lists/
+weights/

.gitmodules

@@ -0,0 +1,6 @@
+[submodule "video-inpainting-evaluation"]
+	path = video-inpainting-evaluation
+	url = https://github.com/MichiganCOG/video-inpainting-evaluation.git
+[submodule "video-inpainting-projects/STTN"]
+	path = video-inpainting-projects/STTN
+	url = https://github.com/MichiganCOG/STTN.git

LICENSE

@@ -0,0 +1,19 @@
+Copyright 2021 Ryan Szeto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,122 @@
+# The DEVIL Benchmark
+
+This code implements the Diagnostic Evaluation of Video Inpainting on Landscapes (DEVIL) benchmark, which is composed
+of a curated video/occlusion mask dataset and a comprehensive evaluation scheme. It has been tested on a Ubuntu 20.04
+machine with a GTX 1080Ti GPU.
+
+Further details are available in [our paper](https://arxiv.org/abs/2105.05332). If you use this work, please cite our
+paper:
+
+```
+@article{szeto2021devil,
+  author={Szeto, Ryan and Corso, Jason J.},
+  title={The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting},
+  journal={arXiv preprint arXiv:2105.05332},
+  year={2021}
+}
+```
+
+## Setup
+
+```bash
+# Clone project and evaluation submodule via git
+git clone https://github.com/MichiganCOG/devil.git
+cd devil
+git submodule update --init video-inpainting-evaluation
+
+# Initialize the DEVIL Python environment
+conda create -p ./env -y
+conda activate ./env
+conda env update -f environment.yml
+
+# Initialize the video inpainting evaluation library
+cd video-inpainting-evaluation
+# Follow the library's installation instructions. If these are out-of-date, refer to their instructions.
+conda create -p ./env -y
+conda activate ./env
+conda env update -f environment.yml
+./scripts/setup/install-flownet2.sh
+./scripts/setup/download-models.sh
+```
+
+The remaining instructions should be run from this project's root folder with the DEVIL conda environment activated 
+unless otherwise stated.
+
+## Quick Start
+
+The following script is a minimum working example for our benchmark. It downloads one DEVIL split and one set of
+predictions, evaluates the predictions on the split, and prints quantitative results to disk.
+
+```bash
+# Download DEVIL splits
+python -m src.main.download_devil_splits flickr-all -e
+python -m src.main.download_devil_splits fvi-fgd-h -e
+
+# Extract evaluation features
+source scripts/compute-devil-evaluation-features.sh flickr-all
+
+# Download sample predictions
+python -m src.main.download_inpainting_results cpnet flickr-all fvi-fgd-h
+
+# Run evaluation
+cd slurm
+python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local
+python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local -a ::include fid ::update
+
+# Print quantitative results to `inpainting-results-quantitative-summary.tsv`
+cd ..
+./scripts/print-quant-results-table.sh
+```
+
+The contents of `inpainting-results-quantitative-summary.tsv` should now look similar to the following:
+
+```text
+Method	PSNR ▲	SSIM ▲	LPIPS ▼ 	PVCS ▼	FID ▼	VFID ▼	VFID (clips) ▼	Warp error (mask) ▼	PCons (PSNR, mask) ▲
+../inpainting-results-quantitative/devil/flickr-all_fvi-fgd-h/cpnet.npz	36.62 	0.9807	0.002363	 0.1462 	 4.85	0.0367	0.0622	0.000957	40.33
+```
+
+## Running and Evaluating Custom Methods
+
+We encourage you to run and evaluate your own video inpainting method in our benchmark! To do so, first prepare an 
+execution script for your method, and run it on one of our DEVIL splits (instructions are available 
+[here](docs/running-video-inpainting-models.md#custom-video-inpainting-algorithms)). Then, evaluate your method using
+our evaluation helper script (instructions are available [here](docs/running-evaluation-details.md#running-evaluation)).
+
+## DEVIL Splits
+
+The DEVIL splits used in our paper are composed of the following pairs of video and mask subsets:
+
+| Video subset | Mask subset |
+|--------------|-------------|
+| flickr-all   | fvi-fgd-h   |
+| flickr-all   | fvi-fgd-l   |
+| flickr-all   | fvi-fgm-h   |
+| flickr-all   | fvi-fgm-l   |
+| flickr-all   | fvi-fgs-h   |
+| flickr-all   | fvi-fgs-l   |
+| flickr-bsm-h | fvi-all     |
+| flickr-bsm-l | fvi-all     |
+| flickr-cm-h  | fvi-all     |
+| flickr-cm-l  | fvi-all     |
+
+## More Information
+
+For more details on how to run video inpainting models and evaluation through our benchmark, please refer to the usage
+guides under the `docs` folder.
+
+### Licenses
+
+The code in this repository is available under the MIT License in `LICENSE` with the exceptions listed below:
+
+* Code snippets attributed to Stack Exchange and/or Stack Overflow is available under the
+  [Creative Commons Attribution-ShareAlike (CC BY-SA) License](https://creativecommons.org/licenses/by-sa/4.0/).
+* Code under each path in the table below is available under separate licenses; refer to the licenses in those paths for
+  more details.
+
+  | Path                           | License | Source                                                     |
+  |--------------------------------|---------|------------------------------------------------------------|
+  | `/src/raft/`                   | MIT     | https://github.com/princeton-vl/RAFT                       |
+  | `/video-inpainting-evaluation` | MIT     | https://github.com/MichiganCOG/video-inpainting-evaluation |
+
+* Code that is attributed, but does not fall under the above scenarios, is unlicensed. We thank the original authors for
+  their open contributions.

docs/devil-dataset-construction.md

@@ -0,0 +1,65 @@
+# DEVIL Dataset Construction
+
+This document explains how to re-generate the video and mask splits used in the DEVIL benchmark. It is mainly for
+completeness, since the pre-generated splits are already available online and can be downloaded.
+
+## Instructions
+
+### Setup
+
+```
+# Download weights for models used by DEVIL
+./scripts/download-weights.sh
+```
+
+### Download Flickr videos and construct clips
+
+```
+python -m src.main.download_flickr_videos
+python -m src.main.create_flickr_clips
+```
+
+### Score/classify Flickr clips
+
+```
+python -m src.main.score_flickr_clips
+```
+
+### Render occlusion masks
+
+```
+splits=( \
+  devil_fgs-l_fgm-l_fgd-l \
+  devil_fgs-l_fgm-l_fgd-h \
+  devil_fgs-l_fgm-h_fgd-l \
+  devil_fgs-l_fgm-h_fgd-h \
+  devil_fgs-h_fgm-l_fgd-l \
+  devil_fgs-h_fgm-l_fgd-h \
+  devil_fgs-h_fgm-h_fgd-l \
+  devil_fgs-h_fgm-h_fgd-h \
+)
+for split in ${splits[@]}; do python -m src.main.render_fvi_masks $split; done
+```
+
+### Construct DEVIL splits
+
+```
+python -m src.main.create_devil_video_lists
+for file in $(ls video-lists/fvi-masks); do python -m src.main.create_devil_split masks ${file%.txt}; done
+
+# Create source video splits. For some reason, only the "all" split got shuffled for our publication...
+python -m src.main.create_devil_split frames all
+python -m src.main.create_devil_split frames bsm-h --no_shuffle
+python -m src.main.create_devil_split frames bsm-l --no_shuffle
+python -m src.main.create_devil_split frames cm-h --no_shuffle
+python -m src.main.create_devil_split frames cm-l --no_shuffle
+```
+
+### OPTIONAL: Archive DEVIL splits in tar files
+
+You can save the DEVIL splits in `.tar` archives, which are a bit more NFS-friendly than the normal directory structure.
+To create them, run the following script:
+
+```
+./scripts/archive-devil-splits.sh
+```

docs/running-evaluation-details.md

@@ -0,0 +1,122 @@
+# Running Evaluation - Details
+
+This document describes how to perform evaluation on our DEVIL splits. It covers various details such as using `.tar`
+files, environment variables, and Slurm.
+
+## Computing Evaluation Features
+
+Note: Evaluation features take up a lot of space (~50GB per DEVIL split x 5 DEVIL splits). Before running the following
+commands, make sure that `<project root>/eval-data` points to a disk with sufficient capacity (you can use a symbolic
+link if necessary).
+
+First, make sure that the desired DEVIL splits corresponding to real videos exist as normal directories in
+`datasets/devil`. If not, download and extract them:
+
+```bash
+for split in ( all bsm-h bsm-l cm-h cm-l  ); do
+  python -m src.main.download_devil_splits flickr-$split -e;
+done
+```
+
+Then, generate the evaluation features:
+
+```bash
+for split in ( all bsm-h bsm-l cm-h cm-l  ); do
+  source scripts/compute-devil-evaluation-features.sh flickr-$split;
+done
+```
+
+### OPTIONAL: Archive evaluation features
+
+The evaluation features can be stored in `.tar` files, which are slightly more NFS-friendly than normal directories. To
+create them, run the following script:
+
+```bash
+./scripts/archive-evaluation-features.sh
+```
+
+## Downloading Inpainting Results
+
+Previously generated inpainting results can downloaded with `download_inpainting_results.py`. For instance, the
+following example downloads CPNet predictions on the `fvi-fgd-h` split:
+
+```bash
+python -m src.main.download_inpainting_results cpnet flickr-all fvi-fgd-h
+```
+
+For the list of supported models and DEVIL splits, refer to `src/main/download_inpainting_results.py`.  Results will be
+saved under the `inpainting-results/devil` folder, and will overwrite any existing results.
+
+## Running Evaluation
+
+**NOTE:** The commands in this section should be run from the `slurm` directory.
+
+The `submit-evaluate-inpainting.py` script runs evaluation metrics on our data. Below is an example of running this 
+script on CPNet predictions:
+
+```bash
+python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local
+```
+
+More generally, given a call of the following form:
+
+```bash
+python submit-evaluate-inpainting.py <model> <source split> <mask split> -m local
+```
+
+the script will look for results in `<project root>/inpainting-results/devil/<source split>_<mask split>/<model>` and 
+compare them to the ground-truth data at `<project root>/datasets/devil/<source split>` and 
+`<project root>/datasets/devil/<mask split>`.
+
+If you have predictions for your own model (e.g., at 
+`<project root>/inpainting-results/devil/flickr-all_fvi-fgd-h/mymodel`), your can use our evaluation script on them.
+
+### FID
+
+For some reason, the evaluation script usually fails when FID is computed alongside the other metrics; for this reason,
+FID is excluded from the default configuration of `submit-evaluate-inpainting.py` and should be computed in a separate
+run. The following example computes FID and adds it to the existing CPNet evaluation results:
+
+```bash
+python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local -a ::include fid ::update
+```
+
+### OPTIONAL: Advanced features
+
+If you want to use the `.tar` files that were optionally generated (i.e., for the DEVIL datasets and evaluation 
+features), use the `--use_tar` flag:
+
+```bash
+python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local --use_tar
+```
+
+Environment variables can be passed in as string pairs with the `-e` flag. The following example runs evaluation with a
+custom scratch storage location (which is used to temporally store data during evaluation):
+
+```bash
+python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local -e SCRATCH_ROOT /tmp/$USER
+```
+
+This script also supports Slurm job allocation, which can be done by removing the `-m local` flag:
+
+```bash
+python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h
+```
+
+`sbatch` arguments can be passed in as string pairs with the `-s` flag. This illustrative example runs evaluation on a
+custom partition:
+
+```bash
+python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -s partition my-partition
+```
+
+If necessary, `-s` and `-e` can be used at the same time.
+
+## Printing Evaluation Results
+
+The following script writes all quantitative results to `inpainting-results-quantitative-summary.tsv` as a tab-separated
+text table:
+
+```bash
+./scripts/print-quant-results-table.sh
+```