diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..aafa2a4 --- /dev/null +++ b/.gitignore @@ -0,0 +1,21 @@ +*.avi +*.egg-info +*.pyc +*.swp +.vscode +.idea/ +datasets +datasets/ +devkit/outputs/ +env/ +eval-data +eval-data/ +inpainting-results +inpainting-results/ +inpainting-results-quantitative/ +inpainting-results-quantitative-summary.tsv +old/ +slurm/logs/ +src/debug/ +video-lists/ +weights/ diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000..6b7f559 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,6 @@ +[submodule "video-inpainting-evaluation"] + path = video-inpainting-evaluation + url = https://github.com/MichiganCOG/video-inpainting-evaluation.git +[submodule "video-inpainting-projects/STTN"] + path = video-inpainting-projects/STTN + url = https://github.com/MichiganCOG/STTN.git diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..57d2169 --- /dev/null +++ b/LICENSE @@ -0,0 +1,19 @@ +Copyright 2021 Ryan Szeto + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..cbc505b --- /dev/null +++ b/README.md @@ -0,0 +1,122 @@ +# The DEVIL Benchmark + +This code implements the Diagnostic Evaluation of Video Inpainting on Landscapes (DEVIL) benchmark, which is composed +of a curated video/occlusion mask dataset and a comprehensive evaluation scheme. It has been tested on a Ubuntu 20.04 +machine with a GTX 1080Ti GPU. + +Further details are available in [our paper](https://arxiv.org/abs/2105.05332). If you use this work, please cite our +paper: + +``` +@article{szeto2021devil, + author={Szeto, Ryan and Corso, Jason J.}, + title={The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting}, + journal={arXiv preprint arXiv:2105.05332}, + year={2021} +} +``` + +## Setup + +```bash +# Clone project and evaluation submodule via git +git clone https://github.com/MichiganCOG/devil.git +cd devil +git submodule update --init video-inpainting-evaluation + +# Initialize the DEVIL Python environment +conda create -p ./env -y +conda activate ./env +conda env update -f environment.yml + +# Initialize the video inpainting evaluation library +cd video-inpainting-evaluation +# Follow the library's installation instructions. If these are out-of-date, refer to their instructions. +conda create -p ./env -y +conda activate ./env +conda env update -f environment.yml +./scripts/setup/install-flownet2.sh +./scripts/setup/download-models.sh +``` + +The remaining instructions should be run from this project's root folder with the DEVIL conda environment activated +unless otherwise stated. + +## Quick Start + +The following script is a minimum working example for our benchmark. It downloads one DEVIL split and one set of +predictions, evaluates the predictions on the split, and prints quantitative results to disk. + +```bash +# Download DEVIL splits +python -m src.main.download_devil_splits flickr-all -e +python -m src.main.download_devil_splits fvi-fgd-h -e + +# Extract evaluation features +source scripts/compute-devil-evaluation-features.sh flickr-all + +# Download sample predictions +python -m src.main.download_inpainting_results cpnet flickr-all fvi-fgd-h + +# Run evaluation +cd slurm +python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local +python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local -a ::include fid ::update + +# Print quantitative results to `inpainting-results-quantitative-summary.tsv` +cd .. +./scripts/print-quant-results-table.sh +``` + +The contents of `inpainting-results-quantitative-summary.tsv` should now look similar to the following: + +```text +Method PSNR ▲ SSIM ▲ LPIPS ▼ PVCS ▼ FID ▼ VFID ▼ VFID (clips) ▼ Warp error (mask) ▼ PCons (PSNR, mask) ▲ +../inpainting-results-quantitative/devil/flickr-all_fvi-fgd-h/cpnet.npz 36.62 0.9807 0.002363 0.1462 4.85 0.0367 0.0622 0.000957 40.33 +``` + +## Running and Evaluating Custom Methods + +We encourage you to run and evaluate your own video inpainting method in our benchmark! To do so, first prepare an +execution script for your method, and run it on one of our DEVIL splits (instructions are available +[here](docs/running-video-inpainting-models.md#custom-video-inpainting-algorithms)). Then, evaluate your method using +our evaluation helper script (instructions are available [here](docs/running-evaluation-details.md#running-evaluation)). + +## DEVIL Splits + +The DEVIL splits used in our paper are composed of the following pairs of video and mask subsets: + +| Video subset | Mask subset | +|--------------|-------------| +| flickr-all | fvi-fgd-h | +| flickr-all | fvi-fgd-l | +| flickr-all | fvi-fgm-h | +| flickr-all | fvi-fgm-l | +| flickr-all | fvi-fgs-h | +| flickr-all | fvi-fgs-l | +| flickr-bsm-h | fvi-all | +| flickr-bsm-l | fvi-all | +| flickr-cm-h | fvi-all | +| flickr-cm-l | fvi-all | + +## More Information + +For more details on how to run video inpainting models and evaluation through our benchmark, please refer to the usage +guides under the `docs` folder. + +### Licenses + +The code in this repository is available under the MIT License in `LICENSE` with the exceptions listed below: + +* Code snippets attributed to Stack Exchange and/or Stack Overflow is available under the + [Creative Commons Attribution-ShareAlike (CC BY-SA) License](https://creativecommons.org/licenses/by-sa/4.0/). +* Code under each path in the table below is available under separate licenses; refer to the licenses in those paths for + more details. + + | Path | License | Source | + |--------------------------------|---------|------------------------------------------------------------| + | `/src/raft/` | MIT | https://github.com/princeton-vl/RAFT | + | `/video-inpainting-evaluation` | MIT | https://github.com/MichiganCOG/video-inpainting-evaluation | + +* Code that is attributed, but does not fall under the above scenarios, is unlicensed. We thank the original authors for + their open contributions. diff --git a/devkit/outputs/.gitkeep b/devkit/outputs/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/devkit/sample-inpainting-inputs/frame_0000_gt.png b/devkit/sample-inpainting-inputs/frame_0000_gt.png new file mode 100644 index 0000000..1634632 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0000_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0000_mask.png b/devkit/sample-inpainting-inputs/frame_0000_mask.png new file mode 100644 index 0000000..8bb551e Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0000_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0001_gt.png b/devkit/sample-inpainting-inputs/frame_0001_gt.png new file mode 100644 index 0000000..5afc3fb Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0001_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0001_mask.png b/devkit/sample-inpainting-inputs/frame_0001_mask.png new file mode 100644 index 0000000..2e8a580 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0001_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0002_gt.png b/devkit/sample-inpainting-inputs/frame_0002_gt.png new file mode 100644 index 0000000..527010f Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0002_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0002_mask.png b/devkit/sample-inpainting-inputs/frame_0002_mask.png new file mode 100644 index 0000000..a725314 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0002_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0003_gt.png b/devkit/sample-inpainting-inputs/frame_0003_gt.png new file mode 100644 index 0000000..93cd25f Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0003_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0003_mask.png b/devkit/sample-inpainting-inputs/frame_0003_mask.png new file mode 100644 index 0000000..d602848 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0003_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0004_gt.png b/devkit/sample-inpainting-inputs/frame_0004_gt.png new file mode 100644 index 0000000..c1ca39c Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0004_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0004_mask.png b/devkit/sample-inpainting-inputs/frame_0004_mask.png new file mode 100644 index 0000000..6423a8b Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0004_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0005_gt.png b/devkit/sample-inpainting-inputs/frame_0005_gt.png new file mode 100644 index 0000000..5dec2e6 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0005_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0005_mask.png b/devkit/sample-inpainting-inputs/frame_0005_mask.png new file mode 100644 index 0000000..b468b34 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0005_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0006_gt.png b/devkit/sample-inpainting-inputs/frame_0006_gt.png new file mode 100644 index 0000000..af7b987 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0006_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0006_mask.png b/devkit/sample-inpainting-inputs/frame_0006_mask.png new file mode 100644 index 0000000..265ff45 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0006_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0007_gt.png b/devkit/sample-inpainting-inputs/frame_0007_gt.png new file mode 100644 index 0000000..fc064d1 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0007_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0007_mask.png b/devkit/sample-inpainting-inputs/frame_0007_mask.png new file mode 100644 index 0000000..96a260c Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0007_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0008_gt.png b/devkit/sample-inpainting-inputs/frame_0008_gt.png new file mode 100644 index 0000000..c478ef3 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0008_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0008_mask.png b/devkit/sample-inpainting-inputs/frame_0008_mask.png new file mode 100644 index 0000000..f6ed8ac Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0008_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0009_gt.png b/devkit/sample-inpainting-inputs/frame_0009_gt.png new file mode 100644 index 0000000..52b0a57 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0009_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0009_mask.png b/devkit/sample-inpainting-inputs/frame_0009_mask.png new file mode 100644 index 0000000..363f374 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0009_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0010_gt.png b/devkit/sample-inpainting-inputs/frame_0010_gt.png new file mode 100644 index 0000000..f23bc3a Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0010_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0010_mask.png b/devkit/sample-inpainting-inputs/frame_0010_mask.png new file mode 100644 index 0000000..f393620 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0010_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0011_gt.png b/devkit/sample-inpainting-inputs/frame_0011_gt.png new file mode 100644 index 0000000..63db648 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0011_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0011_mask.png b/devkit/sample-inpainting-inputs/frame_0011_mask.png new file mode 100644 index 0000000..ea86cc0 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0011_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0012_gt.png b/devkit/sample-inpainting-inputs/frame_0012_gt.png new file mode 100644 index 0000000..f7b53ea Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0012_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0012_mask.png b/devkit/sample-inpainting-inputs/frame_0012_mask.png new file mode 100644 index 0000000..f5b966c Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0012_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0013_gt.png b/devkit/sample-inpainting-inputs/frame_0013_gt.png new file mode 100644 index 0000000..3d49a84 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0013_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0013_mask.png b/devkit/sample-inpainting-inputs/frame_0013_mask.png new file mode 100644 index 0000000..23574ce Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0013_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0014_gt.png b/devkit/sample-inpainting-inputs/frame_0014_gt.png new file mode 100644 index 0000000..23b50f6 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0014_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0014_mask.png b/devkit/sample-inpainting-inputs/frame_0014_mask.png new file mode 100644 index 0000000..29f63fc Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0014_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0015_gt.png b/devkit/sample-inpainting-inputs/frame_0015_gt.png new file mode 100644 index 0000000..a2b3eef Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0015_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0015_mask.png b/devkit/sample-inpainting-inputs/frame_0015_mask.png new file mode 100644 index 0000000..31b1f41 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0015_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0016_gt.png b/devkit/sample-inpainting-inputs/frame_0016_gt.png new file mode 100644 index 0000000..9d815fa Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0016_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0016_mask.png b/devkit/sample-inpainting-inputs/frame_0016_mask.png new file mode 100644 index 0000000..63f7246 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0016_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0017_gt.png b/devkit/sample-inpainting-inputs/frame_0017_gt.png new file mode 100644 index 0000000..15175a7 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0017_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0017_mask.png b/devkit/sample-inpainting-inputs/frame_0017_mask.png new file mode 100644 index 0000000..cbbc15b Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0017_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0018_gt.png b/devkit/sample-inpainting-inputs/frame_0018_gt.png new file mode 100644 index 0000000..581088b Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0018_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0018_mask.png b/devkit/sample-inpainting-inputs/frame_0018_mask.png new file mode 100644 index 0000000..9d42736 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0018_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0019_gt.png b/devkit/sample-inpainting-inputs/frame_0019_gt.png new file mode 100644 index 0000000..6b3038e Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0019_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0019_mask.png b/devkit/sample-inpainting-inputs/frame_0019_mask.png new file mode 100644 index 0000000..4696e58 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0019_mask.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0020_gt.png b/devkit/sample-inpainting-inputs/frame_0020_gt.png new file mode 100644 index 0000000..f723007 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0020_gt.png differ diff --git a/devkit/sample-inpainting-inputs/frame_0020_mask.png b/devkit/sample-inpainting-inputs/frame_0020_mask.png new file mode 100644 index 0000000..2ff9fb8 Binary files /dev/null and b/devkit/sample-inpainting-inputs/frame_0020_mask.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0000_pred.png b/devkit/sample-inpainting-outputs/frame_0000_pred.png new file mode 100644 index 0000000..19be569 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0000_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0001_pred.png b/devkit/sample-inpainting-outputs/frame_0001_pred.png new file mode 100644 index 0000000..b796c41 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0001_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0002_pred.png b/devkit/sample-inpainting-outputs/frame_0002_pred.png new file mode 100644 index 0000000..256f7c3 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0002_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0003_pred.png b/devkit/sample-inpainting-outputs/frame_0003_pred.png new file mode 100644 index 0000000..c151df3 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0003_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0004_pred.png b/devkit/sample-inpainting-outputs/frame_0004_pred.png new file mode 100644 index 0000000..b1c3580 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0004_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0005_pred.png b/devkit/sample-inpainting-outputs/frame_0005_pred.png new file mode 100644 index 0000000..0b80375 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0005_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0006_pred.png b/devkit/sample-inpainting-outputs/frame_0006_pred.png new file mode 100644 index 0000000..ea81780 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0006_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0007_pred.png b/devkit/sample-inpainting-outputs/frame_0007_pred.png new file mode 100644 index 0000000..3e54392 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0007_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0008_pred.png b/devkit/sample-inpainting-outputs/frame_0008_pred.png new file mode 100644 index 0000000..fcd72a1 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0008_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0009_pred.png b/devkit/sample-inpainting-outputs/frame_0009_pred.png new file mode 100644 index 0000000..0d58a9c Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0009_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0010_pred.png b/devkit/sample-inpainting-outputs/frame_0010_pred.png new file mode 100644 index 0000000..5203354 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0010_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0011_pred.png b/devkit/sample-inpainting-outputs/frame_0011_pred.png new file mode 100644 index 0000000..bfd2161 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0011_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0012_pred.png b/devkit/sample-inpainting-outputs/frame_0012_pred.png new file mode 100644 index 0000000..b4fc3d7 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0012_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0013_pred.png b/devkit/sample-inpainting-outputs/frame_0013_pred.png new file mode 100644 index 0000000..b020606 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0013_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0014_pred.png b/devkit/sample-inpainting-outputs/frame_0014_pred.png new file mode 100644 index 0000000..27d4af3 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0014_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0015_pred.png b/devkit/sample-inpainting-outputs/frame_0015_pred.png new file mode 100644 index 0000000..d7e54ef Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0015_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0016_pred.png b/devkit/sample-inpainting-outputs/frame_0016_pred.png new file mode 100644 index 0000000..15eeb45 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0016_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0017_pred.png b/devkit/sample-inpainting-outputs/frame_0017_pred.png new file mode 100644 index 0000000..43c7bb5 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0017_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0018_pred.png b/devkit/sample-inpainting-outputs/frame_0018_pred.png new file mode 100644 index 0000000..28cd2b3 Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0018_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0019_pred.png b/devkit/sample-inpainting-outputs/frame_0019_pred.png new file mode 100644 index 0000000..c2ba16d Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0019_pred.png differ diff --git a/devkit/sample-inpainting-outputs/frame_0020_pred.png b/devkit/sample-inpainting-outputs/frame_0020_pred.png new file mode 100644 index 0000000..8e44d8a Binary files /dev/null and b/devkit/sample-inpainting-outputs/frame_0020_pred.png differ diff --git a/docs/devil-dataset-construction.md b/docs/devil-dataset-construction.md new file mode 100644 index 0000000..f9ac953 --- /dev/null +++ b/docs/devil-dataset-construction.md @@ -0,0 +1,65 @@ +# DEVIL Dataset Construction + +This document explains how to re-generate the video and mask splits used in the DEVIL benchmark. It is mainly for +completeness, since the pre-generated splits are already available online and can be downloaded. + +## Instructions + +### Setup + +``` +# Download weights for models used by DEVIL +./scripts/download-weights.sh +``` + +### Download Flickr videos and construct clips + +``` +python -m src.main.download_flickr_videos +python -m src.main.create_flickr_clips +``` + +### Score/classify Flickr clips + +``` +python -m src.main.score_flickr_clips +``` + +### Render occlusion masks + +``` +splits=( \ + devil_fgs-l_fgm-l_fgd-l \ + devil_fgs-l_fgm-l_fgd-h \ + devil_fgs-l_fgm-h_fgd-l \ + devil_fgs-l_fgm-h_fgd-h \ + devil_fgs-h_fgm-l_fgd-l \ + devil_fgs-h_fgm-l_fgd-h \ + devil_fgs-h_fgm-h_fgd-l \ + devil_fgs-h_fgm-h_fgd-h \ +) +for split in ${splits[@]}; do python -m src.main.render_fvi_masks $split; done +``` + +### Construct DEVIL splits + +``` +python -m src.main.create_devil_video_lists +for file in $(ls video-lists/fvi-masks); do python -m src.main.create_devil_split masks ${file%.txt}; done + +# Create source video splits. For some reason, only the "all" split got shuffled for our publication... +python -m src.main.create_devil_split frames all +python -m src.main.create_devil_split frames bsm-h --no_shuffle +python -m src.main.create_devil_split frames bsm-l --no_shuffle +python -m src.main.create_devil_split frames cm-h --no_shuffle +python -m src.main.create_devil_split frames cm-l --no_shuffle +``` + +### OPTIONAL: Archive DEVIL splits in tar files + +You can save the DEVIL splits in `.tar` archives, which are a bit more NFS-friendly than the normal directory structure. +To create them, run the following script: + +``` +./scripts/archive-devil-splits.sh +``` diff --git a/docs/running-evaluation-details.md b/docs/running-evaluation-details.md new file mode 100644 index 0000000..a2aac32 --- /dev/null +++ b/docs/running-evaluation-details.md @@ -0,0 +1,122 @@ +# Running Evaluation - Details + +This document describes how to perform evaluation on our DEVIL splits. It covers various details such as using `.tar` +files, environment variables, and Slurm. + +## Computing Evaluation Features + +Note: Evaluation features take up a lot of space (~50GB per DEVIL split x 5 DEVIL splits). Before running the following +commands, make sure that `/eval-data` points to a disk with sufficient capacity (you can use a symbolic +link if necessary). + +First, make sure that the desired DEVIL splits corresponding to real videos exist as normal directories in +`datasets/devil`. If not, download and extract them: + +```bash +for split in ( all bsm-h bsm-l cm-h cm-l ); do + python -m src.main.download_devil_splits flickr-$split -e; +done +``` + +Then, generate the evaluation features: + +```bash +for split in ( all bsm-h bsm-l cm-h cm-l ); do + source scripts/compute-devil-evaluation-features.sh flickr-$split; +done +``` + +### OPTIONAL: Archive evaluation features + +The evaluation features can be stored in `.tar` files, which are slightly more NFS-friendly than normal directories. To +create them, run the following script: + +```bash +./scripts/archive-evaluation-features.sh +``` + +## Downloading Inpainting Results + +Previously generated inpainting results can downloaded with `download_inpainting_results.py`. For instance, the +following example downloads CPNet predictions on the `fvi-fgd-h` split: + +```bash +python -m src.main.download_inpainting_results cpnet flickr-all fvi-fgd-h +``` + +For the list of supported models and DEVIL splits, refer to `src/main/download_inpainting_results.py`. Results will be +saved under the `inpainting-results/devil` folder, and will overwrite any existing results. + +## Running Evaluation + +**NOTE:** The commands in this section should be run from the `slurm` directory. + +The `submit-evaluate-inpainting.py` script runs evaluation metrics on our data. Below is an example of running this +script on CPNet predictions: + +```bash +python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local +``` + +More generally, given a call of the following form: + +```bash +python submit-evaluate-inpainting.py -m local +``` + +the script will look for results in `/inpainting-results/devil/_/` and +compare them to the ground-truth data at `/datasets/devil/` and +`/datasets/devil/`. + +If you have predictions for your own model (e.g., at +`/inpainting-results/devil/flickr-all_fvi-fgd-h/mymodel`), your can use our evaluation script on them. + +### FID + +For some reason, the evaluation script usually fails when FID is computed alongside the other metrics; for this reason, +FID is excluded from the default configuration of `submit-evaluate-inpainting.py` and should be computed in a separate +run. The following example computes FID and adds it to the existing CPNet evaluation results: + +```bash +python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local -a ::include fid ::update +``` + +### OPTIONAL: Advanced features + +If you want to use the `.tar` files that were optionally generated (i.e., for the DEVIL datasets and evaluation +features), use the `--use_tar` flag: + +```bash +python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local --use_tar +``` + +Environment variables can be passed in as string pairs with the `-e` flag. The following example runs evaluation with a +custom scratch storage location (which is used to temporally store data during evaluation): + +```bash +python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -m local -e SCRATCH_ROOT /tmp/$USER +``` + +This script also supports Slurm job allocation, which can be done by removing the `-m local` flag: + +```bash +python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h +``` + +`sbatch` arguments can be passed in as string pairs with the `-s` flag. This illustrative example runs evaluation on a +custom partition: + +```bash +python submit-evaluate-inpainting.py cpnet flickr-all fvi-fgd-h -s partition my-partition +``` + +If necessary, `-s` and `-e` can be used at the same time. + +## Printing Evaluation Results + +The following script writes all quantitative results to `inpainting-results-quantitative-summary.tsv` as a tab-separated +text table: + +```bash +./scripts/print-quant-results-table.sh +``` diff --git a/docs/running-video-inpainting-models.md b/docs/running-video-inpainting-models.md new file mode 100644 index 0000000..8965f69 --- /dev/null +++ b/docs/running-video-inpainting-models.md @@ -0,0 +1,195 @@ +# Running Video Inpainting Models + +This document explains how to set up and run the sample video inpainting projects on our dataset, as well as how to +prepare your own algorithm for DEVIL evaluation. + +## Example: Running STTN + +This section explains how to run STTN1, a project which we have modified for compatibility with our +benchmark. + +### Initialize the STTN submodule + +``` +git submodule update --init video-inpainting-projects/STTN +cd video-inpainting-projects/STTN + +# Follows setup instructions in STTN project. Refer to their instructions if these become out-of-date. +conda create -p ./env -y +conda activate ./env +conda env update -f environment.yml +wget -O checkpoints/sttn.pth https://umich.box.com/shared/static/7yqut3gaxz7860ce1ckyyu5lgma99kwe.pth +``` + +### Run the video inpainting helper + +The `inpaint_videos.py` script is the helper script that runs video inpainting methods on our datasets. Below is an +example of running STTN on the `fvi-fgd-h` split: + +``` +python -m src.main.inpaint_videos \ + --frames_dataset_path=datasets/devil/flickr-all \ + --masks_dataset_path=datasets/devil/fvi-fgd-h \ + --inpainting_results_root=inpainting-results/devil/flickr-all_fvi-fgd-h/sttn \ + --run_path video-inpainting-projects/STTN/devil-run.sh \ + --image_size_divisor 432 240 +``` + +For more details on this script's arguments, run `python -m src.main.inpaint_videos -h`. + +### OPTIONAL: Meta helper scripts + +**NOTE:** The commands in this section should be run from the `slurm` directory. + +We have included meta helper scripts to run `inpaint_videos.py` in a less verbose manner. It also supports the Slurm job +scheduler. + +Below is an example of running the inpainting meta scripts with STTN on the `fvi-fgd-h` DEVIL split: + +``` +python submit-inpaint-videos.py sttn flickr-all fvi-fgd-h -m local +``` + +Environment variables can be passed in as string pairs with the `-e` flag. The following example runs inpainting with a +custom scratch storage location (i.e., where intermediate inpainting data is stored): + +``` +python submit-inpaint-videos.py sttn flickr-all fvi-fgd-h -m local -e TEMP_ROOT /tmp/$USER +``` + +To launch a job on a Slurm cluster, remove the `-m local` flag: + +``` +python submit-inpaint-videos.py sttn flickr-all fvi-fgd-h +``` + +`sbatch` arguments can be passed in as string pairs with the `-s` flag. This illustrative example runs inpainting on a +custom partition: + +``` +python submit-inpaint-videos.py sttn flickr-all fvi-fgd-h -s partition my-partition +``` + +If necessary, the `-s` and `-e` flags can be used at the same time. + +## Custom Video Inpainting Algorithms + +The DEVIL benchmark processes data in a very particular manner to ensure fair comparisons across different inpainting +methods. For this reason, we highly suggest using our helper scripts to run your method for DEVIL evaluation. This +section covers what you need to do to properly run your method with our helper scripts. + +### Execution script + +Prepare a Bash script that executes your method on a given input folder and saves results to a given output folder. You +can expect our benchmark to call the script in an equivalent manner to the following code: + +```bash +cd