Image Crop Analysis

This is a repo for the code used for reproducing our Image Crop Analysis paper as shared on our blog post.

If you plan to use this code please cite our paper as follows:

@article{TwitterImageCrop2021,
  author = {Yee, Kyra and Tantipongpipat, Uthaipon and Mishra, Shubhanshu},
  title = {Image Cropping on Twitter: Fairness Metrics, Their Limitations, and the Importance of Representation, Design, and Agency},
  year = {2021},
  issue_date = {October 2021},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  volume = {5},
  number = {CSCW2},
  url = {https://doi.org/10.1145/3479594},
  doi = {10.1145/3479594},
  journal = {Proceedings of the ACM on Human-Computer Interaction},
  month = oct,
  articleno = {450},
  numpages = {24},
  keywords = {image cropping, ethical HCI, fairness in machine learning, demographic parity, representational harm}
}
@article{TwitterImageCrop2021ArXiv,
       author = {{Yee}, Kyra and {Tantipongpipat}, Uthaipon and {Mishra}, Shubhanshu},
        title = "{Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency}",
      journal = {arXiv e-prints},
     keywords = {Computer Science - Computers and Society, Computer Science - Computer Vision and Pattern Recognition, Computer Science - Human-Computer Interaction, Computer Science - Machine Learning},
         year = 2021,
        month = may,
          eid = {arXiv:2105.08667},
        pages = {arXiv:2105.08667},
archivePrefix = {arXiv},
       eprint = {2105.08667},
 primaryClass = {cs.CY},
}

Instructions

• Install Miniconda or Anaconda and then follow these steps:
  • create a conda environment using conda env create -f environment.yml
  • activate the environment using conda activate image-crop-analysis
• Put a dummy jpeg image at data/dummy.jpeg
• Put any additional images with *.jpeg extension in DATA_DIR, which is ./data
• If you just want to investigate how the model predicts the saliency map then you can use the notebook notebooks/Image Annotation Dash.ipynb
• To reproduce the analysis first prepare the data using notebooks/Data Preparation.ipynb and then run notebooks/Demographic Bias Analysis.ipynb
• To reproduce the plots first run notebooks/Demographic Bias Plots.ipynb
• If you want to explore how the library behind the dashboard works see notebooks/Image Crop Analysis.ipynb
• If you have the dataset prepared from the above steps then you can create the gender gaze dataset by running notebooks/Gender Gaze Analysis.ipynb

Docker Run

• Install docker
• Run the following commands in this root directory of this project:

docker build -t "image_crop" -f docker/Dockerfile .
docker run -p 9000:9000 -p 8900:8900 -it image_crop

• Open the jupyter lab URL shown in terminal.

Run on Google Colab

• Open a google colab notebook
• Run the following code in the cell where HOME_DIR variable is set:

try:
    import google.colab
    ! pip install pandas scikit-learn scikit-image statsmodels requests dash
    ! [[ -d image-crop-analysis ]] || git clone https://github.com/twitter-research/image-crop-analysis.git
    HOME_DIR = Path("./image_crop_analysis").expanduser()
    IN_COLAB = True
except:
    IN_COLAB = False

• Try the notebooks/Image Crop Analysis.ipynb notebook for example.

Security Issues?

Please report sensitive security issues via Twitter's bug-bounty program (https://hackerone.com/twitter) rather than GitHub.