This project automatically updates a MySQL database of villager data from the video game Animal Crossing: New Horizons. Villager popularity data is scraped using Beautiful Soup on Animal Crossing Portal's popularity tier list. Villager attributes are obtained from a Kaggle dataset. Both sources are joined and appended to MySQL table automatically every two weeks using a CRON job via WSL on a Windows computer.
- ETL
- Web Scraping
- Data Cleansing
- Python
- Linux (WSL)
- Selenium
- bs4
- Pandas
- Kaggle
- sqlalchemy
- mysql-connector-python
On the command line of your operating system, navigate to the repository directory (ideally using a Python virtual environment).
Run the following code on the command line to install requirements:
pip install -r requirements.txt
Run the following code on the command line to run this project:
Python ac_pop.py
ACNH_tier_rank_queries.sql- Example queries for villager analysischromedriver.exe- Chrome Driver used to Operate JavaScript on website before scrapingconfig.ini- Configurations for MySQL connectionrequirements.txt- Python package requirementsvillagers.csv- Kaggle dataset of Animal Crossing villager data