Scrape Swiss real estate websites for new apartment listings and post them to Discord.
Every new listing is formatted into a neat Discord embed and immediately sent to a channel.
The data presented in the embed includes:
- Monthly rent in CHF
- Number of rooms
- Living space
- Location
- Images
- Distance (optional)
Supported websites:
You can edit the URLs that will be scraped periodically. Simply select your desired filters on the Immo websites, copy the URLs and set them as an ENV Variable (SCRAPE_URLS).
Built for Python 3.10 as aiohttp is not functional with 3.11+
Environment variables:
| ENV_VAR | Description | Required |
|---|---|---|
| DISCORD_WEBHOOK | New apartments will be sent to this Discord channel. | Yes |
| SCRAPE_URLS | a list of URLs to scrape (can contain multiple URLs per one hostname) | Yes |
| GOOGLE_MAPS_API_KEY | API key for accessing Distance Matrix API (optional) | No |
| GOOGLE_MAPS_DESTINATION | The destination address to use when computing distance to a new apartment listing | No |
| SCRAPING_INTERVAL | The amount of time (in seconds) to wait between individual scraping attempts (default 120s) | No |
| SENTRY_DSN | The DSN for Sentry | No |
How to run the script:
- Docker container
- Running the script directly after installing requirements
Build and run the image from the supplied Dockerfile or use sh build-and-run.sh.
Python version 3.10+ is required. Install requirements.txt on your virtual environment and run the script from the root of the repo.
$ pip install -r requirements.txt
...
$ python3 -m app.main -f etc/urls.json
2022-03-27 01:16:01.154 app.manager.www.immoscout24.ch INFO Initialized
2022-03-27 01:16:01.154 app.manager.www.homegate.ch INFO Initialized
This script optionally computes the distance and duration from the listing address to your destination address. For instance, from the listed apartment address to your work address.
In order to use this feature, don't forget to set the GOOGLE_MAPS_API_KEY environment variable.