Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🏗️ Build mixin + spiders: bcc #5

Merged
merged 2 commits into from
Mar 4, 2024
Merged

🏗️ Build mixin + spiders: bcc #5

merged 2 commits into from
Mar 4, 2024

Conversation

SimmonsRitchie
Copy link
Contributor

@SimmonsRitchie SimmonsRitchie commented Mar 4, 2024

What's this PR do?

Adds a mixin and spiders to scrape multiple meetings from the website of Bismarck City Commission The mixin that handles the core scraping logic is called bcc.

[Note: This PR was prematurely merged and was promptly reverted. Please refer to #7 for the final, reviewed version of these changes)

Why are we doing this?

Requested by our site partners.

Steps to manually test

  1. Ensure the project is installed:
pipenv sync --dev
  1. Activate the virtual env and enter the pipenv shell:
pipenv shell
  1. Run the spiders:
scrapy list | grep 'bcc' | xargs -I {} scrapy crawl {} -O {}.csv
  1. Monitor the output and ensure no errors are raised.

  2. Inspect the CSVs to ensure the data looks valid.

Are there any smells or added technical debt to note?

@SimmonsRitchie SimmonsRitchie marked this pull request as ready for review March 4, 2024 22:23
@SimmonsRitchie SimmonsRitchie merged commit a17f70c into main Mar 4, 2024
2 checks passed
@SimmonsRitchie SimmonsRitchie deleted the bisnd_bcc branch March 4, 2024 22:24
@SimmonsRitchie SimmonsRitchie restored the bisnd_bcc branch March 4, 2024 22:25
SimmonsRitchie added a commit that referenced this pull request Mar 4, 2024
This reverts commit a17f70c, reversing
changes made to 02a8af1.
SimmonsRitchie added a commit that referenced this pull request Mar 4, 2024
Revert "Merge pull request #5 from City-Bureau/bisnd_bcc"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant