Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🏗️ Build spider: Morton County Commission #10

Merged
merged 3 commits into from
Mar 19, 2024
Merged

Conversation

SimmonsRitchie
Copy link
Contributor

@SimmonsRitchie SimmonsRitchie commented Mar 14, 2024

What's this PR do?

Adds a spider to scrape meetings from the website of Morton County Commission. The new spider is called bisnd_mcc.

Why are we doing this?

Requested by our site partners.

Steps to manually test

  1. Ensure the project is installed:
pipenv sync --dev
  1. Activate the virtual env and enter the pipenv shell:
pipenv shell
  1. Run the spider:
scrapy crawl bisnd_mcc -O test_output.csv
  1. Monitor the output and ensure no errors are raised.

  2. Inspect test_output.csv to ensure the data looks valid. Target website is here.

Are there any smells or added technical debt to note?

  • This agency provides great links to agendas and minutes but otherwise doesn't provide great meeting info (eg. meeting title or meeting start time). Certain key details are hardcoded.

@SimmonsRitchie SimmonsRitchie requested a review from a team March 14, 2024 15:18
@SimmonsRitchie SimmonsRitchie marked this pull request as ready for review March 14, 2024 15:19
Copy link

@LienDang LienDang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Fixes conflict with Pipfile.lock by rebuilding lock file.
@SimmonsRitchie SimmonsRitchie merged commit bc52020 into main Mar 19, 2024
2 checks passed
@SimmonsRitchie SimmonsRitchie deleted the morton--simple branch March 19, 2024 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants