-
-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🏗️ Build mixin + spiders: bisnd_mc #8
Conversation
city_scrapers/mixins/mc.py
Outdated
|
||
def start_requests(self): | ||
""" | ||
sdfsdfsdf |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dummy text?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep. Now fixed!
city_scrapers/mixins/mc.py
Outdated
# Calculate dates for one month prior and one year ahead | ||
today = datetime.today() | ||
one_month_prior = today - relativedelta(months=1) | ||
one_year_ahead = today + relativedelta(months=6) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Naming issue one_year_ahead
vs. half_year_ahead
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep. Now fixed!
city_scrapers/mixins/mc.py
Outdated
links.append( | ||
{ | ||
"title": file["name"], | ||
"href": f"https://mandannd.api.civicclerk.com/v1/Meetings/GetMeetingFileStream(fileId={file['fileId']},plainText=false)", # noqa |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can replace https://mandannd.api.civicclerk.com
with {self.base_url}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep. That's def tidier. Thanks, @LienDang.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
What's this PR do?
Adds a mixin and spiders to scrape multiple commission/boards from the website of the City of Mandan in Mandan, ND. The mixin that handles the core scraping logic is called
bisnd_mc
.Why are we doing this?
Requested by our site partners.
Steps to manually test
Monitor the output and ensure no errors are raised.
Inspect the CSVs to ensure the data looks valid. You can view the value in the "source" column of the rows to ensure the data matches.
Are there any smells or added technical debt to note?