🏗️ Build spider: WAMPO – Transportation Policy Body

[SimmonsRitchie]

What's this PR do?

Adds a spider to scrape meetings from the website of Wichita Area Metropolitan Planning Organization - Transportation Policy Body. The new spider is called wicks_wampo_tpb.

[Note: Much like a separate PR, I accidentally merged this to main before review. Silly me 😒. Here's the original PR #13 and the revert #14 ]

Why are we doing this?

Requested by our site partners.

Steps to manually test

1. Ensure the project is installed:

pipenv sync --dev

2. Activate the virtual env and enter the pipenv shell:

pipenv shell

3. Run the spider:

scrapy crawl wicks_wampo_tpb -O test_output.csv

4. Monitor the output and ensure no errors are raised.

5. Inspect test_output.csv to ensure the data looks valid. Target website is here.

Are there any smells or added technical debt to note?

• Page is lacking some key details but has great links for attachments. Certain details are hardcoded.

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1206640968188925

[LienDang]

These 2 links should be unified into a single link:

{"href": "https://youtu.be/LsMI1EClvnI", "title": "Recording"}

[LienDang]

@SimmonsRitchie I noticed that the page renders complex html structure for a single link, eg. for Agenda link

And for Recording link, it breaks into 2 <a> elements. So I think you should rework on the _parse_links function.

[LienDang]

Please fix these 2 functions:

test_links()

_parse_links(item)

[SimmonsRitchie]

Please fix these 2 functions:

test_links()

_parse_links(item)

Ah, my bad, @LienDang. That's sloppy work on my part. I'm experimenting with a new automation to build tests quickly and I'll admit I gave the links retrieved a pretty cursory look. I've now fixed the parsing to handle the messiness of this agency's HTML. I think it now works pretty well.

[LienDang]

LGTM