Skip to content

Commit

Permalink
Merge pull request #66 from City-Bureau/enable-logs
Browse files Browse the repository at this point in the history
Tweaks to repo
  • Loading branch information
SimmonsRitchie authored Apr 30, 2024
2 parents d842785 + 23ad15d commit 7b4e6ee
Show file tree
Hide file tree
Showing 3 changed files with 26 additions and 16 deletions.
2 changes: 1 addition & 1 deletion .deploy.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
pipenv run scrapy list | xargs -I {} pipenv run scrapy crawl {} -s LOG_ENABLED=False &
pipenv run scrapy list | xargs -I {} pipenv run scrapy crawl {} -s LOG_ENABLED=True &

# Output to the screen every 9 minutes to prevent a travis timeout
# https://stackoverflow.com/a/40800348
Expand Down
39 changes: 24 additions & 15 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,28 @@
## Summary
## What's this PR do?
<!-- eg. This PR updates the scraper for Cleveland City Council because of changes to how they display their meeting schedule. -->

**Issue:** #ISSUE_NUMBER
## Why are we doing this?
<!-- eg. The website's layout was recently updated, causing our existing scraper to fail. This change ensures our scraper remains functional and continues to provide timely updates on council meetings. -->

Replace "ISSUE_NUMBER" with the number of your issue so that GitHub will link this pull request with the issue and make review easier.
## Steps to manually test
<!-- Text here is not always necessary but it is generally recommended in order to aid a reviewer.
eg.
1. Ensure the project is installed:
```
pipenv sync --dev
```
2. Activate the virtual env and enter the pipenv shell:
```
pipenv shell
```
3. Run the spider:
```
scrapy crawl <spider-name> -O test_output.csv
```
4. Monitor the output and ensure no errors are raised.
## Checklist
5. Inspect `test_output.csv` to ensure the data looks valid.
-->

All checks are run in [GitHub Actions](https://github.com/features/actions). You'll be able to see the results of the checks at the bottom of the pull request page after it's been opened, and you can click on any of the specific checks listed to see the output of each step and debug failures.

- [ ] Tests are implemented
- [ ] All tests are passing
- [ ] Style checks run (see [documentation](https://cityscrapers.org/docs/development/) for more details)
- [ ] Style checks are passing
- [ ] Code comments from template removed

## Questions

Include any questions you have about what you're working on.
## Are there any smells or added technical debt to note?
<!-- eg. The new scraping logic includes a more complex parsing routine, which might be less efficient. Future optimization or a more robust parsing strategy may be needed if the website's layout continues to evolve. -->
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -134,3 +134,4 @@ logs/*.log
travis/*.json

# output files: local gitignore added to city_scrapers/local_outputs/
*.csv

0 comments on commit 7b4e6ee

Please sign in to comment.