New Source: Blacktown City Council (#1382)

* Added Blacktown City Council

* raise error instad of returning [] + Reformatting

---------

Co-authored-by: 5ila5 <5ila5@users.noreply.github.com>

README.md

@@ -29,6 +29,7 @@ Waste collection schedules in the following formats and countries are supported.
 - [Australian Capital Territory (ACT)](/doc/source/act_gov_au.md) / cityservices.act.gov.au/recycling-and-waste
 - [Banyule City Council](/doc/source/banyule_vic_gov_au.md) / banyule.vic.gov.au
 - [Belmont City Council](/doc/source/belmont_wa_gov_au.md) / belmont.wa.gov.au
+- [Blacktown City Council (NSW)](/doc/source/blacktown_nsw_gov_au.md) / blacktown.nsw.gov.au
 - [Brisbane City Council](/doc/source/brisbane_qld_gov_au.md) / brisbane.qld.gov.au
 - [Campbelltown City Council (NSW)](/doc/source/campbelltown_nsw_gov_au.md) / campbelltown.nsw.gov.au
 - [Cardinia Shire Council](/doc/source/cardinia_vic_gov_au.md) / cardinia.vic.gov.au

custom_components/waste_collection_schedule/waste_collection_schedule/source/blacktown_nsw_gov_au.py

@@ -0,0 +1,128 @@
+import datetime
+import json
+
+import requests
+from bs4 import BeautifulSoup
+from requests.utils import requote_uri
+from waste_collection_schedule import Collection  # type: ignore[attr-defined]
+
+TITLE = "Blacktown City Council (NSW)"
+DESCRIPTION = "Source for Blacktown City Council rubbish collection."
+URL = "https://www.blacktown.nsw.gov.au/"
+TEST_CASES = {
+    "Plumpton Marketplace": {
+        "post_code": "2761",
+        "suburb": "Plumpton",
+        "street_name": "Jersey Rd",
+        "street_number": "260",
+    },
+    "Rooty Hill Tennis & Squash Centre": {
+        "post_code": "2766",
+        "suburb": "Rooty Hill",
+        "street_name": "Learmonth St",
+        "street_number": "13-15",
+    },
+    "Workers Blacktown": {
+        "post_code": "2148",
+        "suburb": "Blacktown",
+        "street_name": "Campbell St",
+        "street_number": "55",
+    },
+    "Hythe St": {
+        "post_code": "2770",
+        "suburb": "Mount Druitt",
+        "street_name": "Hythe St",
+        "street_number": "9-11",
+    },
+}
+
+API_URLS = {
+    "address_search": "https://www.blacktown.nsw.gov.au/api/v1/myarea/search?keywords={}",
+    "collection": "https://www.blacktown.nsw.gov.au/ocapi/Public/myarea/wasteservices?geolocationid={}&ocsvclang=en-AU",
+}
+
+HEADERS = {"user-agent": "Mozilla/5.0"}
+
+ICON_MAP = {
+    "General Waste": "trash-can",
+    "Recycling": "mdi:recycle",
+}
+
+
+class Source:
+    def __init__(
+        self, post_code: str, suburb: str, street_name: str, street_number: str
+    ):
+        self.post_code = post_code
+        self.suburb = suburb
+        self.street_name = street_name
+        self.street_number = street_number
+
+    def fetch(self):
+        locationId = 0
+
+        address = "{} {} {} NSW {}".format(
+            self.street_number, self.street_name, self.suburb, self.post_code
+        )
+
+        q = requote_uri(str(API_URLS["address_search"]).format(address))
+
+        # Retrieve suburbs
+        r = requests.get(q, headers=HEADERS)
+
+        data = json.loads(r.text)
+
+        # Find the ID for our suburb
+        for item in data["Items"]:
+            locationId = item["Id"]
+            break
+
+        if locationId == 0:
+            raise ValueError(
+                f"Unable to find location ID for {address}, maybe you misspelled your address?"
+            )
+
+        # Retrieve the upcoming collections for our property
+        q = requote_uri(str(API_URLS["collection"]).format(locationId))
+
+        r = requests.get(q, headers=HEADERS)
+
+        data = json.loads(r.text)
+
+        responseContent = data["responseContent"]
+
+        soup = BeautifulSoup(responseContent, "html.parser")
+        services = soup.find_all("div", attrs={"class": "waste-services-result"})
+
+        entries = []
+
+        for item in services:
+            # test if <div> contains a valid date. If not, is is not a collection item.
+            date_text = item.find("div", attrs={"class": "next-service"})
+
+            # The date format currently used on https://www.blacktown.nsw.gov.au/Services/Waste-services-and-collection/Waste-collection-days
+            date_format = "%a %d/%m/%Y"
+
+            try:
+                # Strip carriage returns and newlines out of the HTML content
+                cleaned_date_text = (
+                    date_text.text.replace("\r", "").replace("\n", "").strip()
+                )
+
+                # Parse the date
+                date = datetime.datetime.strptime(cleaned_date_text, date_format).date()
+
+            except ValueError:
+                continue
+
+            waste_type = item.find("h3").text.strip()
+
+            entries.append(
+                Collection(
+                    date=date,
+                    t=waste_type,
+                    icon=ICON_MAP.get(waste_type, "mdi:trash-can"),
+                )
+            )
+
+        return entries

doc/source/blacktown_nsw_gov_au.md

@@ -0,0 +1,64 @@
+# Blacktown City Council (NSW)
+
+Support for schedules provided by [Blacktown City Council Waste and Recycling](https://www.blacktown.nsw.gov.au/Services/Waste-services-and-collection/Waste-collection-days).
+
+## Configuration via configuration.yaml
+
+```yaml
+waste_collection_schedule:
+  sources:
+    - name: blacktown_nsw_gov_au
+      args:
+        post_code: POST_CODE
+        suburb: SUBURB
+        street_name: STREET_NAME
+        street_number: STREET_NUMBER
+```
+
+### Configuration Variables
+
+**post_code**  
+*(string) (required)*
+
+**suburb**  
+*(string) (required)*
+
+**street_name**  
+*(string) (required)*
+
+**street_number**  
+*(string) (required)*
+
+## Example
+
+```yaml
+waste_collection_schedule:
+  sources:
+    - name: blacktown_nsw_gov_au
+      args:
+        post_code: 2770
+        suburb: Emerton
+        street_name: Helena Ave
+        street_number: 80
+```
+
+## How to get the source arguments
+
+Visit the [Blacktown City Council Waste Collection Days](https://www.blacktown.nsw.gov.au/Services/Waste-services-and-collection/Waste-collection-days) page, follow the quick link to *Check my collection day*, and search for your address. The street address arguments used to configure hacs_waste_collection_schedule should exactly match the street address shown in the autocomplete result.
+
+## How this integration uses Blacktown Council's APIs
+
+Two API calls are currently needed to retrieve waste collection schedule results from Blacktown Council:
+1. The address search API at https://www.blacktown.nsw.gov.au/api/v1/myarea/search
+2. The waste services API at https://www.blacktown.nsw.gov.au/ocapi/Public/myarea/wasteservices
+
+This integration does the following:
+1. Calls the address search API to retrieve the "location ID" for the given location. Eg. https://www.blacktown.nsw.gov.au/api/v1/myarea/search?keywords=80+Helena+Ave+Emerton+NSW+2770
+2. Retrieves waste/collection info from the waste services API using the "location ID" retrieved in step #1. Eg. https://www.blacktown.nsw.gov.au/ocapi/Public/myarea/wasteservices?geolocationid=6177cbfa-6f35-4fbf-9208-63d4dde7f048&ocsvclang=en-AU
+3. Parses the HTML returned by the waste services API in step #2 to extract the data
+
+
+# Similarities with other sources
+
+For future work it's good to note that Blacktown City Council uses the same APIs as Campbelltown City Council, to the point I was able to copy and paste their files with minimal modification.
+ 

info.md

@@ -16,7 +16,7 @@ Waste collection schedules from service provider web sites are updated daily, de
 |--|--|
 | Generic | ICS / iCal files |
 | Static | User-defined dates or repeating date patterns |<!--Begin of country section-->
-| Australia | Armadale (Western Australia), Australian Capital Territory (ACT), Banyule City Council, Belmont City Council, Brisbane City Council, Campbelltown City Council (NSW), Cardinia Shire Council, City of Canada Bay Council, City of Greater Geelong, City of Kingston, City of Onkaparinga Council, Cumberland Council (NSW), Gold Coast City Council, Hume City Council, Inner West Council (NSW), Ipswich City Council, Ku-ring-gai Council, Lake Macquarie City Council, Logan City Council, Macedon Ranges Shire Council, Mansfield Shire Council, Maribyrnong Council, Maroondah City Council, Melton City Council, Moreton Bay, Nillumbik Shire Council, North Adelaide Waste Management Authority, Port Adelaide Enfield, South Australia, RecycleSmart, Shellharbour City Council, Stonnington City Council, The Hills Shire Council, Sydney, Unley City Council (SA), Whittlesea City Council, Wollongong City Council, Wyndham City Council, Melbourne |
+| Australia | Armadale (Western Australia), Australian Capital Territory (ACT), Banyule City Council, Belmont City Council,Blacktown City Council (NSW), Brisbane City Council, Campbelltown City Council (NSW), Cardinia Shire Council, City of Canada Bay Council, City of Greater Geelong, City of Kingston, City of Onkaparinga Council, Cumberland Council (NSW), Gold Coast City Council, Hume City Council, Inner West Council (NSW), Ipswich City Council, Ku-ring-gai Council, Lake Macquarie City Council, Logan City Council, Macedon Ranges Shire Council, Mansfield Shire Council, Maribyrnong Council, Maroondah City Council, Melton City Council, Moreton Bay, Nillumbik Shire Council, North Adelaide Waste Management Authority, Port Adelaide Enfield, South Australia, RecycleSmart, Shellharbour City Council, Stonnington City Council, The Hills Shire Council, Sydney, Unley City Council (SA), Whittlesea City Council, Wollongong City Council, Wyndham City Council, Melbourne |
 | Austria | Abfallverband Hollabrunn, Abfallverband Korneuburg, Abfallverband Schwechat, Abfallwirtschaft Stadt Krems, Abfallwirtschaft Stadt St Pölten, Altenmarkt an der Triesting, Andau, Apetlon, App CITIES, AWV Neunkirchen, AWV Wr. Neustadt, Bad Blumau, Bad Gleichenberg, Bad Loipersdorf, Bad Radkersburg, Bad Tatzmannsdorf, Bernstein, Bildein, Breitenbrunn am Neusiedler See, Breitenstein, Bromberg, Bruckneudorf, Buch - St. Magdalena, Burgau, Burgauberg-Neudauberg, Burgenländischer Müllverband, Dechantskirchen, Deutsch Goritz, Deutsch Jahrndorf, Deutsch Kaltenbrunn, Deutschkreutz, Die NÖ Umweltverbände, Dobl-Zwaring, Drasenhofen, Draßmarkt, Eberau, Eberndorf, Ebersdorf, Eberstein, Edelsbach bei Feldbach, Eggersdorf bei Graz, Eisenstadt, Fehring, Feistritz ob Bleiburg, Feldbach, Feldkirchen in Kärnten, Ferndorf, Frankenau-Unterpullendorf, Frauenkirchen, Freistadt, Fresach, Friedberg, Frohnleiten, Fürstenfeld, Gabersdorf, GABL, Gattendorf, GAUL Laa an der Thaya, GAUM Mistelbach, GDA Amstetten, Gemeindeverband Horn, Gols, Grafendorf bei Hartberg, Grafenschachen, Grafenstein, Gratkorn, Gratwein-Straßengel, Großwarasdorf, Großwilfersdorf, Gutenberg, GV Gmünd, GV Krems, GV Zwettl, GVA Baden, GVA Baden, GVA Lilienfeld, GVA Mödling, GVA Tulln, GVA Waidhofen/Thaya, GVU Bezirk Gänserndorf, GVU Melk, GVU Scheibbs, GVU Scheibbs, GVU St. Pölten, Güssing, Hagenberg im Mühlkreis, Hannersdorf, Hartberg, Heiligenkreuz, Heiligenkreuz am Waasen, Heimschuh, Hofstätten an der Raab, Horitschon, Horn, Hornstein, Hüttenberg, Ilz, infeo, Innsbrucker Kommunalbetriebe, Inzenhof, Jabing, Jagerberg, Kaindorf, Kaisersdorf, Kalsdorf bei Graz, Kapfenstein, Kemeten, Kirchberg an der Raab, Kittsee, Kleinmürbisch, Klingenbach, Klosterneuburg, Klöch, Kohfidisch, Korneuburg, Krensdorf, Laa an der Thaya, Lackenbach, Lackendorf, Langau, Langenrohr, Leithaprodersdorf, Leutschach an der Weinstraße, Lieboch, Litzelsdorf, Lockenhaus Marktgemeinde, Loipersbach im Burgenland, Mariasdorf, Markt Hartmannsdorf, Markt Neuhodis, Marktgemeinde Edlitz, Marz, Mattersburg, Meiseldorf, Melk, Mettersdorf am Saßbach, Miesenbach, Mischendorf, Mistelbach, Mitterdorf an der Raab, Mureck, Mönchhof, Mörbisch am See, Neudorf bei Parndorf, Neudörfl, Neufeld an der Leitha, Neusiedl am See, Neustift bei Güssing, Nickelsdorf, Oberpullendorf, Oberschützen, Oberwart, Oslip, Ottendorf an der Rittschein, Paldau, Pama, Pamhagen, Parndorf, Payerbach, Peggau, Pernegg an der Mur, Pernegg im Waldviertel, Pfarrwerfen, Pilgersdorf, Pinggau, Pinkafeld, Podersdorf am See, Poggersdorf, Potzneusiedl, Poysdorf, Pöchlarn, Raach am Hochgebirge, Radmer, Ragnitz, Raiding, Reichenau, Rohr bei Hartberg, Rudersdorf, Rust, Sankt Georgen an der Stiefing, Sankt Gilgen, Sankt Oswald bei Plankenwarth, Schrattenberg, Schwadorf, Schäffern, Schützen am Gebirge, Seiersberg-Pirka, Siegendorf, Sigleß, Sigmundsherberg, Sinabelkirchen, St. Andrä, St. Andrä am Zicksee, St. Anna am Aigen, St. Egyden am Steinfeld, St. Johann in der Haide, St. Lorenzen am Wechsel, St. Margarethen an der Raab, St. Margarethen im Burgenland, St. Peter - Freienstein, St. Peter am Ottersbach, St. Ruprecht an der Raab, St. Veit in der Südsteiermark, Stadt Salzburg, Stadtservice Korneuburg, Stegersbach, Steinbrunn, Steuerberg, Stinatz, Stiwoll, Stockerau, Straß in Steiermark, Söchau, Tadten, Tattendorf, Thal, Tieschen, Tobaj, Tulln an der Donau, Umweltprofis, Unterfrauenhaid, Unterkohlstätten, Unterlamm, Unterwart, Vasoldsberg, Vordernberg, Völkermarkt, Walpersbach, Weiden am See, Weitersfeld, Weiz, Weppersdorf, Werfenweng, Wies, Wiesen, Wiesfleck, Wiesmath, Wimpassing an der Leitha, Winden am See, Wolfau, Wolfsberg, Wolkersdorf im Weinviertel, WSZ Moosburg, Wulkaprodersdorf, Wörterberg, Zagersdorf, Zelking-Matzleinsdorf, Zillingtal, Zurndorf, Übelbach |
 | Belgium | Hygea, Limburg.net, Recycle! |
 | Canada | Aurora (ON), Calgary (AB), Calgary, AB, City of Edmonton, AB, City of Greater Sudbury, ON, City of Peterborough, ON, London (ON), Ottawa, Canada, RM of Morris, MB, Strathcona County, ON, Toronto (ON), Waste Wise APPS |