Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: London Borough of Lewisham 403 error #3079

Closed
6 of 7 tasks
theawesomestrob opened this issue Nov 23, 2024 · 2 comments · Fixed by #3328
Closed
6 of 7 tasks

[Bug]: London Borough of Lewisham 403 error #3079

theawesomestrob opened this issue Nov 23, 2024 · 2 comments · Fixed by #3328
Assignees
@5ila5

Comments

@theawesomestrob
Copy link

I Have A Problem With:

A specific source

What's Your Problem

Trying to configure the source with UPRN, I get this error:

The source returned an invalid response: "403 Client Error: Forbidden for url: https://lewisham.gov.uk/myservices/recycling-and-rubbish/your-bins/collection". Please check the provided arguments and try again.

Pasting that URL into a browser works fine.
Trying postcode+number instead of UPRN, I get the same error with a different URL:

The source returned an invalid response: "403 Client Error: Forbidden for url: https://lewisham.gov.uk/api/AddressFinder?postcodeOrStreet=SE231LB". Please check the provided arguments and try again.

Pasting that into a browser gives an error saying "The page you are looking for cannot be displayed because an invalid method (HTTP verb) is being used."

Trying either URL via curl I get a 403, eg:

$ curl -i 'https://lewisham.gov.uk/myservices/recycling-and-rubbish/your-bins/collection'
HTTP/1.1 403 Forbidden
Server: Microsoft-Azure-Application-Gateway/v2
Date: Sat, 23 Nov 2024 18:16:00 GMT
Content-Type: text/html
Content-Length: 179
Connection: keep-alive

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>Microsoft-Azure-Application-Gateway/v2</center>
</body>
</html>

Looks like they're doing stupid user-agent filtering, as passing a different user-agent header returns a 200 response.

Seems any user-agent string with 'curl' or 'python' (or 'ruby', even) in it results in a 403:

$ printf '%-32s %s\n' user-agent response; for agent in web-enabled-hair-{straighteners,curlers} 'some script written in '{ruby,python,bash} ; do printf '%-32s ' "${agent}"; curl -s -o /dev/null -w '%{http_code}\n' -A "$agent" 'https://lewisham.gov.uk/myservices/recycling-and-rubbish/your-bins/collection' ; d
one
user-agent                       response
web-enabled-hair-straighteners   200
web-enabled-hair-curlers         403
some script written in ruby      403
some script written in python    403
some script written in bash      200

Source (if relevant)

lewisham_gov_uk

Logs

no relevant logs

Relevant Configuration

No response

Checklist Source Error

  • Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
  • Checked that the website of your service provider is still working
  • Tested my attributes on the service provider website (if possible)
  • I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

Checklist Sensor Error

  • Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

Required

  • I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
  • I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.
@5ila5
Copy link
Collaborator

5ila5 commented Dec 27, 2024

This seems not to be the only issue, If you pass another user agent the first requests pass as expected but requesting https://lewisham.gov.uk/formbuilder?sc_site=website&fxb.FormItemId=cc81fd29-2b6b-4087-a5a9-e62d90ae8d74&fxb.HtmlPrefix=fxb.aa57bdc7-9d26-41f3-aee1-39a061d62d3c fails with 403 Forbidden I did some tinkering but could not figure out what's wrong. I think I did everything exactly like the browser, but the script still fails (I probably did not see some small thing somewhere in the request). I will take a second look at this tomorrow.

@5ila5 5ila5 mentioned this issue Dec 29, 2024
Merged
@5ila5
Copy link
Collaborator

5ila5 commented Dec 29, 2024

fixed in #3328

This will be part of the next release or is available now as master (git) version

install master/git: Developer tools -> actions -> select update: Install update -> click Coose entity and select WCS -> select version and type master in the input field -> run the action -> restart HA

Older versions of HACS:

install master with HACS version < 2.0: (3-dot menu -> redownload -> change version to master -> fully restart HA (this will increase update cycle by a lot)

@5ila5 5ila5 self-assigned this Dec 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@5ila5 5ila5

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix lewisham_gov_uk 5ila5/hacs_waste_collection_schedule
2 participants
@theawesomestrob @5ila5