Skip to content

Commit

Permalink
fix check for non utf-8 response
Browse files Browse the repository at this point in the history
  • Loading branch information
voslucas committed Sep 22, 2023
1 parent 94fef5c commit f3e2125
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
if len(sys.argv) > 1:
ISSUE = sys.argv[1]
else:
ISSUE = "668"
ISSUE = "670"

paperdata = {
"issue": ISSUE
Expand All @@ -37,7 +37,11 @@ def download_html(url):
else:
html = response.read()
#convert to string
html = html.decode("utf-8")
#Check if the encoding is utf-8, otherwise convert to utf-8
if response.info().get_content_charset() == 'utf-8':
html = html.decode("utf-8")
else:
html = html.decode("latin-1")
except urllib.error.HTTPError as e:
html = "Could not download this url."
return html
Expand Down

0 comments on commit f3e2125

Please sign in to comment.