A Responsive Web Visualization Dashboard using Flask framework to create a application that scrapes various websites for data related to the Mars and displays the information in a single HTML page.
In building this dashboard based on two parts. First part consist of scraping the data from using jupyter noebook and designing a dashboard using templated in Flask Framework.
Completed initial scraping using Jupyter Notebook, BeautifulSoup, Pandas, and Requests/Splinter.
- Created a Jupyter Notebook file called
mission_to_mars.ipynb
and used this to complete all of the scraping and analysis tasks.
- Scraped the NASA Mars News Site and collected the latest News Title and Paragraph Text. Assigned the text to variables that can be referenced later.
# Code:
news_title = "NASA's Next Mars Mission to Investigate Interior of Red Planet"
news_p = "Preparation of NASA's next spacecraft to Mars, InSight, has ramped up this summer, on course for launch next May from Vandenberg Air Force Base in central California -- the first interplanetary launch in history from America's West Coast."
-
Visited the url for JPL Featured Space Image here.
-
Used splinter to navigate the site and find the image url for the current Featured Mars Image and assigned the url string to a variable called
featured_image_url
. -
Made sure to find the image url to the full size
.jpg
image. -
Made sure to save a complete url string for this image.
# Example:
featured_image_url = 'https://www.jpl.nasa.gov/spaceimages/images/largesize/PIA16225_hires.jpg'
-
Visited the Mars Facts webpage here and used Pandas to scrape the table containing facts about the planet including Diameter, Mass, etc.
-
Used Pandas to convert the data to a HTML table string.
-
Visited the USGS Astrogeology site here to obtain high resolution images for each of Mar's hemispheres.
-
Used splinter to click each of the links to the hemispheres in order to find the image url to the full resolution image.
-
Saved both the image url string for the full resolution hemisphere image, and the Hemisphere title containing the hemisphere name. Used a Python dictionary to store the data using the keys
img_url
andtitle
. -
Appended the dictionary with the image url string and the hemisphere title to a list. This list will contain one dictionary for each hemisphere.
# Example:
hemisphere_image_urls = [
{"title": "Valles Marineris Hemisphere", "img_url": "..."},
{"title": "Cerberus Hemisphere", "img_url": "..."},
{"title": "Schiaparelli Hemisphere", "img_url": "..."},
{"title": "Syrtis Major Hemisphere", "img_url": "..."},
]
-
Used Splinter to navigate the sites when needed and BeautifulSoup to help find and parse out the necessary data.
-
Used Pymongo for CRUD applications for your database.
-
Used Bootstrap to structure my HTML template.
Used MongoDB with Flask templating to create a new HTML page that displays all of the information that was scraped from the URLs above.
-
Started by converting my Jupyter notebook into a Python script called
scrape_mars.py
with a function calledscrape
that will execute all of my scraping code from above and returned one Python dictionary containing all of the scraped data. -
Next, created a route called
/scrape
that will import yourscrape_mars.py
script and called myscrape
function.- Stored the return value in Mongo as a Python dictionary.
-
Created a root route
/
that will query your Mongo database and passed the mars data into an HTML template to display the data. -
Created a template HTML file called
index.html
that will take the mars data dictionary and displayed all of the data in the appropriate HTML elements.
- UWA Data Science