wake_up_songs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Nasa Wake up Music Project\n",
    "This notebook does some cleaning and adds more data to the Nasa Wakeup Songs archive produced by Nasa's history department, and digitalised by Ross Spencer. \n",
    "Notebook by Matthew Allinson. Contact me via [mattallinson.com](http://mattallinson.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Getting genre data from Discogs\n",
    "\n",
    "The first part of this notebook gets data from [discogs](https://www.discogs.com) to supplement the song information. I have included Genre and release date in my extended database, and have included a link to the discogs URI and ID too so it will be easier to get more data in the future. This method wasn't perfect though, and required some manual polishing afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Get initial data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Initialization\n",
    "\n",
    "df = pd.read_csv('./v2/nasawakeupcalls_no-genre.csv')\n",
    "\n",
    "for col in ['year','genre', 'style', 'master_url', 'uri']:\n",
    "    #adds the columns we're going to scrape from discogs\n",
    "    if col not in df:\n",
    "        df.insert(len(df.columns),col,None)\n",
    "    \n",
    "DISCOGS_URL = 'https://www.discogs.com'\n",
    "DISCOGS_API = 'https://api.discogs.com/database/search?'\n",
    "\n",
    "with open('token.txt') as token_file: # Get an API token from discogs and save as token.txt\n",
    "    DISCOGS_TOKEN = token_file.read()\n",
    "\n",
    "def save_csv(filename):    \n",
    "    with open(filename,'w') as outfile:\n",
    "        df.to_csv(outfile, index=False, sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1.2 Clean up data\n",
    "\n",
    "After doing visual inspection on the csv file, there were some errors in the converter from PDF that left some song names with trailing punctuation or the word \"Performed\" so this sorts that, runs once.\n",
    "\n",
    "Cleaning by hand was also done in Microsoft Excel to consolidate various duplicate spellings/name of repeat songs like Anchors Aweigh (Anchors Away, the Naval Hymn, etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in df.itertuples(): \n",
    "    song = i.Song\n",
    "    if type(song) == float: #to do, work out why some of these are NaN\n",
    "        pass\n",
    "    elif 'performed' in song:\n",
    "        df.at[i,'Song'] = song.strip('performed')\n",
    "    else:\n",
    "        df.at[i,'Song'] = song.strip(',- ').title()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Getting genre data from Discogs\n",
    "\n",
    "We're going to search by track name (if possible) and use this to find genres for the track."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def discogs_api(song=None, artist=None):\n",
    "    '''retrieves the top search result from discogs for a given song &\n",
    "    artist combination. \n",
    "    '''\n",
    "    header = {'Authorization':'Discogs token='+DISCOGS_TOKEN}\n",
    "    payload = {'per_page':1,'page':1}\n",
    "    \n",
    "    if artist != None:\n",
    "        payload.update({'artist':artist})\n",
    "    if song != None:\n",
    "        payload.update({'release_title':song})\n",
    "\n",
    "    req = requests.get(DISCOGS_API,headers=header,params=payload)\n",
    "    \n",
    "    return req.json()['results']\n",
    "\n",
    "def get_song_info(index):\n",
    "    '''Searches the discogs database using the data for the\n",
    "    given index in the data frame\n",
    "    '''\n",
    "    row = df.iloc[index]\n",
    "    song = row['Song']\n",
    "    artist = row['Artist']\n",
    "    \n",
    "    # Doesn't bother searching if song information is missing or bad quality\n",
    "    if type(song) == float:\n",
    "        return\n",
    "    for u in [\"unknown\",\"unidentified\",\"untitled\",\"medley\"]:   \n",
    "        if u in song.lower():\n",
    "            return\n",
    "    \n",
    "    # Stops \"Unidentified\" being handed to the search API\n",
    "    if artist == \"Unidentified\":\n",
    "        artist = None\n",
    "    \n",
    "    data = discogs_api(song, artist) #does the search\n",
    "    \n",
    "    if len(data) == 0: #no results\n",
    "        return\n",
    "    \n",
    "    else:\n",
    "        data = data[0]\n",
    "        \n",
    "    for key in ['genre','style','year','master_url']:\n",
    "        if key not in data:\n",
    "            continue # if the data is missing, leave as None\n",
    "        elif type(data[key]) != list:\n",
    "            df.at[index,key] = data[key] #if it's not a list, take the value\n",
    "        elif len(data[key]) == 0: \n",
    "            continue # if it's an empty list, leave as None\n",
    "        else:\n",
    "            df.at[index,key] = data[key][0] #if it is a list, take the 1st value\n",
    "    \n",
    "    if 'uri' in data: #special case for formatting URI\n",
    "        df.at[index,'uri'] = DISCOGS_URL + data['uri']\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Do the search!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 % complete\n"
     ]
    }
   ],
   "source": [
    "start = 0 #for restarting at a later point if discogs API throws a fit\n",
    "\n",
    "for i in range(start,len(df)):\n",
    "    get_song_info(i)\n",
    "    prog =(i/len(df)-1)*100\n",
    "    save_csv('nasawakeupcalls.csv') #this could maybe be more efficient?\n",
    "    \n",
    "    if i%10 == 0: #keep us updated\n",
    "        print(int(prog),'% complete')\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Sorting out the JSON\n",
    "\n",
    "Takes all the genre data and adds it back into the JSON format. The final version of the csv had _even more_ hand work done on it. For Shuttle Missions if there was no genre data for a song, I found it manually using a discogs search of the artist. I was unable to make a simple machine way of doing this given the time constraints of the job I was doing it for, perhaps this can be revisted in the future.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "from datetime import datetime "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Get initial data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#gets the metadata and structure from the original json\n",
    "#loads up the data from new, cleaned-up and expanded csv fle\n",
    "\n",
    "with open('./v2/nasawakeupcalls_no-genre.json') as f:\n",
    "    song_json = json.load(f)\n",
    "\n",
    "import_df = pd.read_csv('nasawakeupcalls.csv')\n",
    "songs_df = import_df.where(pd.notnull(import_df), None) #Removes NaNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Create a structured dictionary\n",
    "Uses the form ```{'Mission name':data}``` where ```data``` matches the format used in the original json. I fear that this makes sense to me now but given that it's a hideous mess of nested loops and if statements it's going to be nonsense if I ever have to return to it. I hope my commenting is good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "programs = [i['Title'] for i in song_json['Programs']]\n",
    "program_data = {}\n",
    "\n",
    "#but-it-runs-meme.jpg\n",
    "for p in programs:\n",
    "    mission_data = [] # a blank list that will contain the mission data\n",
    "    missions = list(songs_df.loc[songs_df['Program'] == p].Mission.unique()) #the labels of all the missions\n",
    "    p_df = songs_df.loc[songs_df['Program'] == p].drop(columns=['Program']) # a data frame of all the info from that program\n",
    "    for m in missions: \n",
    "        m_df = p_df.loc[p_df['Mission'] == m].drop(columns=['Mission']) # a data frame of all the info from that mission\n",
    "        mission_dictionary = m_df.to_dict('records') # Makes a dictionary of that data\n",
    "        wakeup_data = [] #a blank list that will contain the data for each day\n",
    "        for wakeup_song in mission_dictionary:\n",
    "            key = wakeup_song.pop('Dates')\n",
    "            if len(wakeup_data) == 0:\n",
    "                wakeup_data.append({key:[wakeup_song]}) #a dictionary {date:[{song info}]}\n",
    "            elif key not in wakeup_data[-1]: # one song on that day\n",
    "                wakeup_data.append({key:[wakeup_song]}) #a dictionary {date:[{song info}]}\n",
    "            else: # multiple songs on that day\n",
    "                wakeup_data[-1][key].append(wakeup_song)\n",
    "                #a dictionary {date:[{song1 info},{song2 info}]}\n",
    "        mission_data.append({'Mission':m,\"WakeupCalls\":wakeup_data}) \n",
    "    program_data[p]=mission_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Add the data to the JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "for program in song_json['Programs']:\n",
    "    program_name = program['Title']\n",
    "    program['Missions'] = program_data[program_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Update the Metadata and save"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "song_json['Metadata'][0]['UpdatedBy'] = 'Ross Spencer & Matthew Allinson'\n",
    "song_json['Metadata'][0]['LastUpdatedDate'] = datetime.now().strftime('%Y-%m-%d')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open ('nasawakeupcalls.json', 'w') as outfile:\n",
    "    json.dump(song_json, outfile, sort_keys=True, indent=4, separators=(\",\", \": \"))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}