exercise-dropout-and-batch-normalization.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"**This notebook is an exercise in the [Intro to Deep Learning](https://www.kaggle.com/learn/intro-to-deep-learning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/dropout-and-batch-normalization).**\n\n---\n"},{"metadata":{},"cell_type":"markdown","source":"# Introduction #\n\nIn this exercise, you'll add dropout to the *Spotify* model from Exercise 4 and see how batch normalization can let you successfully train models on difficult datasets.\n\nRun the next cell to get started!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setup plotting\nimport matplotlib.pyplot as plt\nplt.style.use('seaborn-whitegrid')\n# Set Matplotlib defaults\nplt.rc('figure', autolayout=True)\nplt.rc('axes', labelweight='bold', labelsize='large',\n       titleweight='bold', titlesize=18, titlepad=10)\nplt.rc('animation', html='html5')\n\n# Setup feedback system\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.deep_learning_intro.ex5 import *","execution_count":2,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First load the *Spotify* dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nfrom sklearn.preprocessing import StandardScaler, OneHotEncoder\nfrom sklearn.compose import make_column_transformer\nfrom sklearn.model_selection import GroupShuffleSplit\n\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import callbacks\n\nspotify = pd.read_csv('../input/dl-course-data/spotify.csv')\n\nX = spotify.copy().dropna()\ny = X.pop('track_popularity')\nartists = X['track_artist']\n\nfeatures_num = ['danceability', 'energy', 'key', 'loudness', 'mode',\n                'speechiness', 'acousticness', 'instrumentalness',\n                'liveness', 'valence', 'tempo', 'duration_ms']\nfeatures_cat = ['playlist_genre']\n\npreprocessor = make_column_transformer(\n    (StandardScaler(), features_num),\n    (OneHotEncoder(), features_cat),\n)\n\ndef group_split(X, y, group, train_size=0.75):\n    splitter = GroupShuffleSplit(train_size=train_size)\n    train, test = next(splitter.split(X, y, groups=group))\n    return (X.iloc[train], X.iloc[test], y.iloc[train], y.iloc[test])\n\nX_train, X_valid, y_train, y_valid = group_split(X, y, artists)\n\nX_train = preprocessor.fit_transform(X_train)\nX_valid = preprocessor.transform(X_valid)\ny_train = y_train / 100\ny_valid = y_valid / 100\n\ninput_shape = [X_train.shape[1]]\nprint(\"Input shape: {}\".format(input_shape))","execution_count":3,"outputs":[{"output_type":"stream","text":"Input shape: [18]\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"# 1) Add Dropout to Spotify Model\n\nHere is the last model from Exercise 4. Add two dropout layers, one after the `Dense` layer with 128 units, and one after the `Dense` layer with 64 units. Set the dropout rate on both to `0.3`."},{"metadata":{"lines_to_next_cell":2,"trusted":true},"cell_type":"code","source":"# YOUR CODE HERE: Add two 30% dropout layers, one after 128 and one after 64\nmodel = keras.Sequential([\n    layers.Dense(128, activation='relu', input_shape=input_shape),\n    layers.Dropout(rate=0.3),\n    layers.Dense(64, activation='relu'),\n    layers.Dropout(rate=0.3),\n    layers.Dense(1)\n])\n\n# Check your answer\nq_1.check()","execution_count":8,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.25, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"1_Q1\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_1.hint()\n#q_1.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now run this next cell to train the model see the effect of adding dropout."},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(\n    optimizer='adam',\n    loss='mae',\n)\nhistory = model.fit(\n    X_train, y_train,\n    validation_data=(X_valid, y_valid),\n    batch_size=512,\n    epochs=50,\n    verbose=0,\n)\nhistory_df = pd.DataFrame(history.history)\nhistory_df.loc[:, ['loss', 'val_loss']].plot()\nprint(\"Minimum Validation Loss: {:0.4f}\".format(history_df['val_loss'].min()))","execution_count":9,"outputs":[{"output_type":"stream","text":"Minimum Validation Loss: 0.1943\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzL0lEQVR4nO3deXxU5cH3/8+ZNctkXwm7bIkgKKiIRGkLVWu5W8UqabWl3o8oVSv+ntIqaMHeKoL1sSi16uPya9W7imCKXVQs3i5YEqhpBZEoq2EnCQnZJ8kszx9nMhASEiCBZA7f9+s1r3PmnJkz11yEfHNd5zrXMYLBYBAREZFextbTBRAREWmPAkpERHolBZSIiPRKCigREemVFFAiItIrOXrqg4uKinrqo0VEpJcZN25cm209FlDQfoFOVnFxMTk5Od1QGmtS/XRM9XN8qpuOqX46djL1c7wGi7r4RESkV1JAiYhIr6SAEhGRXkkBJSIivZICSkREeiUFlIiI9EoKKBER6ZUUUCIi0ispoERELCQ/P5/Fixf3dDG6RY/OJNFVS1ZvIVBXjS7mFhGxnogOqPeKS4miqaeLISLS6/zhD3/grbfeAmDy5MnceuutfPzxxyxZsoSoqChSUlJ47LHHWLduXZttTqezh0tviuiAinXbqa3VHetFpPd5o2gPr3+yu1uPecOF/bluXL9OX7dnzx4KCgpYsWIFANdffz1XXXUVr7zyCvfeey8XXngh7777LocPH253W1paWreW+1RF9Dkoj9tBvS/Q08UQEelVNm/ezJgxY3A4HDgcDkaPHs0XX3zBVVddxYIFC3jmmWfIyckhLS2t3W29RYS3oBw0NCugRKT3uW5cvxNq7ZwOhmEQDB7pXQoGg9hsNq655houu+wyVq9ezU9+8hOeeOKJdrcNGTKkR8p9rIhuQZkBpS4+EZGjnXvuuXz66af4fD58Ph8bNmwgJyeHp556CofDwfTp07n66qvZvn17u9t6i4huQXnUghIRaaNv376MHz+em266iWAwyPXXX0/fvn3Jysri5ptvJj4+nvj4eG6++Wbq6urabOstIjqgYl0OGv1BfP4ADntENwZFRLrFtGnTwus33nhjq33XXnst1157bafbeouI/q0e67YDUNfk7+GSiIhId4vogPK4zQZgXaOvh0siIiLdLaIDKjYUULUKKBERy4nogPIooERELCuiAypWXXwiIpYV4QEVGiShgBIRsZyIDqgjXXwaxSciYjURHVDq4hMROTXf+MY3qKurO+7+8ePHn8HStC+iA0qDJERErCuiZ5JwO2zYDLWgRKQX+vRV+Pcr3XvMC26C87/f4UuuueYafve735GVlcXevXu58847SU9Pp76+Hq/Xyy9/+UtGjx59wh/55Zdf8l//9V/YbDZiY2NZtGgRdrudu+++m6amJpqampg/fz4DBgxote2HP/whOV28m2xEB5RhGMQ4bQooEZGQKVOm8P7773PjjTfy3nvvMXnyZLKzs5kyZQoFBQU899xzLF269ISP9/DDD/OLX/yCMWPG8MILL/DSSy+RnZ1NRkYGCxcuZPfu3ezYsYO9e/e22vbhhx92+btEdEABRDsNDZIQkd7n/O932to5Ha644goWL14cDqi5c+fywgsv8MILL9DU1ERMTMxJHW/btm2MGTMGgAsvvJCnn36avLw8lixZwvz587niiiuYNGkSpaWlrbaNGzeuy98los9BAUQ71IISEWkxfPhwSktL2b9/PzU1NaxevZqMjAxeffVVHnjggZM+nmEY4fVAIIDNZiM9PZ0333yTK664gldffZXf/va3bbYtW7asy98l4ltQMS4bdU0KKBGRFpMmTeI3v/kNkydPpqKighEjRgCwevVqmpubT+pYw4YN49///jcXXHAB//znPxk1ahRr166lubmZSZMmMXToUB544IE22+bMmdPl7xHxARXtsGkUn4jIUa644gry8vL4y1/+Ql1dHffccw/vvPMON954I3/961954403TvhY999/P7/61a8wDIOEhAQeeeQRDh8+zM9//nOef/55DMPgrrvuIjMzs9W2a665psvfI+IDKsZpUKaAEhEJGz16NJs3bw4/f/vtt8PrkydPBuC6667r8Bjr1q0DYOjQobz88sut9nk8Hl599dU27zl6W3Fx8ckX/BgRH1DRTht11RokISJyst577z1+//vft9n+ox/9iG9+85tnvkDHsERA1TY29nQxREQizuTJk8Mtqt7ohAJq4cKFbNiwAcMwmDdvXquLvAoLC3n88cex2WwMHjyYhx9+mIaGBu655x6qqqpobm7mjjvu4LLLLjstX6DlOqhgMNhqtImIiES2TgNq/fr1lJSUsGzZMrZt28bcuXNZvnx5eP/8+fN56aWXyMzM5K677mLNmjXs3r2bwYMH87Of/YyDBw8yY8YM3nnnndPyBaKdBr5AkEZfgCin/bR8hoiInHmdXgdVUFDAlClTAPNkWXV1NbW1teH9+fn5ZGZmApCcnExlZSVJSUkcPnwYgOrqapKSkk5D0U3RDvMr6FooERFr6bQFVV5ezsiRI8PPU1JSKCsrw+PxAISXpaWlrF27ltmzZ5OUlER+fj7f/OY3qa6u5tlnn2332N0zysMcILFh85f0iXN2+XhW4/V6u6WerUr1c3yqm46pfjrWHfXTaUAFg8E2z48913Po0CFmzZrF/PnzSUpK4s033yQrK4sXXniBL774gvvuu6/dcfddnUgQ4B8lnwCQ2W8QOVnxXT6e1RQXF3dLPVuV6uf4VDcdU/107GTqp6ioqN3tnXbxZWRkUF5eHn5eWlpKampq+HltbS0zZ85k9uzZ5ObmAvCvf/0rvJ6dnc3Bgwfx+U5PF1y00wxLzSYhImItnQbUxIkTWbVqFQCbN28mPT093K0HsGjRImbMmMGkSZPC2wYOHMiGDRsA2Lt3L7GxsTgcp2dEe7TT/AqaTUJExFo6TY2xY8cycuRI8vLyMAyDBQsWkJ+fT1xcHLm5uaxcuZKSkhJWrFgBwNSpU5k+fTrz5s3jpptuwufzndIEhScqRoMkREQs6YSaNcdO+pednR1e37RpU7vveeKJJ7pQrBPX0oJSQImIWEvk324jdA5K94QSEbGWiA+oGLWgREQsKeIDym4zcOumhSIilhPxAQXgcTs0ik9ExGIsEVCxbodaUCIiFmOZgNIgCRERa7FEQHncdrWgREQsxhIBFet2aKojERGLsUxAaZCEiIi1WCKgPC4NkhARsRpLBJQ5ik+DJERErMQSAeVx26lr8rW5d5WIiEQuSwRUrNtBMAj1TWpFiYhYhWUCCjQfn4iIlVgioDyhgNJIPhER67BEQB1pQamLT0TEKiwSUHZALSgRESuxREB5dA5KRMRyLBFQ4S4+TXckImIZlggoDZIQEbEeSwSUhpmLiFiPJQIqxtkySEKj+ERErMISAWWzGcS6dE8oERErsURAgW77LiJiNZYJKI/uCSUiYimWCSi1oERErMVCAWXXVEciIhZimYBSF5+IiLVYJqBi3Q7NJCEiYiHWCii1oERELMMyAaUuPhERa7FMQMW6HHibA/j8gZ4uioiIdAPrBFTonlB1TRrJJyJiBZYJKN0TSkTEWiwTUJrRXETEWk4ooBYuXMj06dPJy8tj48aNrfYVFhZyww03kJeXx9y5cwkEzHNAf/7zn/nOd77DtGnT+PDDD7u/5MfQPaFERKyl04Bav349JSUlLFu2jIceeogHH3yw1f758+fz5JNP8tprr1FXV8eaNWuorKzkqaee4o9//CPPPPMMq1evPm1foMWRFpTOQYmIWIGjsxcUFBQwZcoUAIYOHUp1dTW1tbV4PB4A8vPzw+vJyclUVlZSUFDAhAkT8Hg8eDyeNqF2OrQMklALSkTEGjoNqPLyckaOHBl+npKSQllZWTiUWpalpaWsXbuW2bNns3z5coLBIHfffTelpaX89Kc/ZcKECW2OXVxc3OUv4PV6KS4u5kB1MwBbv9rFQHtll49rFS31I+1T/Ryf6qZjqp+OdUf9dBpQwWCwzXPDMFptO3ToELNmzWL+/PkkJSUBcPDgQX7729+yb98+fvSjH/H++++3eV9OTk6XCg9myOXk5JBW2wh/2k1CSjo5OYO6fFyraKkfaZ/q5/hUNx1T/XTsZOqnqKio3e2dnoPKyMigvLw8/Ly0tJTU1NTw89raWmbOnMns2bPJzc0FzFbWBRdcgMPhYMCAAcTGxlJRUXFCBT1VGiQhImItnQbUxIkTWbVqFQCbN28mPT093K0HsGjRImbMmMGkSZPC23JzcyksLCQQCFBRUUF9fX24ZXW6uB027DZDw8xFRCyi0y6+sWPHMnLkSPLy8jAMgwULFpCfn09cXBy5ubmsXLmSkpISVqxYAcDUqVOZPn06V155JTNmzKChoYH7778fm+30XnJlGAaxLt0TSkTEKjoNKIA5c+a0ep6dnR1e37RpU7vvycvLIy8vrwtFO3maMFZExDosM5ME6JYbIiJWYrmAUgtKRMQaLBVQHrWgREQsw1IBFevWIAkREauwWECpi09ExCosFVAaxSciYh2WCqiWUXzHTs8kIiKRx1IB5XE78AWCNPoCPV0UERHpIksFVKzLvOWGRvKJiEQ+awWUblooImIZlgoozWguImIdlgqocAuqSQElIhLpLBlQakGJiEQ+SwWUJ3wOSgElIhLpLBVQsW6N4hMRsQpLBdSRQRIaxSciEuksFVCx6uITEbEMSwWU027D5bApoERELMBSAQWaMFZExCosF1DmPaEUUCIikc56AeVyaJCEiIgFWC6gdNt3ERFrsFxAxbodmupIRMQCLBdQGiQhImINlgsoDZIQEbEGCwaUQ/eDEhGxAMsFlCd0DioYDPZ0UUREpAssF1CxbgfBINQ3qRUlIhLJLBlQoPn4REQineUCyhO65YZG8omIRDbLBVSsq6UFpS4+EZFIZrmA8ui27yIilmC5gNI5KBERa7BuQGm6IxGRiHZCAbVw4UKmT59OXl4eGzdubLWvsLCQG264gby8PObOnUsgEAjv83q9TJ48mfz8/O4tdQfUxSciYg2dBtT69espKSlh2bJlPPTQQzz44IOt9s+fP58nn3yS1157jbq6OtasWRPe9/TTT5OYmNjthe5IbGgUn7r4REQiW6cBVVBQwJQpUwAYOnQo1dXV1NbWhvfn5+eTmZkJQHJyMpWVlQBs376dbdu28bWvfe00FPv4Wkbx6Z5QIiKRrdOAKi8vJykpKfw8JSWFsrKy8HOPxwNAaWkpa9euZdKkSQAsXryYe++9t7vL2ymbzSDGpQljRUQinaOzFxw7p10wGMQwjFbbDh06xKxZs5g/fz5JSUmsXLmS888/n/79+3d47OLi4lMocmter7fNcdx22HuwvFuOH+naqx85QvVzfKqbjql+OtYd9dNpQGVkZFBeXh5+XlpaSmpqavh5bW0tM2fOZPbs2eTm5gLwwQcfsHv3bj744AMOHDiAy+UiMzOTSy+9tNWxc3JyulR4MEPu2OMkxh7EEe3pluNHuvbqR45Q/Ryf6qZjqp+OnUz9FBUVtbu904CaOHEiS5cuJS8vj82bN5Oenh7u1gNYtGgRM2bMCHftASxZsiS8vnTpUvr27dsmnE4n3RNKRCTydRpQY8eOZeTIkeTl5WEYBgsWLCA/P5+4uDhyc3NZuXIlJSUlrFixAoCpU6cyffr0017wjsS6dE8oEZFI12lAAcyZM6fV8+zs7PD6pk2bOnzvT3/601MoVtd43A72V3nP+OeKiEj3sdxMEhC6q65mkhARiWjWDSidgxIRiWiWDCiP266pjkREIpwlAyrW7cDbHMDnD3T+YhER6ZUsGVCe8IzmGsknIhKpLBlQuieUiEjkU0CJiEivZMmA8oRuuaGBEiIikcuSAdVyyw3NJiEiErmsGVC6q66ISMSzZEB5dA5KRCTiWTKgwoMkNN2RiEjEsmRAedTFJyIS8SwZUFFOGzZDXXwiIpHMkgFlGEZowliN4hMRiVSWDCgwu/nUxSciErksG1C65YaISGSzdECpBSUiErksG1Aet10tKBGRCGbZgIp1aZCEiEgks2xApcW52Xe4QTctFBGJUJEdUGseJ3H7ynZ3XXJOCjWNPjburTqzZRIRkW4R2QG1q4DkLcva3TVxaCqGAR9vLT/DhRIRke4Q2QHV/2Lc1TuhvqLNruRYF6OyEhRQIiIRKsID6hJzueeTdnfnDkvlX7sqNdxcRCQCRXZA9R1H0LDD7sJ2d182NBVfIEjh9kNnuGAiItJVkR1Qrhi8icNh17p2d48blESU08bH29TNJyISaSI7oICG1NGwtwj8zW32uR12xg9OYc3Wsh4omYiIdEXEB1R92mjwNcCBje3uv2xYKtvL6th3uOEMl0xERLoi4gOqIXW0uXKcbr7cYamAhpuLiESaiA8oX3QaJAw47kCJERlxpMW5WaPzUCIiESXiAwqAAePNFlQw2GaXYRjkDk3lH9vKCQTa7hcRkd7JGgHVfzzUHoDDu9rdfdmwVCrqmti8v/oMF0xERE6VNQJqQOiC3d3HOQ81NHQeSt18IiIRwxoBlX4uuOJgV/vnodLjoxiREafh5iIiEcRxIi9auHAhGzZswDAM5s2bx+jRo8P7CgsLefzxx7HZbAwePJiHH34Ym83Go48+SlFRET6fj9tuu40rrrjitH0JbHbodyHsXn/cl+QOS+XlwhK8zX6inPbTVxYREekWnbag1q9fT0lJCcuWLeOhhx7iwQcfbLV//vz5PPnkk7z22mvU1dWxZs0aCgsL2bp1K8uWLeP5559n4cKFp+0LhA24BEo/B2/755lyh6XS5AuwfmfbiWVFRKT36TSgCgoKmDJlCgBDhw6lurqa2tra8P78/HwyMzMBSE5OprKykosuuognnngCgISEBBoaGvD7T/PdbfuPh2AA9vyz3d3jByfjsmvaIxGRSNFpF195eTkjR44MP09JSaGsrAyPxwMQXpaWlrJ27Vpmz56N3W4nJiYGgOXLl3P55Zdjt7ftVisuLu7yF/B6vRQXF2NrjmO4YaP833+jvDmr3dfmpLlY/dkerh3c5Y+NGC31I+1T/Ryf6qZjqp+OdUf9dBpQwWOuLQoGgxiG0WrboUOHmDVrFvPnzycpKSm8ffXq1axYsYIXX3yx3WPn5OScSplbKS4uPnKctSNJa9hO2nGOe8UBJ79e9SWp/c4hLc7d5c+OBK3qR9pQ/Ryf6qZjqp+OnUz9FBUVtbu90y6+jIwMysuPdIuVlpaSmpoafl5bW8vMmTOZPXs2ubm54e1r1qzhmWee4bnnniMuLu6ECtll/S8x7w3lb//+T5eFpj36h7r5RER6vU4DauLEiaxatQqAzZs3k56eHu7WA1i0aBEzZsxg0qRJ4W01NTU8+uijPPvssyQmJnZ/qY9nwCXQVGsOlmjHyKwEEmOcrNG8fCIivV6nXXxjx45l5MiR5OXlYRgGCxYsID8/n7i4OHJzc1m5ciUlJSWsWLECgKlTpwJQWVnJ3XffHT7O4sWLycpq/9xQt+l/sbnctQ76jGmz224zmDg0lY+3lbXbVSkiIr3HCV0HNWfOnFbPs7Ozw+ubNm1q9z3Tp0/vQrFOUUJ/iMsyZ5QYf2u7L7lsaCp/27ifbaW1DMs4Q12PIiJy0qwxk0QLwzAnjj3OlEdw5PYbH6mbT0SkV7NWQIE5UKJqN1TtbXd3v6QYBqfG8rGmPRIR6dUsGFCh81DHuT8UmJPHFu6ooKq+7W3iRUSkd7BeQGWeB86YDufly7u4P83+AL98s/3zZyIi0vOsF1B2J/Qdd9yZzcEcbn7X5GH8ecM+/rJh3xksnIiInCjrBRSY10Md+Awaa4/7ktu/NoTz+ydy/8pNHKjynsHCiYjIibBmQPUfD0E/7G1/+gwAh93G4zeModHn5xdvbGwzpZOIiPQsawZUv4sAo8PzUADnpHm47+ocPtpSxiuFJWembCIickKsGVDRiZCeA1vfhfqO7/900yUDuXx4Gg+/VcyOsuN3CYqIyJllzYACOO962LMe/s8IWP5j2PYeBAJtXmYYBr/+3mjcDjv/+/UN+PxtXyMiImeedQPqsv8NP1kLF/4v2PEBvDINnhgN7z8Ch3e1emlGfBQPXTOKT3cf5ukPtvdMeUVEpBXrBhRAxkj41iL42Zfwvf8fUofBh4thyWh49futRvn9x5gsvjMmiyfe28pne6p6sNAiIgJWD6gWDjeMmgY//BPcvREu/zlsWQVv3AKBI7eif/C7o0j1uPn/Xv8Ub/NpvkW9iIh06OwIqKMlDoBv3AdXPwpb3oZV94V3JcQ4efR7o9lWWsvv1NUnItKjzr6AanHRLXDJ7bDuaVj/XHjz5cPTuOb8LJ75YDs7y+t6sIAiIme3szegAK54CIZ/C97+BWz9e3jzvG/n4HbYmP/mJl3AKyLSQ87ugLLZ4brnIWOUORT9gDl5bHpcFHOuHMGareW89dmBni2jiMhZ6uwOKAC3B36wDNxx8MfpUGMG0k2XDGRkVjz/9dfPqfHqthwiImeaAgogPssMqYZKeDUPmuqw2wweumYUpTWNLFm9tadLKCJy1lFAtegzBr73AuzfAPm3QiDABQOS+P7FA/j92q/YvK+6p0soInJWUUAdbcS34MqF8MVf4d37AfjFlSNIiHbyyzc3EQhowISIyJmigDrW+Fnmo/ApKPgdiTEu5n4rm6KSSlYU7enp0omInDUUUMcyDLMVlfMfsGoefL6S68b246JBSTzydjGVdU09XUIRkbOCAqo9NjtMew76Xwz5t2LbXciD14yi2uvj0VVf9HTpRETOCgqo43FGQ96rkNgfXvs+2fYD/OfEQby6fjerPte1USIip5sCqiOxKXDjCrA54L+v4+5LEsjpE89tLxex4M1NmlBWROQ0UkB1JnmweY1UXTmxK37An24Zzc0TB/GHghL+Y+nHGn4uInKaKKBORN9x5v2kDmwkauUtLLh6BH/4z4s53NDMNU/9g+fX7NAQdBGRbqaAOlEjroJv/x/Y+i786VYmDUnindmXMWlEGg/9rZgfvbieA1Xeni6liIhlKKBOxoX/CVN+BZvegBU3kxJt4//+cByPTDuPopJKrnriI94rPtjTpRQRsQQF1MnKvdu8Tqr4z/D6DAx/E9+/eAB/uyuXfknR3PLSJzzz4XbdpkNEpIsUUKdiwh1w9WPw5d9g2U3Q7OWcNA/Lb7uUq8/rw6K3v2DO8o00+jTKT0TkVCmgTtXFM2HqEvOc1Gvfh+YGol12fvv9C7h7yjDe+NcebnxuHeW1jT1dUhGRiKSA6ooLb4bvPgXb34c/3gBNdRiGwd1ThvPbH1zAZ3ur+O5v/8EXBzQUXUTkZCmguuqCm+DaZ+Grj+GV70FjDQBTR2fx+m0TaPYHuO53a1m9WYMnREROxgkF1MKFC5k+fTp5eXls3Lix1b7CwkJuuOEG8vLymDt3LoFAoNP3WM6Y6eat43evgyfOh/95CKr3M6Z/In++M5dz0jzMfPkTHnm7mJJDdT1dWhGRiODo7AXr16+npKSEZcuWsW3bNubOncvy5cvD++fPn89LL71EZmYmd911F2vWrCE6OrrD91jSqOsgvh/8Ywl89Bh8/Bs49xoyx8/i9dsmMO9Pn/F/P9rBsx/u4MKBSVw7ti9Tz8siIcbZ0yUXEemVOg2ogoICpkyZAsDQoUOprq6mtrYWj8cDQH5+fng9OTmZyspKPv300w7fY1kDxsOAV6FiJ/zzefjXy7BpBdF9x/Gb8bOYM+VKVm4s40//3st9f9rEr/68mck56Uwb249Jw9NwOdTjKiLSotOAKi8vZ+TIkeHnKSkplJWVhcOmZVlaWsratWuZPXs2jz/+eIfvaVFcXNzlL+D1ervlON1uwE0YfaaR+NVbJG1djjt/JpnOOG7qM4HvnjuRjReMY9Uu+GBbKW9vOkC002BYipthKW6Gp7oZnuImw+PAMIw2h3Y0lOHZ9w88+z4m+tDnVA28krJRtxJ0xrR57fHqx1FfStDmxB+VdFq+fqTotT8/vYDqpmOqn451R/10GlDHXnAaDAbb/NI8dOgQs2bNYv78+SQlJZ3QewBycnJOpcytFBcXd8txTpvR4yBwH+z4H+yf/4mEL98hYde79LM5uHrgpfivuIp1zot5e280G/cc5i9f1NDkrwIgKcbJef0SGdc/gRsHVJC67wPY8jbs32AeO3EADLyElC3LSDnwMVz9KGR/u9XHt6mfqj3w4WL493+DIwqmPAAX3QK2s7P11ut/fnqQ6qZjqp+OnUz9FBUVtbu904DKyMigvLw8/Ly0tJTU1NTw89raWmbOnMns2bPJzc09ofecdWw2GDrFfAT8sLcIvnwLvnwH+7vzuBS41JMJdhfBVGgOBGn0Q6MviHdPgKiSWlKNagLYaMwcR/TkBTDiW5CWbd4BePd6+Mvd8NoPIHsqfGsxJPRrXYa6cljzuNn1SNAMpUNb4e2fw+f58J2lkDqsBypHRKR9nQbUxIkTWbp0KXl5eWzevJn09PRWXXWLFi1ixowZTJo06YTfc1az2c079fa/2Gy9VOyELe/Agc8gGMQIBnARxBUMEkcQggHq/A7y64fz6x0D2P9VLJOj0rmtXwYXpYEB5rFu+xAKnoIPFsFT4+Hr98HFt2JrqoX/eRgKfwfN9XD+D2DSPWbrKxiET/8Iq+bC0xPh63Nhwk/B3umPhYjIadfpb6KxY8cycuRI8vLyMAyDBQsWkJ+fT1xcHLm5uaxcuZKSkhJWrFgBwNSpU5k+fXqb98hxJA+GS37S4UtigWnA1+uaeLmwhN+v/Yobni3gggGJzLzsHM7rm0BSrIvYibMxRl4Df5tjhs6/X2HI4d3QVA3nfhe+fj+kDT9yYMOAC240W3Zv/QxWPwCf/8m8+DjzvNP4pUVEOmcEe2hW06KiIsaNG9fl45yN/cANTX5W/GsPz320g10V9eHtTrtBUoyL5BgnV9nWkVf7EjWuDIJTHmDImFzstrbnAVv5fCW8NQcaKuHC/wXZV0P/S8AZdXq/UA86G39+TpTqpmOqn46d7Dmo9vJAfTkRKNpl54eXDOQHFw+gYPsh9lc1UFnfRGV9M5V1TVTUNfGP+lzebL6YneV1sKyaxL/8nUuHpDBxaCqXDU1jQErbEX+MvAYGXw6r7oNPXoD1z4IjGgZOgHO+DkO+DukjT3xARWMN7PmneY5sVyHs+zfE94UBl8CACeaw/IT+ZktOROQYCqgIZrcZ5A7rePBJwb82UWpL4uOt5Xy8rZy3PjsAwIDkGC4YkMjg1FgGp8YyKCWWQamxJMQkw7VPmyMCv/oH7HjfnGvw77+EvwOxadB/PEQngSv2yMMZWhoG7PsUdhfCwc8hGAAMyBgF537HHEW48XUzAAHiskKBdQl4MiDgMx/+Zgg0g99nLl0ecxBH6nCISek81HyNcHg31B+CjHPBHdfl+haRM0sBZXGJ0XYm5PTlu+f3JRgMsr2sjo+3lvHxtkN88lUlf96wj6M7eZNjXQxKiWFAcgwZCQPJiJtFxuV3099RQd+K9STu/xj7gQ3QVAtN9eYyeMxtRVwe6HchXP5zM8z6XQRR8Uf2B/xmeO0qhF0F5vLz/BP/UlGJZlClDjMfMSlm8FWWwOESc1mzHwh9McNmnlMbMCH0uATiMk+1SkXkDFFAnUUMw2Bouoeh6R5+PHEwAN5mP7sq6tlZXsdX5XV8daiOneV1/POrSspqGmnyB446QjowjbS473PhwCTGD05m/OBkRqS6sPkazLDyN0PiwI5HAtrs0Ge0+Rh/qzmasGoPeKvA7gSbI7R0hpZ2c1/5NijfYg6PL98K296DT/879OVsZvdh4kA452uQNNBcj0owuxZ3FUDRH2DdM+brkwZD/4vpU1MPmxzQVHckcJvqzIdhM8+/OUIPZzQ43Oa6zW4GbUuLL+A78jwYNL+DzR5aOo58D0c0xGeZoygT+0PCAPOSANdRXa5NdUeF7Veh9V3mKMxgAAianxEMHFm6PWYL1JNhhq8nHTyZEJdhlrfhMHgPm+cXj14P+Mx6ShkCyeeYLdqz9Jo46X0UUGe5KKed4RlxDM9o2wUWDAY5XN/MgWovB6u9lFY3crDay87yOtbtrODtTWZ3YWKMk4sGJYcCK5Vh8QZR9pMohGGYv6zpf/zXRCdB0iAYNqX1dm81NFSYv1gdrvbfm321ufQ3w/6NsGut2Wrb+RGxPh/EJIa6KkO/5F2xZmAEg+DzQrMXfA1mt2F9hbktGGgbQi1BBBAImK8P1LYOr6Z6qNlnrh8tJtUMltqDUFfWep/LYwaay2PWlWEzHxhmmBgOsyvz4GaoK2177OMyQkF71OsdUWZ4J59DejAO9vUFuyv03Vzm92v548E4KsjCXa6hZVOtWab6itAytN5QYR4r+ZxQKA45Eo6JA1v/GwYC4G+E5lDdt/wbhP9NvEe2+5vNbtzoZLNFHZNstrSPDtvmBqjeB1W7zT+IWh4EzfqPTQs9UkOPNLM+GqvNP5C8oWVjNXirSdm7E8pSQw31UGs9eFSr3ZNu/tEUn2U+ohJ0vvUkKaDkuAzDICnWRVKsi5w+8W3276msZ92OCtbtPMS6nRX8PXRLEcOAfknRDE3zMCTNbLENSfdwTmos8dFOHDaj3ZlFTklUfOvuw47YndBvnPm49KcAbOuJkVgBv9kFeXi3+cvycIm5XnsQ+o41f1EnDTKDImngiZ1zCx87YIZAzQGoPQA1B81f5NFJEJ1o/tJuWXcnAEGo3gsVO+DQdnNZsRMqtpNUWQLbmk8i8I5lmJ8Tk2I+EvtDnzFmeSq2w8bl0Fh11MvtZrD4mkKh03SKn3v05yeZD28V1Je3fYknw/zcujLzXOdJSD/Z4jhjzKCK62OGlTM61DKPMVvqzhjzud1l/uHQ8oeIzW6W0WY3v1Orn4Wj1oN+s5XdVG+GcXNd2/Wmuta9BC3bg4HQcY2jljZz3eE2/zhq75zzmDwY9s2TrYkTpoCSU9YvKYZ+42K4bpw5a8X+qgaKSirZVlrLttJatpfVsXb7IRp9gTbvddgMHHYDp82Gw27gsNs4JzWWy4enMWl4Guf2icfW2bD4SGWzm916Cf2ACd18bNuRFgCjTuw9iQPMxzlfa7X5y5bwDgRCA1aazdAI+Mxl+OTlMa0HguCKM8PJ1kFTOhg0W1bhYNxuBoUjKtSVGupSPbpr9diu1paH3WWGXX2lecyGitatNne8OWK0pd4T+plh4XAfKYu3ypxxpb7cLEddmdlCc8ebgRIVf9R6Al/s2E12zrmhL2O0bkUGfOYfHNX7zD8AqvcdWa/Zb/4R4GsIhUe92SL0d/Pdt+2hunPFhpahkPGkHxU2HnOfYQv9+wVbL4OhnoBwmIUeDXvMZd+xCiiJDH0Sopk6OrrVtkAgyN7DDWwrq2VnWR31TT6a/UF8gQA+fzC83tgcYNO+Kn696kt+vepLUj0uLhtmhlXusFRSPe4e+laCzQY295Ff5t3FMI6E6YDx3XvsUylLdKL5YOgJvSVoP3ikS/dYNpfZYkzsoNv6WAG/GViBZjMcAn6zVdSyDAZC5yBbCnDMJayGzWyFuWLMcLfAjDCR/w2kV7PZDPonx9A/OYavj+j89aU1Xj7eWs6HW8r4cIt5axKAvonRxLjsxLjsRLvsxLgcRDvN9bgoBxnxUWTGR5ERH0VGvJvMhChiXPrxlghis5uDXSRM/4OlV0mPi2La2H5MG9uPQCDIpn1VfLSljB3ldTQ0+alv8tPQ5Ke0xktDaL2qoZm6Jn+bY8VFOeiTEMXAlNhW13sNTo0lI97dfefBROS0UEBJr2WzGYzul8jofomdvra20ceBKnO04cFqrznysMrL3sNeviqv48MtZTQddS4s2mlnQHIMvuZG7O+U4vMHafK3dDsGaPYHSIhx0jcxmqzE6PDSXI8iMyEaj1v/fUROJ/0PE0vwuB3ha7za4w8E2V/VwFfl9ewsr2VneT27KuqpqfGTlODB6bDhtBk47aFBGzaDww3N7DvcQOH2Qxyo9hI4pss/xmUnLc5Nepyb9Lgocz3eTUqsi2iXgxhn6y7JGJcdh93gUG0TZTWNlNU2msuaRsprG6lqaCYzPopz0jyckxbLkLRYBiTH6k7LctZSQMlZwW4zzFGHSTGtpoc60Qktff4AB2sa2Xe4gb2VDeZ1YTWN5qPaS/H+aj7a0khN48kPyfa4HaTFuYmPclC8v4blRXtalbt/UjTnpHlI87hJjHESH+0kMcZJQrSTxGgXCdFOPFGhc3JOO1EuGy67rVUXZiAQpMbrC83Z2MTh+mYq65uob/Ljcthwhx92c+m0cbCikcHNfqKcJ3NRm0j3UUCJnACH3UbfUFffRYOO/7r6Jh+V9c00NPmoP+qcWV3ouc8fJMXjItVjtrxSPW6iXa0DoNrbzM6yOnaU17KjrM58lNfx+b4qDtc3tzts/1g2g/AgEn8gSFVDc5sW4Al977/tY3hGHKP7JTCqbwKj+yUwIjMOt0OhJaefAkqkG5ldeV37bxUf5WRM/0TG9E9sd7+32RwYUtXQzOF6c1nX6KOh2QzDhmY/3qPWDQOSYlwkxrhIijFbX+a6i1iXnSZ/gEZfgCafuWxs9tPoC1C8/SuqjDg+21vFO58f4LV/7gbM27oMz4hjcGqsOUIzKYZ+SdH0T44hKzGqW8IrGAxS7fWZ5xOrzEdNo4/hGR5GZZn3PxPrU0CJRJgop50op52M+NN7n640fzk5OdmAGRh7Khv4bG8Vn+2tYlNouerzAzT7jzTNDAMy4qJIjXMRH2V2Q8ZHOUmIcRIf5SA+2ondZlDfaLYqw63L0POqhmYOVjdyoMpLQ3PbkZkt+iZGc17fBM7rl8DIrHhGZiWQEuvq8sXdwWCQfVVeNu+rZvO+ahqa/YwdkMiFg5JJViiecQooEemUYRy5nu3q8/qEt/sDQQ5We9lT2cDuinpzWVlPRV0T1Q3NbCutpdprtvK8zW27JqOcNmJdDqJddmJdDuKiHJybFc83stPN69oSzOvbMuOjiHHb+fJATTggN4VadkfKaJ7Pi49yEhflCD2OrMe6HcS5zaUn9Ih1OzhU18jne6vZvN98HK5vDh/PYTN4JhTAQ9JiuWhQMhcOSubiQckEg0GafAEO1zdxKHQftpZHbaOPIWkezuuXQFZClC5pOEUKKBE5ZXabER5+f/Hg5A5f2+jzU93gIxAMhi66dnR+l+djpA51M3HokUEu1d5mPt9bzRcHqqmsb6a6oZkar48abzPV3mYOVnvZVuqjttFHrdd3zOz8R7gdNrIz4/jWqEzOzUrg3D7x5PSJw2YYfLa3in9+VcEnX1Xy1mf7w12dbrtBo39np2VOiXVxXr8Es8UXavWletw47d03OjMYDOIPBHF04zF7AwWUiJwRboedtLjuHVwRH+VkwpAUJgxJOaHXN/r81DX6qWv0UeP1UdfkIz7KyZC02OP+cr9oUDIXDTLDNxAIsrW01gysL3dxTr9MkmNd4UdKaHLlaKedLQdr2LS3io17zO7Qj7aUtRqo4rAZRDvtuJ12ol228ChMt8OO02Fe8uC0myMynXbzuT9gnpur8ZpBXB1a1jb68AeCxLkdJMY6SW51ztEc6WkYEAiaYRYMQiAYJBA0W4r9k2IYkelhWEYc8VHHmb6pByigROSsYQ6jt5/y+SSbzWBEZhwjMuMYl1BPTs6w4772ggFJXDAgKfy8oclvdiPuq6KqoTk0mCVgLkMDWuqb/OZgleZAqMUXpMnnpzl0AbnNMIiLMrsxsxKjGBEVF37usBscrm/mcH0TlaHLCHaU13K4rrnV5Q82w+yybVkGg8FW5xGzEqIYnhkXvg1PepybWLeDWLfZDdvSNXomrs9TQImInAHRLjvjBiYxbmBS5y/uZoFA0LyVWDvnwlomdN5aWsOXB2rZcrCGLw/UsHb7oVazrxzLaTe48+vDmD3l+CHdVQooERGL62h049ETOn8jOyO83ecPsKuiPjzoo6VrtK7JR12jj9pGP5cOPbGu1VOlgBIRkTYcdlto2q2eK4O1hnyIiIhlKKBERKRXUkCJiEivpIASEZFeSQElIiK9kgJKRER6JQWUiIj0SgooERHplRRQIiLSKxnBYPAUbgTddUVFRT3xsSIi0guNGzeuzbYeCygREZGOqItPRER6JQWUiIj0SgooERHplSL2dhsLFy5kw4YNGIbBvHnzGD16dE8XqVfYsmULt99+Oz/+8Y+56aab2L9/P7/4xS/w+/2kpaXx61//Gpfr1O4magWPPvooRUVF+Hw+brvtNs477zzVD9DQ0MC9997LoUOHaGxs5Pbbbyc7O1t1cwyv18u3v/1t7rjjDiZMmKD6Cdm0aRO33347AwcOBGD48OHccsstXa6fiGxBrV+/npKSEpYtW8ZDDz3Egw8+2NNF6hXq6+t58MEHmTBhQnjbk08+yQ9+8AP++Mc/0rdvX1asWNGDJexZhYWFbN26lWXLlvH888+zcOFC1U/I+++/z6hRo3jllVdYsmQJixYtUt204+mnnyYxMRHQ/62j1dfXc+WVV/Lyyy/z8ssv88tf/rJb6iciA6qgoIApU6YAMHToUKqrq6mtre3hUvU8l8vFc889R3p6enjbunXrmDx5MgCTJ0+moKCgp4rX4y666CKeeOIJABISEmhoaFD9hFx99dXMnDkTgP3795ORkaG6Ocb27dvZtm0bX/va1wD93zpaXV1dm23dUT8RGVDl5eUkJSWFn6ekpFBWVtaDJeodHA4HUVFRrbY1NDSEm9VpaWlndT3Z7XZiYmIAWL58OZdffrnq5xh5eXnMmTOHefPmqW6OsXjxYu69997wc9XPEfX19RQVFXHLLbdw4403UlhY2C31E5HnoI69dCsYDGIYRg+Vpnc7ul50yZtp9erVrFixghdffJErr7wyvF31A6+99hrFxcX8/Oc/18/OUVauXMn5559P//79w9tUP0dkZ2dzxx13MHnyZHbu3MnNN9+Mz+cL7z/V+onIgMrIyKC8vDz8vLS0lNTU1B4sUe8VHR2N1+slKiqKgwcPtur+OxutWbOGZ555hueff564uDjVT8imTZtISUmhT58+5OTk4Pf7VTdH+eCDD9i9ezcffPABBw4cwOVyqX6OMmTIEIYMGQLA4MGDSU1NZf/+/V2un4js4ps4cSKrVq0CYPPmzaSnp+PxeHq4VL3TpZdeGq6rd999l8suu6yHS9RzampqePTRR3n22WfDJ7pVP6ZPPvmEF198ETC70Ovr61U3R1myZAlvvPEGr7/+Otdffz2333676ucoK1as4KWXXgKgrKyMQ4cOMW3atC7XT8ROdfTYY4/xySefYBgGCxYsIDs7u6eL1OM2bdrE4sWL2bt3Lw6Hg4yMDB577DHuvfdeGhsbycrK4pFHHsHpdPZ0UXvEsmXLWLp0KYMHDw5vW7RoEffff/9ZXz9er5f77rsv/FfvnXfeyahRo7jnnnvO+ro51tKlS+nbty+5ubmqn5CqqirmzJlDfX09TU1N3HnnneTk5HS5fiI2oERExNoisotPRESsTwElIiK9kgJKRER6JQWUiIj0SgooERHplRRQIiLSKymgRESkV/p/H8CeAgYB8QsAAAAASUVORK5CYII=\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"# 2) Evaluate Dropout\n\nRecall from Exercise 4 that this model tended to overfit the data around epoch 5. Did adding dropout seem to help prevent overfitting this time?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# View the solution (Run this cell to receive credit!)\nq_2.check()","execution_count":10,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.25, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"2_Q2\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct: \n\n\nFrom the learning curves, you can see that the validation loss remains near a constant minimum even though the training loss continues to decrease. So we can see that adding dropout did prevent overfitting this time. Moreover, by making it harder for the network to fit spurious patterns, dropout may have encouraged the network to seek out more of the true patterns, possibly improving the validation loss some as well).","text/markdown":"<span style=\"color:#33cc33\">Correct:</span> \n\n\nFrom the learning curves, you can see that the validation loss remains near a constant minimum even though the training loss continues to decrease. So we can see that adding dropout did prevent overfitting this time. Moreover, by making it harder for the network to fit spurious patterns, dropout may have encouraged the network to seek out more of the true patterns, possibly improving the validation loss some as well).\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"Now, we'll switch topics to explore how batch normalization can fix problems in training.\n\nLoad the *Concrete* dataset. We won't do any standardization this time. This will make the effect of batch normalization much more apparent."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\n\nconcrete = pd.read_csv('../input/dl-course-data/concrete.csv')\ndf = concrete.copy()\n\ndf_train = df.sample(frac=0.7, random_state=0)\ndf_valid = df.drop(df_train.index)\n\nX_train = df_train.drop('CompressiveStrength', axis=1)\nX_valid = df_valid.drop('CompressiveStrength', axis=1)\ny_train = df_train['CompressiveStrength']\ny_valid = df_valid['CompressiveStrength']\n\ninput_shape = [X_train.shape[1]]","execution_count":12,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run the following cell to train the network on the unstandardized *Concrete* data."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = keras.Sequential([\n    layers.Dense(512, activation='relu', input_shape=input_shape),\n    layers.Dense(512, activation='relu'),    \n    layers.Dense(512, activation='relu'),\n    layers.Dense(1),\n])\nmodel.compile(\n    optimizer='sgd', # SGD is more sensitive to differences of scale\n    loss='mae',\n    metrics=['mae'],\n)\nhistory = model.fit(\n    X_train, y_train,\n    validation_data=(X_valid, y_valid),\n    batch_size=64,\n    epochs=100,\n    verbose=0,\n)\n\nhistory_df = pd.DataFrame(history.history)\nhistory_df.loc[0:, ['loss', 'val_loss']].plot()\nprint((\"Minimum Validation Loss: {:0.4f}\").format(history_df['val_loss'].min()))","execution_count":13,"outputs":[{"output_type":"stream","text":"Minimum Validation Loss: nan\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYBUlEQVR4nO3cf0xV9/3H8dcFpE754RC5/mxihgbGFLUQl+LEfBV0pGtq9Q78kSyNq8vmfrhiW+xmYYvFH2mdRrFWpalzThEk6symjSskXbjTDTZ0lW3qEnttV7k3ShCUAHq+fzTe9Hqv2h0u+On1+fiLcz6cw+e8/+iz94LXYVmWJQAADBP1sDcAAEAoBAoAYCQCBQAwEoECABiJQAEAjBTzsDdwt8bGxoe9BQDAAHviiSeCzhkXKCn0Rr+oWlpalJ6e/rC3YRzmEoyZBGMmoUXaXO71woS3+AAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAggtTW1mrDhg0PexthQaAAAEYy8qOOAAB9s2fPHv3+97+XJM2ePVvLly/Xn/70J23evFmDBw/W8OHD9frrr+vUqVNB5wYNGvSQd/8pAgUA/eBQ42Ud/KsnrPf8dtY4LXhi7AO/7/Lly3K73aqpqZEkuVwuzZs3T7/5zW9UUlKirKwsvfvuu2prawt5bsSIEWHdt128xQcAEebcuXPKzMxUTEyMYmJiNHnyZP3zn//UvHnzVFpaqh07dig9PV0jRowIec4UvIICgH6w4Imxn+vVTn9wOByyLMt/bFmWoqKi9Mwzz+gb3/iGTp48qe9///vasmVLyHNf+cpXHsq+78YrKACIMF/96lf197//Xb29vert7VVzc7PS09NVUVGhmJgYFRYWqqCgQBcvXgx5zhS8ggKACDNmzBhNnz5dS5culWVZcrlcGjNmjEaPHq3nnntOCQkJSkhI0HPPPafOzs6gc6YgUAAQQZ599ln/10uWLAlYmz9/vubPn//Ac6bgLT4AgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAPAI+r//+z91dnbec3369OkDuJvQCBQAwEh8kgQA9Ie/75f+9pvw3nPqUmnKovt+yzPPPKPt27dr9OjR+uijj/TDH/5QKSkpunHjhrq6urRmzRpNnjz5c//If/3rX/rlL3+pqKgoDR06VOvXr1d0dLRWrlyp7u5udXd369VXX9Xjjz8edC4jI6NPj0ugACCCzJkzR3V1dVqyZIn++Mc/avbs2UpLS9OcOXPkdru1a9cubd269XPf77XXXtNLL72kzMxMVVZW6te//rXS0tLkdDpVXl4uj8ej//znP/roo4+CzvUVgQKA/jBl0QNf7fSH/Px8bdiwwR+o1atXq7KyUpWVleru7taQIUP+p/tduHBBmZmZkqSsrCy9+eabKioq0ubNm/Xqq68qPz9fubm5am1tDTrXV/wOCgAiyMSJE9Xa2qr//ve/un79uk6ePCmn06n9+/errKzsf76fw+Hwf3379m1FRUUpJSVFR44cUX5+vvbv369t27aFPNdXtgNVXl6uwsJCFRUV6cyZMwFrDQ0NWrhwoQoLC1VRURGw1tXVpdmzZ6u2ttbujwYA3Edubq5+9atfafbs2bp27Zoef/xxSdLJkyfV09PzP91rwoQJ+tvf/iZJ+stf/qKvfe1ramhoUENDg2bMmKE1a9boH//4R8hzfWXrLb7Tp0/r0qVLqqqq0oULF7R69WpVV1f719euXavKyko5nU4tXrxYc+fOVWpqqiTpzTff1LBhw/q8cQBAaPn5+SoqKtLvfvc7dXZ26uWXX9bx48e1ZMkSHTt2TIcOHfrc9/r5z3+uX/ziF3I4HEpMTNS6devU1tamF198Ubt375bD4dCPf/xjjRw5MuhcX9kKlNvt1pw5cyRJqampam9vV0dHh+Li4uTxeJSYmKhRo0ZJ+rTkbrdbqampunjxoi5cuKBZs2b1eeMAgNAmT56sc+fO+Y//8Ic/+L+ePXu2JGnBggX3vcepU6ckffrf+L179wasxcXFaf/+/UHXhDrXF7YC5fP5Av58cPjw4fJ6vYqLi5PX61VSUpJ/LTk5WR6PR5K0YcMGrVmzRocPH77v/VtaWuxsy0hdXV0R9TzhwlyCMZNgzCS0cM3l9OnTOnLkSND5b33rW/r617/e5/v3la1AWZYVdHznF2l3r0mf/pLt8OHDmjJlisaNG/fA+6enp9vZlpFaWloi6nnChbkEYybBmElo4ZpLenq6vvOd74RhR33T2NgY8rytQDmdTvl8Pv9xa2urkpOTQ65duXJFI0aMUH19vTwej+rr6/XJJ58oNjZWI0eO1JNPPmlnCwCACGcrUDk5Odq6dauKiop07tw5paSkKC4uTpI0duxYdXR06PLlyxo5cqTq6ur0+uuva+nSpf7rt27dqjFjxhAnAMA92QrUtGnTlJGRoaKiIjkcDpWWlqq2tlbx8fHKy8tTWVmZiouLJUkFBQUaP358WDcNAIh8tj9JYtWqVQHHaWlp/q+zs7NVVVV1z2t/9KMf2f2xAIBHBJ8kAQAwEoECABiJQAEAjESgAABGIlAAACMRKACAkQgUAMBIBAoAYCQCBQAwEoECABiJQAEAjESgAABGIlAAACMRKACAkQgUAMBIBAoAYCQCBQAwEoECABiJQAEAjESgAABGIlAAACMRKACAkQgUAMBIBAoAYCQCBQAwEoECABiJQAEAjESgAABGIlAAACMRKACAkQgUAMBIBAoAYCQCBQAwEoECABiJQAEAjESgAABGIlAAACPF2L2wvLxczc3NcjgceuWVVzR58mT/WkNDgzZt2qTo6GjNnDlTK1askCRt3LhRjY2N6u3t1fe+9z3l5+f3/QkAABHJVqBOnz6tS5cuqaqqShcuXNDq1atVXV3tX1+7dq0qKyvldDq1ePFizZ07Vz6fT+fPn1dVVZWuXbum+fPnEygAwD3ZCpTb7dacOXMkSampqWpvb1dHR4fi4uLk8XiUmJioUaNGSZJyc3Pldru1ePFi/6usxMRE3bx5U7du3VJ0dHSYHgUAEElsBcrn8ykjI8N/PHz4cHm9XsXFxcnr9SopKcm/lpycLI/Ho+joaA0ZMkSSVF1drZkzZ94zTi0tLXa2ZaSurq6Iep5wYS7BmEkwZhLaozIXW4GyLCvo2OFwhFyT5F+TpJMnT6qmpkZvv/32Pe+fnp5uZ1tGamlpiajnCRfmEoyZBGMmoUXaXBobG0OetxUop9Mpn8/nP25tbVVycnLItStXrmjEiBGSpPfff187duzQ7t27FR8fb+dHAwAeEbb+zDwnJ0cnTpyQJJ07d04pKSmKi4uTJI0dO1YdHR26fPmyent7VVdXp5ycHF2/fl0bN27UW2+9pWHDhoXtAQAAkcnWK6hp06YpIyNDRUVFcjgcKi0tVW1treLj45WXl6eysjIVFxdLkgoKCjR+/Hj/X++tXLnSf58NGzZo9OjRYXkQAEBksf3voFatWhVwnJaW5v86OztbVVVVAeuFhYUqLCy0++MAAI8YPkkCAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARoqxe2F5ebmam5vlcDj0yiuvaPLkyf61hoYGbdq0SdHR0Zo5c6ZWrFjxwGsAAPgsW4E6ffq0Ll26pKqqKl24cEGrV69WdXW1f33t2rWqrKyU0+nU4sWLNXfuXF29evW+1wAA8Fm2AuV2uzVnzhxJUmpqqtrb29XR0aG4uDh5PB4lJiZq1KhRkqTc3Fy53W5dvXr1ntcAAHA3W4Hy+XzKyMjwHw8fPlxer1dxcXHyer1KSkryryUnJ8vj8ejatWv3vOZuLS0tdrZlpK6uroh6nnBhLsGYSTBmEtqjMhdbgbIsK+jY4XCEXJMkh8Nx32vulp6ebmdbRmppaYmo5wkX5hKMmQRjJqFF2lwaGxtDnrcVKKfTKZ/P5z9ubW1VcnJyyLUrV65oxIgRiomJuec1AADczdafmefk5OjEiROSpHPnziklJcX/Vt3YsWPV0dGhy5cvq7e3V3V1dcrJybnvNQAA3M3WK6hp06YpIyNDRUVFcjgcKi0tVW1treLj45WXl6eysjIVFxdLkgoKCjR+/HiNHz8+6BoAAO7F9r+DWrVqVcBxWlqa/+vs7GxVVVU98BoAAO6FT5IAABiJQAEAjESgAABGIlAAACMRKACAkQgUAMBIBAoAYCQCBQAwEoECABiJQAEAjESgAABGIlAAACMRKACAkQgUAMBIBAoAYCQCBQAwEoECABiJQAEAjESgAABGIlAAACMRKACAkQgUAMBIBAoAYCQCBQAwEoECABiJQAEAjESgAABGIlAAACMRKACAkQgUAMBIBAoAYCQCBQAwEoECABiJQAEAjESgAABGIlAAACMRKACAkWLsXNTT06OSkhJ9/PHHio6O1rp16zRu3LiA7zl69Kj27NmjqKgoFRYWauHChert7dXPfvYzeTwe9fb26qWXXlJWVlZYHgQAEFlsvYI6duyYEhIStH//fj3//PN64403AtZv3LihiooKvfPOO9q7d692796ttrY2HTlyRF/60pf029/+Vq+99prWr18flocAAEQeW4Fyu93Ky8uTJM2YMUONjY0B683NzZo0aZLi4+M1ePBgZWVlqampSU8//bRWr14tSUpKSlJbW1vfdg8AiFi23uLz+XxKSkqSJEVHRysqKkrd3d2KjY0NWpek5ORkeb1eDRo0yH9uz549euqpp0Lev6Wlxc62jNTV1RVRzxMuzCUYMwnGTEJ7VObywEBVV1eruro64Fxzc3PAsWVZcjgcAcf3W9+3b58++OAD7dixI+TPTE9Pf/DOvyBaWloi6nnChbkEYybBmElokTaXu9+Fu+OBgXK5XHK5XAHnSkpK5PV6lZaWpp6eHlmWFfDqyOl0qr6+3n/c2tqqKVOmSPo0eO+99562b98ecA0AAJ9l63dQOTk5On78uCSprq5O06dPD1jPzMzU2bNn1d7ers7OTjU1NSkrK0sej0cHDhzQtm3b9Nhjj/V99wCAiGXrd1AFBQVqaGjQokWLFBsb6/9rvJ07dyo7O1tTp05VcXGxli1bJofDoRUrVig+Pl67du1SW1ubli9f7r9XZWWl/3dXAADcYStQd/7t090+G5558+Zp3rx5AesvvPCCXnjhBTs/EgDwiOGTJAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkW4Hq6elRcXGxFi1apKVLl8rj8QR9z9GjR7VgwQK5XC7V1NQErPl8PmVnZ+vUqVP2dg0AiHi2AnXs2DElJCRo//79ev755/XGG28ErN+4cUMVFRV65513tHfvXu3evVttbW3+9Y0bN2rcuHF92jgAILLZCpTb7VZeXp4kacaMGWpsbAxYb25u1qRJkxQfH6/BgwcrKytLTU1N/muHDh2qiRMn9nHrAIBIFmPnIp/Pp6SkJElSdHS0oqKi1N3drdjY2KB1SUpOTpbX61V3d7cqKiq0fft2lZeX3/P+LS0tdrZlpK6uroh6nnBhLsGYSTBmEtqjMpcHBqq6ulrV1dUB55qbmwOOLcuSw+EIOA61vnPnTrlcLiUkJNz3Z6anpz9w418ULS0tEfU84cJcgjGTYMwktEiby93vwt3xwEC5XC65XK6AcyUlJfJ6vUpLS1NPT48sy9KgQYP8606nU/X19f7j1tZWTZkyRbW1tbp9+7b27dunDz/8UGfOnNGWLVs0YcIEm48FAIhUtn4HlZOTo+PHj0uS6urqNH369ID1zMxMnT17Vu3t7ers7FRTU5OysrJ04MABHTx4UAcPHtSsWbNUWlpKnAAAIdn6HVRBQYEaGhq0aNEixcbGav369ZKknTt3Kjs7W1OnTlVxcbGWLVsmh8OhFStWKD4+PqwbBwBENluBio6O1rp164LOL1++3P/1vHnzNG/evHve407UAAAIhU+SAAAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjESgAgJEIFADASAQKAGAkAgUAMBKBAgAYiUABAIxEoAAARiJQAAAjOSzLsh72Jj6rsbHxYW8BADDAnnjiiaBzxgUKAACJt/gAAIYiUAAAIxEoAICRCFQY9PT0qLi4WIsWLdLSpUvl8XiCvufo0aNasGCBXC6XampqAtZ8Pp+ys7N16tSpgdpyv7M7k97eXr388stavHixvv3tb+uvf/3rQG+9X5SXl6uwsFBFRUU6c+ZMwFpDQ4MWLlyowsJCVVRUfK5rIoWduWzcuFGFhYVasGCB3n333YHecr+zMxNJ6urq0uzZs1VbWzuQ2+1fFvqstrbWKisrsyzLsurr662f/OQnAeudnZ1Wfn6+1d7ebt28edOaO3eude3aNf/6iy++aM2fP9/685//PIC77l92Z1JTU2OVlpZalmVZ//73v60FCxYM8M7D79SpU9by5csty7Ks8+fPWwsXLgxY/+Y3v2l9/PHH1q1bt6zCwkLr/PnzD7wmEtiZi9vttr773e9almVZV69etXJzcwd62/3Kzkzu2LRpk/Xss89ahw4dGtA99ydeQYWB2+1WXl6eJGnGjBlBfyrf3NysSZMmKT4+XoMHD1ZWVpaampr81w4dOlQTJ04c8H33J7szefrpp7V69WpJUlJSktra2gZ662Hndrs1Z84cSVJqaqra29vV0dEhSfJ4PEpMTNSoUaMUFRWl3Nxcud3u+14TKezMJTs7W1u2bJEkJSYm6ubNm7p169ZDe4ZwszMTSbp48aIuXLigWbNmPayt9wsCFQY+n09JSUmSpOjoaEVFRam7uzvkuiQlJyfL6/Wqu7tbFRUV+ulPfzrge+5vdmcyaNAgPfbYY5KkPXv26KmnnhrYjfcDn8+nL3/5y/7j4cOHy+v1SpK8Xm/IOdzvmkhhZy7R0dEaMmSIJKm6ulozZ85UdHT0wG68H9mZiSRt2LBBJSUlA7vZARDzsDfwRVNdXa3q6uqAc83NzQHHlmXJ4XAEHIda37lzp1wulxISEvpvwwMgnDO5Y9++ffrggw+0Y8eOftjxwLrfs969JkkOh+OB84kEduZyx8mTJ1VTU6O33367fzc5wOzM5PDhw5oyZYrGjRs3IHscSATqf+RyueRyuQLOlZSUyOv1Ki0tTT09PbIsS4MGDfKvO51O1dfX+49bW1s1ZcoU1dbW6vbt29q3b58+/PBDnTlzRlu2bNGECRMG6nHCIpwzkT4N3nvvvaft27cHXPNF5XQ65fP5/Metra1KTk4OuXblyhWNGDFCMTEx97wmUtiZiyS9//772rFjh3bv3q34+PiB3XQ/szOT+vp6eTwe1dfX65NPPlFsbKxGjhypJ598csD3H268xRcGOTk5On78uCSprq5O06dPD1jPzMzU2bNn1d7ers7OTjU1NSkrK0sHDhzQwYMHdfDgQc2aNUulpaVfuDjdi92ZeDweHThwQNu2bfO/1fdFl5OToxMnTkiSzp07p5SUFMXFxUmSxo4dq46ODl2+fFm9vb2qq6tTTk7Ofa+JFHbmcv36dW3cuFFvvfWWhg0b9hB33z/szGTz5s06dOiQDh48KJfLpR/84AcRESeJV1BhUVBQoIaGBi1atEixsbFav369JGnnzp3Kzs7W1KlTVVxcrGXLlsnhcGjFihUR939+d7M7k127dqmtrU3Lly/336uyslKxsbEP61H6bNq0acrIyFBRUZEcDodKS0tVW1ur+Ph45eXlqaysTMXFxZI+ndv48eM1fvz4oGsijZ25VFVV6dq1a1q5cqX/Phs2bNDo0aMf0lOEl52ZRDI+iw8AYCTe4gMAGIlAAQCMRKAAAEYiUAAAIxEoAICRCBQAwEgECgBgpP8HoNfhe8kDebMAAAAASUVORK5CYII=\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"Did you end up with a blank graph? Trying to train this network on this dataset will usually fail. Even when it does converge (due to a lucky weight initialization), it tends to converge to a very large number.\n\n# 3) Add Batch Normalization Layers\n\nBatch normalization can help correct problems like this.\n\nAdd four `BatchNormalization` layers, one before each of the dense layers. (Remember to move the `input_shape` argument to the new first layer.)"},{"metadata":{"lines_to_next_cell":0,"trusted":true},"cell_type":"code","source":"# YOUR CODE HERE: Add a BatchNormalization layer before each Dense layer\nmodel = keras.Sequential([\n    layers.BatchNormalization(input_shape=input_shape),\n    layers.Dense(512, activation='relu'),\n    layers.BatchNormalization(),\n    layers.Dense(512, activation='relu'),\n    layers.BatchNormalization(),\n    layers.Dense(512, activation='relu'),\n    layers.BatchNormalization(),\n    layers.Dense(1),\n])\n\n# Check your answer\nq_3.check()","execution_count":14,"outputs":[{"output_type":"error","ename":"SyntaxError","evalue":"invalid syntax (<ipython-input-14-825b6451d0d9>, line 6)","traceback":["\u001b[0;36m  File \u001b[0;32m\"<ipython-input-14-825b6451d0d9>\"\u001b[0;36m, line \u001b[0;32m6\u001b[0m\n\u001b[0;31m    layers.Dense(512, activation='relu'),\u001b[0m\n\u001b[0m         ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"]}]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_3.hint()\n#q_3.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run the next cell to see if batch normalization will let us train the model."},{"metadata":{"trusted":false},"cell_type":"code","source":"model.compile(\n    optimizer='sgd',\n    loss='mae',\n    metrics=['mae'],\n)\nEPOCHS = 100\nhistory = model.fit(\n    X_train, y_train,\n    validation_data=(X_valid, y_valid),\n    batch_size=64,\n    epochs=EPOCHS,\n    verbose=0,\n)\n\nhistory_df = pd.DataFrame(history.history)\nhistory_df.loc[0:, ['loss', 'val_loss']].plot()\nprint((\"Minimum Validation Loss: {:0.4f}\").format(history_df['val_loss'].min()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4) Evaluate Batch Normalization\n\nDid adding batch normalization help?"},{"metadata":{"lines_to_next_cell":0,"trusted":false},"cell_type":"code","source":"# View the solution (Run this cell to receive credit!)\nq_4.check()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Keep Going #\n\n[**Create neural networks**](https://www.kaggle.com/ryanholbrook/binary-classification) for binary classification."},{"metadata":{},"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/191966) to chat with other Learners.*"}],"metadata":{"jupytext":{"cell_metadata_filter":"-all","formats":"ipynb"},"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.7.9","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}