Simple_Python_NN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Python neural network\n",
    "\n",
    "The feed-forward neural network example in Python has been split into two functions; one that makes predictions and one that does the training. It would also be possible to construct this neural network using classes (custom kinds of Python objects), and this may hold certain advantages, like the ability to make adapted sub-classes. However, using functions usually makes it simpler to describe the principles of what is happening.\n",
    "\n",
    "The first function is called `neuralNetPredict`, which takes some input data for the first layer of network nodes, applies the first weighted connections and trigger functions to pass signal to the hidden layer of nodes and then applies the second weights and triggers to generate some output. This is used both during the training of the network, to set up the connection weights, and to make predictions on unseen data. Initially some mathematical functions are imported from the NumPy library, so that we can express the operations concisely as arrays and matrices (although using NumPy can actually be slower for small neural networks):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy\n",
    "from numpy import exp, power, array, zeros, sqrt \n",
    "from numpy import nonzero, random, abs, sum, dot\n",
    "from numpy import tanh, ones, append"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we define the function name and its input arguments; an array of input features (`inputVec`) and two matrices that represent the connection weights. The matrix `weightsIn` represents the strength of connection between the input nodes (which include the bias node we describe below) and the hidden nodes. Likewise, `weightsOut` represents the strengths between the hidden and the output nodes. The weights are represented as matrices so that the rows correspond to a set of nodes in one layer and the columns represent the set of nodes in the other layer; to connect everything in one layer to everything in the other. For example if the network has 4 input, 5 hidden and 2 output nodes, then weightsIn will be a 4 x 5 matrix, and `weightsOut` will be a 5 x 2 matrix. Inside the function the first step is to define the signalIn vector for the network. This is simply a copy of the input features array with an extra value of 1.0 appended to the end. This extra, fixed input is what is known as a bias node, and is present so the baseline (level without meaningful signal) of an input can be adjusted. This gives more flexibility in the trigger function used for the hidden layer of nodes, which improves learning. The weight matrices must be of the right size to account for the bias node, and although weights from the bias node are still adjusted by training they are naturally not affected by the input data. A bias connection going to each hidden node enables the input to that node to be offset, effectively shifting the centre of the trigger function about so that it can better distinguish the input values; the upshot of this is that the programmer doesn’t have to worry about centring input feature values (e.g. making their mean values zero)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neuralNetPredict(inputVec, weightsIn, weightsOut):\n",
    "\n",
    "  signalIn = append(inputVec, 1.0) # input layer\n",
    "\n",
    "  prod = signalIn * weightsIn.T\n",
    "  sums = sum(prod, axis=1)\n",
    "  signalHid = tanh(sums)  # hidden  layer\n",
    "\n",
    "  prod = signalHid * weightsOut.T\n",
    "  sums = sum(prod, axis=1)\n",
    "  signalOut = tanh(sums)  # output  layer\n",
    "\n",
    "  return signalIn, signalHid, signalOut"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main operation of the function involves multiplying the input vector, element-by-element, with the columns of the first matrix of weights. As a result of the training process we describe later, the weight matrix is arranged so that there is a column for each of the hidden nodes. Given we want to apply the input signal to each hidden node, we use the transpose (`.T`) of the weight matrix so that columns are switched with rows for the multiplication. This is a requirement because element multiplication of a one-dimensional NumPy array with a two-dimensional array is done on a per row basis. Next we calculate the summation of the weighted input down each column (`axis=1`), so we get one value for each hidden node. Then to get the signal that comes from the hidden layer we calculate the hyperbolic tangent of the sums; applying the sigmoid shaped trigger function to each. This whole operation is then repeated in the same manner for going from the hidden layer to the output layer; we apply weights to the signal vector, sum over columns and apply the trigger function. The final output vector is the prediction from the network. At the end of the function we return all the signal vectors, and although only the output values are useful in making predictions the other vectors are used in training the network.\n",
    "\n",
    "The second Python function for the feed-forward neural network is a function to train it by the back-propagation method; to find an optimal pair of weight matrices. The objective is to minimise error between the output vectors predicted by the network and the target values (known because this is training data). Here the error is calculated as the sum of the squared differences, but other methods may be more appropriate in certain situations. The function is defined and takes the training data as an argument, which is expected to be an array containing pairs of items; an input feature vector and the known output vector. The next argument is the number of nodes in the hidden layer; the size of input and output layers need not be specified because they can be deduced from the length of the of the input and output vectors used in training. The remaining arguments relate to the number of training steps (cycles over the data) that will be made, a value for the learning rate that governs how strongly weights are adjusted and a momentum factor that allows the each training cycle to use a fraction of the adjustments that were used in the previous cycle, which makes for smoother training. In practice the learning rate and momentum factor can be optimised, but the default values are generally a fair start.\n",
    "\n",
    "<pre>def neuralNetTrain(trainData, numHid, steps=100, rate=0.5, momentum=0.2):</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "       \n",
    "Within the function a few values are initialised. The numbers of nodes in the input and output layers are extracted from size of the first item (index zero) of training data, noting that the number of inputs is then increased by one to accommodate the bias node. The error value which we aim to minimise starts as None, but will be filled with numeric values later. \n",
    "<pre>                          \n",
    "  numInp = len(trainData[0][0])\n",
    "  numOut = len(trainData[0][1])\n",
    "  numInp += 1\n",
    "  minError = None\n",
    "</pre>       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we make the initial signal vectors as arrays of the required sizes (a value comes from each node) with all elements starting out as 1 courtesy of `numpy.ones()`. The input will be the feature vector we pass in and the output will be the prediction.\n",
    "<pre> \n",
    "  sigInp = ones(numInp)\n",
    "  sigHid = ones(numHid)\n",
    "  sigOut = ones(numOut)\n",
    "</pre>       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The initial weight matrices are constructed with random values between -0.5 and 0.5, with the required number of rows and columns in each. The random.random function makes matrices of random numbers in the range 0.0 to 1.0, but by taking 0.5 away (from every element) we shift this range. This range is not strict, but is a fairly good general strategy; too small and the network can get stuck, but too large and the learning is stifled. The best weight matrices, which is what we are going to pass back from the function at the end of training, starts as these initial weights but then improves.\n",
    "<pre>     \n",
    "  wInp = random.random((numInp, numHid))-0.5\n",
    "  wOut = random.random((numHid, numOut))-0.5\n",
    "  bestWeightMatrices = (wInp, wOut)\n",
    "</pre>        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next initialisation is for the change matrices, which will indicate how much the weight matrices differ from one training cycle to the next. These are important so that there is a degree of memory or momentum in the training; strong corrections to the weights will tend to keep going and help convergence.\n",
    "<pre> \n",
    "  cInp = zeros((numInp, numHid))\n",
    "  cOut = zeros((numHid, numOut))\n",
    "</pre>        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final initialisation is for the training data; pairs of input and output vectors. This is done to convert all of the vectors into `numpy.array` data type, thus allowing the training data to be input as lists and/or tuples. We simply loop though the data, extract each pair, convert to arrays and then put the pair back in the list at the appropriate index (`x`).\n",
    "<pre>       \n",
    "  for x, (inputs, knownOut) in enumerate(trainData):\n",
    "    trainData[x] = (array(inputs), array(knownOut))\n",
    "</pre>       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With everything initialised, we can then begin the actual network training, so we go through the required number of loops and in Python 2 use `xrange()` so that a large list doesn’t have to be created. Note we don’t use a while loop to check for convergence on the error because a neural network is not always guaranteed to converge and sometimes it can stall before convergence. For each step we shuffle the training data, which is often very important for training; without this there is a bias in the way the weights get optimised. After the shuffle, the error starts at zero for the cycle.\n",
    "<pre>         \n",
    "  for step in range(steps):  # xrange in Python 2\n",
    "    random.shuffle(trainData)  # Important\n",
    "    error = 0.0\n",
    "</pre>     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we loop through all of the training data, getting the input feature vector and known output for each example. We then use the current values of the weight matrices, with the prediction function described above, to calculate the signal vectors. Initially the output signal vector (the prediction) will be quite different from the known output vector, but this will hopefully improve over time.\n",
    "<pre> \n",
    "    for inputs, knownOut in trainData:\n",
    "      sigIn, sigHid, sigOut = neuralNetPredict(inputs, wInp, wOut)\n",
    "</pre> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the neural network signals that come from the current estimates for weight matrices we now apply the back propagation method to try to reduce the error in the prediction. Thus we calculate the difference between the known output vector and the signal output from the neural network. This difference is squared and summed up over all the features (diff is an array) before being added to the total error for this cycle.\n",
    "<pre> \n",
    "      diff = knownOut - sigOut\n",
    "      error += sum(diff * diff)\n",
    "</pre> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we work out an adjustment that will be made to the output weights, to hopefully reduce the error. The adjustment is calculated from the gradient of the trigger function. Because this example uses a hyperbolic tangent function, the gradient at the signal value is one minus the signal value squared (differentiate *y = tanh(x)* and you get *1 - tanh^2(x)* which equals *1 - y^2*). The signal gradient times the signal difference then represents the change in the signal before the trigger function; which can be used to adjust the weight matrices. Note that all these mathematical operations are performed on all the elements of whole arrays at once, courtesy of NumPy.\n",
    "<pre> \n",
    "      gradient = ones(numOut) - (sigOut*sigOut)\n",
    "      outAdjust = gradient * diff \n",
    "</pre> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same kind of operation is repeated for the hidden layer, to find the adjustment that will be made for the input weight matrix. Again, we calculate a signal difference and a trigger function gradient and multiply them to get an adjustment for what goes into the trigger function. However, this time we can’t compare output vectors, so instead we take the array of signal adjustments just calculated and propagate them back though the network. Thus the signal difference for the hidden layer is calculated by taking the signal adjustment for the output later and passing it through the output weight matrix, i.e. backwards through the last layer. \n",
    "<pre> \n",
    "      diff = sum(outAdjust * wOut, axis=1)\n",
    "      gradient = ones(numHid) - (sigHid*sigHid)\n",
    "      hidAdjust = gradient * diff \n",
    "</pre> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the adjustments calculated it then remains to make the changes to the weight matrices, and hopefully get an improvement in the error. The weight change going from hidden to output layers requires that we calculate a change matrix (the same size as the weights), hence we take the vector of adjustments and the vector of hidden signals and combine them; each row of adjustments (one per output) is multiplied by a column of signals (one per hidden node) to get the new weights. Note how we use the `reshape()` function to convert the array of signals, a single row, into a column vector; it is tipped on its side so that the multiplication can be made to generate a matrix with rows and columns.\n",
    "<pre> \n",
    "      # update output \n",
    "      change = outAdjust * sigHid.reshape(numHid, 1)\n",
    "      wOut += (rate * change) + (momentum * cOut)\n",
    "      cOut = change\n",
    "</pre> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the same manner the changes are made to the input weight matrix.\n",
    "<pre>  \n",
    "      # update input \n",
    "      change = hidAdjust * sigIn.reshape(numInp, 1)\n",
    "      wInp += (rate * change) + (momentum * cInp)\n",
    "      cInp = change\n",
    "</pre> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then finally in the training cycle, we see if the minimum error has been improved on. During the first cycle the minimum error is None, so we always fill it with the first real calculated error value in that case. Each time we find a new minimum error we record the best weight matrices (so far) by taking copies of the current versions, using the handy `.copy()` function of NumPy arrays. Then finally at the end of all of the training cycles, the best weight matrices are returned.\n",
    "<pre> \n",
    "    if (minError is None) or (error < minError):\n",
    "      minError = error\n",
    "      bestWeightMatrices = (wInp.copy(), wOut.copy())\n",
    "      print(\"Step: %d Error: %f\" % (step, error))\n",
    "  \n",
    "  return bestWeightMatrices\n",
    "</pre> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the complete function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neuralNetTrain(trainData, numHid, steps=100, rate=0.5, momentum=0.2):\n",
    "\n",
    "  numInp = len(trainData[0][0])\n",
    "  numOut = len(trainData[0][1])\n",
    "  numInp += 1\n",
    "  minError = None\n",
    "\n",
    "  sigInp = ones(numInp)\n",
    "  sigHid = ones(numHid)\n",
    "  sigOut = ones(numOut)\n",
    "\n",
    "  wInp = random.random((numInp, numHid))-0.5\n",
    "  wOut = random.random((numHid, numOut))-0.5\n",
    "  bestWeightMatrices = (wInp, wOut)\n",
    "\n",
    "  cInp = zeros((numInp, numHid))\n",
    "  cOut = zeros((numHid, numOut))\n",
    "\n",
    "  for x, (inputs, knownOut) in enumerate(trainData):\n",
    "    trainData[x] = (array(inputs), array(knownOut))\n",
    " \n",
    "  for step in range(steps):  # xrange in Python 2\n",
    "    random.shuffle(trainData) # Important\n",
    "    error = 0.0\n",
    " \n",
    "    for inputs, knownOut in trainData:\n",
    "      sigIn, sigHid, sigOut = neuralNetPredict(inputs, wInp, wOut)\n",
    "\n",
    "      diff = knownOut - sigOut\n",
    "      error += sum(diff * diff)\n",
    "\n",
    "      gradient = ones(numOut) - (sigOut*sigOut)\n",
    "      outAdjust = gradient * diff \n",
    "\n",
    "      diff = sum(outAdjust * wOut, axis=1)\n",
    "      gradient = ones(numHid) - (sigHid*sigHid)\n",
    "      hidAdjust = gradient * diff \n",
    "\n",
    "      # update output \n",
    "      change = outAdjust * sigHid.reshape(numHid, 1)\n",
    "      wOut += (rate * change) + (momentum * cOut)\n",
    "      cOut = change\n",
    " \n",
    "      # update input \n",
    "      change = hidAdjust * sigIn.reshape(numInp, 1)\n",
    "      wInp += (rate * change) + (momentum * cInp)\n",
    "      cInp = change\n",
    " \n",
    "    if (minError is None) or (error < minError):\n",
    "      minError = error\n",
    "      bestWeightMatrices = (wInp.copy(), wOut.copy())\n",
    "      print(\"Step: %d Error: %f\" % (step, error))\n",
    "  \n",
    "  return bestWeightMatrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test the feed-forward neural network by using some test training data. As a very simple example, the first test takes input vectors with a pair of numbers which are either one or zero. The output corresponds to the “exclusive or” (XOR) logic function; the output is 1 if either of the inputs is 1, but not both. This test data is a list of `[input, output]`, pairs. Note that even though the output is just a single number it is nonetheless represented as a list with a single item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\nFeed-forward neural network simple test\\n\")\n",
    "\n",
    "data = [[[0,0], [0]],\n",
    "        [[0,1], [1]],\n",
    "        [[1,0], [1]],\n",
    "        [[1,1], [0]]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of hidden nodes used here is simply stated as 2, but in practical situations several values will need to be tried, and their performance evaluated. Then we run the training function in the data to estimate the best weight matrices for the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wMatrixIn, wMatrixOut = neuralNetTrain(data, 2, 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output weight matrices can then be run on test data for evaluation. At the very least they ought to do a reasonable job at predicting the output signals for the training set, although in practice these really ought to be for data that has not been used in the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for inputs, knownOut in data:\n",
    "  sIn, sHid, sOut =  neuralNetPredict(array(inputs), wMatrixIn, wMatrixOut)\n",
    "  print(knownOut, sOut[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Neural Network for Biological Sequences\n",
    "\n",
    "Next we move on from the trivial neural network test example to illustrate how feature vectors may be generated from biological sequences (and category data generally), so that they can be used in such machine learning programs. The example will predict the secondary structure of a residue in the middle of a five amino acid sequence. Both the amino acid sequence and the output secondary structure categories will be represented initially as code letters, but they will be converted into numbers (zeros and ones) before being passed into the feed-forward neural network. Although this example uses protein sequences an analogous procedure can be used for DNA and RNA.\n",
    "\n",
    "The test data that will be illustrated for the example is very small; simply to give a taste of the data and still have it fit here. As a result, a neural network trained on this data would be totally useless at making secondary structure predictions in practice. However, in the supporting material a file (`examples/SecStrucTrainingData.tsv`) with a large data set containing many thousands of sequences is available. Using this as input would give a vastly superior result. The test data is presented as a list of 2-tuples, each tuple has a five-letter residue protein sequence and a secondary structure code. Three secondary structure codes are used to represent three general conformations of protein backbone geometry. These codes are \"E\" for extended conformations (mostly beta-strand), \"H\" for helices (mostly alpha-helix) and \"C\" for random coil or unstructured stretches (everything else)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seqSecStrucData = [('ADTLL','E'),\n",
    "                   ('DTLLI','E'),\n",
    "                   ('TLLIL','E'),\n",
    "                   ('LLILG','E'),\n",
    "                   ('LILGD','E'),\n",
    "                   ('ILGDS','E'),\n",
    "                   ('LGDSL','C'),\n",
    "                   ('GDSLS','H'),\n",
    "                   ('DSLSA','H'),\n",
    "                   ('SLSAG','H'),\n",
    "                   ('LSAGY','H'),\n",
    "                   ('SAGYR','C'),\n",
    "                   ('AGYRM','C'),\n",
    "                   ('GYRMS','C'),\n",
    "                   ('YRMSA','C'),\n",
    "                   ('RMSAS','C')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before the above data can be used it will need to be converted from text strings into numeric feature vectors (arrays of numbers). All of the feature vectors that represent either sequence input or prediction output will contain numbers to represent the presence or absence of a particular category of item. In this example the feature vectors will contain ones to indicate the presence of and zeros to represent the absence of an amino acid (for input) or of a secondary structure code letter (for output). Other numbers could have been used instead with equal success (e.g. +/-1 or +/-0.5), although the two values chosen for presence or absence should naturally be distinct and lie in the range of the trigger function where there is a steep gradient; for the hyperbolic tangent example used here the range from -1 to +1 is usually best. Note that the size of the vector generated is the length of the input sequence times the number of possible letters; each element of the vector represents a different residue at a different sequence position. For a five-letter protein sequence the vector will have 20 elements (one for each amino acid) for each of the five sequence positions, and thus the total length will be 100.\n",
    "\n",
    "To make the feature vectors we first define dictionaries so that a letter can be used as a key to look up the position (index) in the vector that should be set to 1.0. The actual order that we go through the letter codes is unimportant, but it should be consistent for a given program. Here we could have used the list.index(letter) form to get a position index, but this is slower, especially if the number of possible letters and amount of training data is large. To make the index-lookup dictionary for the amino acids we loop through a list of possibilities and associate the residue letter code with the index i, which in this case was generated with enumerate():"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aminoAcids = 'ACDEFGHIKLMNPQRSTVWY'\n",
    "aaIndexDict = {}\n",
    "for i, aa in enumerate(aminoAcids):\n",
    "  aaIndexDict[aa] = i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same sort of thing is repeated for the secondary structure codes, albeit with a smaller number of possible letters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ssIndexDict = {}\n",
    "ssCodes = 'HCE'\n",
    "for i, code in enumerate(ssCodes):\n",
    "  ssIndexDict[code] = i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we actually do the conversion of the training data from the text strings to numeric vectors that can be used in the neural network routine. To help with this the `convertSeqToVector` function defined below is constructed to take a sequence of letters seq, and `indexDict` which can convert each letter to the correct position in the code alphabet. The vector initially starts filled with zeros, but then selective positions are converted to ones, depending on which sequence letters are observed. The actual index in the vector that needs to be set is determined by the index-lookup dictionary for that letter, `indexDict[letter]` and the start point for that sequence position, `pos * numLetters`. For example if the third letter (index 2, counting from 0) in seq is `“F”`, this adds a one at position 45; 40 to get to the start of the block that represents the third sequence position and 5 more because `“F”` is at index 5 in the protein sequence alphabet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convertSeqToVector(seq, indexDict):\n",
    "  \n",
    "  numLetters = len(indexDict)\n",
    "  vector = [0.0] * len(seq) * numLetters\n",
    "\n",
    "  for pos, letter in enumerate(seq):\n",
    "    index = pos * numLetters + indexDict[letter]  \n",
    "    vector[index] = 1.0\n",
    "\n",
    "  return vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The actual training data for the neural network is made by looping through all of the pairs of protein sequence and secondary structure code, and for each of these using the above function to make the feature vector from the text. The trainingData list is constructed as linked pairs of input and output feature vectors. Note that because the secondary structure code ss is only a single letter the output vector will be of length three. Specifically, for the three codes the output vectors will be as follows: `“E”`; `[0.0, 0.0, 1.0]` `“C”`; `[0.0, 1.0, 0.0]` `“H”`; `[1.0, 0.0, 0.0]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainingData = []\n",
    "for seq, ss in seqSecStrucData:\n",
    "\n",
    "  inputVec = convertSeqToVector(seq, aaIndexDict)\n",
    "  outputVec = convertSeqToVector(ss, ssIndexDict)\n",
    "\n",
    "  trainingData.append( (inputVec, outputVec) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of hidden nodes is set to three, by way of example. The training data and network size are then passed into the main training function, in order to generate the predictive weight matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wMatrixIn, wMatrixOut = neuralNetTrain(trainingData, 3, 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After training, the neural network can make secondary structure predictions for any five-letter protein sequence, but as before, this must be converted into the numeric vector form. Because the prediction operates on a whole numpy.array the test vector is converted into an array (here with one element). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "testSeq = 'DLLSA'\n",
    "testVec = convertSeqToVector(testSeq, aaIndexDict)\n",
    "testArray = array( [testVec,] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The weight matrices from the training are used to make predictions on the test array and the output signal, `sOut` is interrogated to find the position of the largest value (i.e. the position predicted to be one not zero). This index represents the best secondary structure code, and the code itself can be obtained by using the index with the original set of codes to get a letter back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sIn, sHid, sOut =  neuralNetPredict(testArray, wMatrixIn, wMatrixOut)\n",
    "index = sOut.argmax()\n",
    "print(\"Test prediction: %s\" % ssCodes[index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}