tutorial_sensitivity.ipynb

{
  "metadata": {
    "name": "Classification Model Parameters -- Sensitivity Analysis"
  }, 
  "nbformat": 3, 
  "nbformat_minor": 0, 
  "worksheets": [
    {
      "cells": [
        {
          "cell_type": "heading", 
          "level": 1, 
          "metadata": {}, 
          "source": [
            "Classification Model Parameters -- Sensitivity Analysis"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "In the ", 
            "*chap_tutorial_searchlight* we made a first attempt at localizing\ninformation in the brain that is relevant to a particular classification\nanalyses. While we were relatively successful, we experienced some problems and\nalso had to wait quite a bit. Here we want to look at another approach to\nlocalization. To get started, we pre-process the data as we have done before\nand perform volume averaging to get a single sample per stimulus category and\noriginal experiment session."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "from mvpa2.tutorial_suite import *\n", 
            "ds = get_raw_haxby2001_data(roi='brain')\n", 
            "print ds.shape"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "poly_detrend(ds, polyord=1, chunks_attr='chunks')\n", 
            "zscore(ds, param_est=('targets', ['rest']))\n", 
            "ds = ds[ds.sa.targets != 'rest']\n", 
            "run_averager = mean_group_sample(['targets', 'chunks'])\n", 
            "ds = ds.get_mapped(run_averager)\n", 
            "print ds.shape"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "A searchlight analysis on this dataset would look exactly as we have seen in\n", 
            "*chap_tutorial_searchlight*, but it would take a bit longer due to a\nhigher number of samples. The error map that is the result of a searchlight\nanalysis only offers an approximate localization. First, it is smeared by the\noverlapping spheres and second the sphere-shaped ROIs probably do not reflect\nthe true shape and extent of functional subregions in the brain. Therefore, it\nmixes and matches things that might not belong together. This can be mitigated\nto some degree by using more clever searchlight algorithms (see\n", 
            "*example_searchlight_surf*).  But it would also be much nicer if we were\nable to obtain a per-feature measure, where each value can really be attributed\nto the respective feature and not just to an area surrounding it."
          ]
        }, 
        {
          "cell_type": "heading", 
          "level": 2, 
          "metadata": {}, 
          "source": [
            "It's A Kind Of Magic"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "One way to get such a measure is to inspect the classifier itself. Each\nclassifier creates a model to map from the training data onto the\nrespective target values. In this model, classifiers typically associate\nsome sort of weight with each feature that is an indication of its impact\non the classifiers decision. How to get this information from a\nclassifier will be the topic of this tutorial.\n\n", 
            "However, if we want to inspect a trained classifier, we first have to train\none. But hey, we have a full brain dataset here with almost 40k features.\nWill we be able to do that? Well, let's try (and hope that there is still a\nwarranty on the machine you are running this on...).\n\n", 
            "We will use a simple cross-validation procedure with a linear support\nvector machine.  We will also be interested in summary statistics of the\nclassification, a confusion matrix in our case of classification:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "clf = LinearCSVMC()\n", 
            "cvte = CrossValidation(clf, NFoldPartitioner(),\n                       enable_ca=['stats'])"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Ready, set, go!"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "results = cvte(ds)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "That was surprisingly quick, wasn't it? But was it any good?"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print np.round(cvte.ca.stats.stats['ACC%'], 1)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print cvte.ca.stats.matrix "
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Well, the accuracy is not exactly at a chance level, but the confusion matrix doesn't\nseem to have any prominent diagonal. It looks like, although we can easily\ntrain a support vector machine on the full brain dataset, it cannot construct\na reliably predicting model.  At least we are in the lucky situation to already know\nthat there is some signal in the data, hence we can attribute this failure\nto the classifier. In most situations it would be as likely that there is\nactually no signal in the data...\n\n", 
            "Often people claim that classification performance improves with\n", 
            "[feature selection](http://pymvpa.org/glossary.html#term-feature-selection). If we can reduce the dataset to the important ones,\nthe classifier wouldn't have to deal with all the noise anymore. A simple\napproach would be to compute a full-brain ANOVA and only go with the voxels\nthat show some level of variance between categories. From the\n", 
            "*chap_tutorial_searchlight* we know how to compute the desired F-scores\nand we could use them to manually select features with some threshold. However,\nPyMVPA offers a more convenient way -- feature selectors:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "fsel = SensitivityBasedFeatureSelection(\n           OneWayAnova(),\n           FixedNElementTailSelector(500, mode='select', tail='upper'))"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "The code snippet above configures such a selector. It uses an ANOVA measure\nto select 500 features with the highest F-scores. There\nare a lot more ways to perform the selection, but we will go with this one\nfor now. The ", 
            "[SensitivityBasedFeatureSelection](http://pymvpa.org/generated/mvpa2.featsel.base.SensitivityBasedFeatureSelection.html#mvpa2-featsel-base-sensitivitybasedfeatureselection)\ninstance is yet another ", 
            "[processing object](http://pymvpa.org/glossary.html#term-processing-object) that can be called with a\ndataset to perform the feature selection:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "fsel.train(ds)\n", 
            "ds_p = fsel(ds)\n", 
            "print ds_p.shape"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "This is the dataset we wanted, so we can rerun the cross-validation and see\nif it helped. But first, take a step back and look at this code snippet again.\nThere is an object that gets called with a dataset and returns a dataset. You\ncannot prevent noticing the striking similarity between a measure in PyMVPA or\na mapper. And yes, feature selection procedures are also\n", 
            "[processing object](http://pymvpa.org/glossary.html#term-processing-object)s and work just like measures or mappers. Now back\nto the analysis:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "results = cvte(ds_p)\n", 
            "print np.round(cvte.ca.stats.stats['ACC%'], 1)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print cvte.ca.stats.matrix"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Yes! We did it. Almost 80% correct classification for an 8-way\nclassification and the confusion matrix has a strong diagonal. Apparently,\nthe ANOVA-selected features were the right ones."
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n**Exercise**", 
            "\n\n", 
            "If you are not yet screaming or haven't started composing an email to the\nPyMVPA mailing list pointing to a major problem in the tutorial, you need\nto reconsider what we have just done. Why is this wrong?"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "# you can use this cell for this exercise"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n", 
            "\n\n", 
            "Let's repeat this analysis on a subset of the data. We select only `bottle`\nand `shoe` samples. In the analysis we just did, they are relatively often\nconfused by the classifier. Let's see how the full brain SVM performs on\nthis binary problem"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "bin_demo = ds[np.array([i in ['bottle', 'shoe'] for i in ds.sa.targets])]\n", 
            "results = cvte(bin_demo)\n", 
            "print np.round(cvte.ca.stats.stats['ACC%'], 1)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Not much, but that doesn't surprise. Let's see what effect our ANOVA-based\nfeature selection has"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "fsel.train(bin_demo)\n", 
            "bin_demo_p = fsel(bin_demo)\n", 
            "results = cvte(bin_demo_p)\n", 
            "print cvte.ca.stats.stats[\"ACC%\"]"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Wow, that is a jump. Perfect classification performance, even though the\nsame categories couldn't be distinguished by the same classifier, when\ntrained on all eight categories. I guess, it is obvious that our way of\nselecting features is somewhat fishy -- if not illegal. The ANOVA measure\nuses the full dataset to compute the F-scores, hence it determines which\nfeatures show category differences in the whole dataset, including our\nsupposed-to-be independent testing data. Once we have found these\ndifferences, we are trying to rediscover them with a classifier.  Being able\nto do that is not surprising, and precisely constitutes the ", 
            "*double-dipping*\nprocedure. As a result, both the obtained prediction\naccuracy and the created model are potentially completely meaningless."
          ]
        }, 
        {
          "cell_type": "heading", 
          "level": 2, 
          "metadata": {}, 
          "source": [
            "Thanks For The Fish"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "To implement an ANOVA-based feature selection ", 
            "*properly* we have to do it on\nthe training dataset ", 
            "**only**. The PyMVPA way of doing this is via a\n", 
            "[FeatureSelectionClassifier](http://pymvpa.org/generated/mvpa2.clfs.meta.FeatureSelectionClassifier.html#mvpa2-clfs-meta-featureselectionclassifier):"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "fclf = FeatureSelectionClassifier(clf, fsel)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "This is a ", 
            "[meta-classifier](http://pymvpa.org/glossary.html#term-meta-classifier) and it just needs two things: A basic\nclassifier to do the actual classification work and a feature selection\nobject. We can simply re-use the object instances we already had. Now we\ngot a meta-classifier that can be used just as any other classifier. Most\nimportantly we can plug it into a cross-validation procedure (almost\nidentical to the one we had in the beginning)."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "cvte = CrossValidation(fclf, NFoldPartitioner(),\n                       enable_ca=['stats'])\n", 
            "results = cvte(bin_demo)\n", 
            "print np.round(cvte.ca.stats.stats['ACC%'], 1)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "This is a lot worse and a lot closer to the truth -- or a so-called\n", 
            "[unbiased estimate](http://pymvpa.org/glossary.html#term-unbiased-estimate) of the generalizability of the classifier model.\nNow we can also run this improved procedure on our original 8-category\ndataset."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "results = cvte(ds)\n", 
            "print np.round(cvte.ca.stats.stats['ACC%'], 1)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print cvte.ca.stats.matrix"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "That is still a respectable accuracy for an 8-way classification and the\nconfusion table also confirms this."
          ]
        }, 
        {
          "cell_type": "heading", 
          "level": 2, 
          "metadata": {}, 
          "source": [
            "Dissect The Classifier"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "But now back to our original goal: getting the classifier's opinion about\nthe importance of features in the dataset. With the approach we have used\nabove, the classifier is trained on 500 features. We can only have its\nopinion about those. Although this is just few times larger than a typical\nsearchlight sphere, we have lifted the spatial constraint of\nsearchlights -- these features can come from all over an ROI.\n\n", 
            "However, we still want to consider more features, so we are changing the\nfeature selection to retain more."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "fsel = SensitivityBasedFeatureSelection(\n           OneWayAnova(),\n           FractionTailSelector(0.05, mode='select', tail='upper'))\n", 
            "fclf = FeatureSelectionClassifier(clf, fsel)\n", 
            "cvte = CrossValidation(fclf, NFoldPartitioner(),\n                       enable_ca=['stats'])\n", 
            "results = cvte(ds)\n", 
            "print cvte.ca.stats.stats['ACC%'] >= 78.1"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print np.round(cvte.ca.stats.stats['ACC%'], 1)  "
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "A drop of 8% in accuracy on about 4 times the number of features. This time\nwe asked for the top 5% of F-scores.\n\n", 
            "But how do we get the weights, finally? In PyMVPA many classifiers\nare accompanied with so-called ", 
            "[sensitivity analyzer](http://pymvpa.org/glossary.html#term-sensitivity-analyzer)s. This is an\nobject that knows how to get them from a particular classifier type (since\neach classification algorithm hides them in different places). To create\nthis ", 
            "*analyzer* we can simply ask the classifier to do it:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "sensana = fclf.get_sensitivity_analyzer()\n", 
            "type(sensana)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "As you can see, this even works for our meta-classifier. And again this\nanalyzer is a ", 
            "[processing object](http://pymvpa.org/glossary.html#term-processing-object) that returns the desired sensitivity\nwhen called with a dataset."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "sens = sensana(ds)\n", 
            "type(sens)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print sens.shape"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Why do we get 28 sensitivity maps from the classifier? The support vector\nmachine constructs a model for binary classification problems. To be able to deal\nwith this 8-category dataset, the data is internally split into all\npossible binary problems (there are exactly 28 of them). The sensitivities\nare extracted for all these partial problems."
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n**Exercise**", 
            "\n\n", 
            "Figure out which sensitivity map belongs to which combination of\ncategories."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "# you can use this cell for this exercise"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n", 
            "\n\n", 
            "If you are not interested in this level of detail, we can combine the maps\ninto one, as we have done with dataset samples before. A feasible\nalgorithm might be to take the per feature maximum of absolute\nsensitivities in any of the maps. The resulting map will be an indication\nof the importance of feature for ", 
            "*some* partial classification."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "sens_comb = sens.get_mapped(maxofabs_sample())"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n**Exercise**", 
            "\n\n", 
            "Project this sensitivity map back into the fMRI volume and compare it to\nthe searchlight maps of different radii from the ", 
            "*previous tutorial*."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "# you can use this cell for this exercise"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n", 
            "\n\n", 
            "You might have noticed some imperfection in our recent approach to computing\na full-brain sensitivity map. We derived it from the full dataset, and not\nfrom cross-validation splits of the data. Rectifying this is easy with a\nmeta-measure. A meta-measure is analogous to a meta-classifier: a measure\nthat takes a basic measure, adds a processing step to it and behaves like a\nmeasure itself. The meta-measure we want to use is\n", 
            "[RepeatedMeasure](http://pymvpa.org/generated/mvpa2.measures.base.RepeatedMeasure.html#mvpa2-measures-base-repeatedmeasure)."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "sensana = fclf.get_sensitivity_analyzer(postproc=maxofabs_sample())\n", 
            "cv_sensana = RepeatedMeasure(sensana,\n                             ChainNode((NFoldPartitioner(),\n                                        Splitter('partitions',\n                                                 attr_values=(1,)))))\n", 
            "sens = cv_sensana(ds)\n", 
            "print sens.shape"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "We re-create our basic sensitivity analyzer, this time automatically applying\nthe post-processing step that combines the sensitivity maps for all partial\nclassifications. Finally, we plug it into the meta-measure that uses the\npartitions generated by ", 
            "[NFoldPartitioner](http://pymvpa.org/generated/mvpa2.datasets.splitters.NFoldPartitioner.html#mvpa2-datasets-splitters-nfoldpartitioner) to\nextract the training portions of the dataset for each fold. Afterwards, we can\nrun the analyzer and we get another dataset, this time with a sensitivity map\nper each cross-validation split.\n\n", 
            "We could combine these maps in a similar way as before, but let's look at\nthe stability of the ANOVA feature selection instead."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "ov = MapOverlap()\n", 
            "overlap_fraction = ov(sens.samples > 0)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "With the ", 
            "[MapOverlap](http://pymvpa.org/generated/mvpa2.misc.support.MapOverlap.html#mvpa2-misc-support-mapoverlap) helper we can easily\ncompute the fraction of features that have non-zero sensitivities in all\ndataset splits."
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n**Exercise**", 
            "\n\n", 
            "Inspect the `ov` object. Access that statistics map with the fraction\nof per-feature selections across all splits and project them back into\nthe fMRI volume to investigate them."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "# you can use this cell for this exercise"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n", 
            "\n\n", 
            "This could be the end of the data processing. However, by using the meta\nmeasure to compute the sensitivity maps we have lost a convenient way to\naccess the total performance of the underlying classifier. To again gain\naccess to it, and get the sensitivities at the same time, we can twist the\nprocessing pipeline a bit."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "sclf = SplitClassifier(fclf, enable_ca=['stats'])\n", 
            "cv_sensana = sclf.get_sensitivity_analyzer()\n", 
            "sens = cv_sensana(ds)\n", 
            "print sens.shape"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print cv_sensana.clf.ca.stats.matrix  "
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "I guess that deserves some explanation. We wrap our\n", 
            "[FeatureSelectionClassifier](http://pymvpa.org/generated/mvpa2.clfs.meta.FeatureSelectionClassifier.html#mvpa2-clfs-meta-featureselectionclassifier) with a new thing, a\n", 
            "[SplitClassifier](http://pymvpa.org/generated/mvpa2.clfs.meta.SplitClassifier.html#mvpa2-clfs-meta-splitclassifier). This is another meta classifier\nthat performs splitting of a dataset and runs training (and prediction) on\neach of the dataset splits separately. It can effectively perform a\ncross-validation analysis internally, and we ask it to compute a confusion\nmatrix of it. The next step is to get a sensitivity analyzer for this meta\nmeta classifier (this time no post-processing). Once we have got that, we\ncan run the analysis and obtain sensitivity maps from all internally\ntrained classifiers. Moreover, the meta sensitivity analyzer also allows\naccess to its internal meta meta classifier that provides us with the\nconfusion statistics. Yeah!\n\n", 
            "While we are at it, it is worth mentioning that the scenario above can be\nfurther extended. We could add more selection or pre-processing steps\ninto the classifier, like projecting the data onto PCA components and\nlimit the classifier to the first 10 components -- for each split. PyMVPA\noffers even more complex meta classifiers (e.g.\n", 
            "[TreeClassifier](http://pymvpa.org/generated/mvpa2.clfs.meta.TreeClassifier.html#mvpa2-clfs-meta-treeclassifier)) that might be very helpful in some\nanalysis scenarios."
          ]
        }, 
        {
          "cell_type": "heading", 
          "level": 2, 
          "metadata": {}, 
          "source": [
            "Closing Words"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "We have seen that sensitivity analyses are a useful approach to localize\ninformation that is less constrained and less demanding than a searchlight\nanalysis.  Specifically, we can use it to discover signals that are\ndistributed throughout the whole set of features (e.g. the full brain),\nbut we could also perform an ROI-based analysis with it. It is less\ncomputationally demanding as we only train the classifier on one set of\nfeatures and not thousands, which results in a significant reduction of\nrequired CPU time.\n\n", 
            "However, there are also caveats. While sensitivities are a much more\ndirect measure of feature importance in the constructed model, being\nclose to the bare metal of classifiers also has problems. Depending on the\nactual classification algorithm and data preprocessing sensitivities might mean something\ncompletely different when compared across classifiers. For example, the\npopular SVM algorithm solves the classification problem by identifying the\ndata samples that are ", 
            "*most tricky* to model. The extracted sensitivities\nreflect this property. Other algorithms, such as \"Gaussian Naive Bayes\"\n(", 
            "[GNB](http://pymvpa.org/generated/mvpa2.clfs.gnb.GNB.html#mvpa2-clfs-gnb-gnb)) make assumptions about the distribution of\nthe samples in each category. GNB sensitivities ", 
            "*might* look completely\ndifferent, even if GNB and SVM classifiers both perform at comparable accuracy levels.\nNote, however, that these properties can also be used to address related\nresearch questions.\n\n", 
            "It should also be noted that sensitivities can not be directly compared to\neach other, even if they stem from the same algorithm and are just\ncomputed on different dataset splits. In an analysis one would have to\nnormalize them first. PyMVPA offers, for example,\n", 
            "[l1_normed()](http://pymvpa.org/generated/mvpa2.misc.transformers.l1_normed.html#mvpa2-misc-transformers-l1-normed) and\n", 
            "[l2_normed()](http://pymvpa.org/generated/mvpa2.misc.transformers.l2_normed.html#mvpa2-misc-transformers-l2-normed) that can be used in conjunction\nwith ", 
            "[FxMapper](http://pymvpa.org/generated/mvpa2.mappers.fx.FxMapper.html#mvpa2-mappers-fx-fxmapper) to do that as a post-processing\nstep.\n\n", 
            "In this tutorial part we also touched the surface of another important\ntopic: ", 
            "[feature selection](http://pymvpa.org/glossary.html#term-feature-selection). We performed an ANOVA-based feature\nselection prior to classification to help SVM achieve acceptable\nperformance. One might wonder if that was a clever idea, since a\n", 
            "*univariate* feature selection step prior to a ", 
            "*multivariate* analysis\nsomewhat contradicts the goal to identify ", 
            "*multivariate* signals. Only\nfeatures will be retained that show some signal on their own. If that\nturns out to be a problem for a particular analysis, PyMVPA offers a\nnumber of multivariate alternatives for features selection. There is an\nimplementation of ", 
            "[recursive feature selection](http://pymvpa.org/glossary.html#term-recursive-feature-selection)\n(", 
            "[RFE](http://pymvpa.org/generated/mvpa2.featsel.rfe.RFE.html#mvpa2-featsel-rfe-rfe)) and a helper to simplify its invocation for a\ntypical use-case (", 
            "[SplitRFE](http://pymvpa.org/generated/mvpa.featself.rfe.SplitRFE.html#mvpa-featself-rfe-splitrfe)).  Also all classifier sensitivities\ncan be used to select features. For classifiers where sensitivities cannot\neasily be extracted PyMVPA provides a noise perturbation measure\n(", 
            "[NoisePerturbationSensitivity](http://pymvpa.org/generated/mvpa2.measures.noiseperturbation.NoisePerturbationSensitivity.html#mvpa2-measures-noiseperturbation-noiseperturbationsensitivity);\nsee ", 
            "*Hanson et al. (2004)* for an example application).\n\n", 
            "With these building blocks it is possible to run fairly complex analyses.\nHowever, interpreting the results might not always be straight-forward. In\nthe ", 
            "*next tutorial part* we will set out\nto take away another constraint of all our previously performed analyses. We\nare going to go beyond spatial analyses and explore the time dimension."
          ]
        }
      ], 
      "metadata": {}
    }
  ]
}